ProductionScale

The question of weather SaaS is Cloud Computing keeps coming up again and again.  So, since I didn't get to attend that session at Cloud Camp SF I'll weigh in here.

I view cloud computing as a commercialized extension to utility/grid computing from government and educational spaces.  My definition of cloud computing is, "cloud computing is a commercial extension of utility computing that enables scalable, elastic, highly available deployment of software applications while minimizing the level of detailed interaction with the underlying technology stack itself."

More details in this article (mine):
http://www.productionscale.com/home/2008/4/24/cloud-computing-get-your-head-in-the-clouds.html

In that definition you can see that I'm positing that SaaS is enabled by Cloud Computing.  SaaS is the "software application" that is enabled and endowed with all those awesome attributes provided by Cloud Computing.  That makes most SaaS offerings Cloud Computing Applications as opposed to traditional stand alone or even web applications.  It does not make the applications themselves Cloud Computing.

The most interesting thing about Cloud Computing to me is that it enables entirely new types of business to exist and be economically viable that never could have persisted before.  The economics changed and this is only the tip of the iceburg.  Fun times!

In the last week I attended Cloud Camp and Structure 08 in San Francisco, CA.  Both events were well worth the time and I thoroughly enjoyed meeting so many great people.

I think that Jonathan Yarmis' talk summed up my feeling the best when he said, "The world is about to change, and change in profoundly interesting ways."  I just can't agree more.  This is one of the most amazing times and there is so much opportunity!

I ran an un-conference session at Cloud Camp on the topic, "Developing and Deploying Web Applications in the Cloud:  Best Practices and Pitfalls.  As best I can make out my scribble on the white board a group of about 30 came up with the following.

The point of this session was to outline things you can do to make the applications you develop and deploy work better in the cloud.

Here are some of the people from the session.

 

Best Practices

Keep some Secrets
Automatate Deployment and Testing
Follow Coding Standards
Log, Log, Log
Monitor very well
Instramentation
Make developers feels the pain too (developers said this!)
Use Change Management
Avoid Shared Resources (for scale) - NFS for example
Stay Stateless
Create best practices to follow
Define your metrics (peformance, etc.) up front
TEST TEST TEST
Profile your code

Pitfalls

<unfortunately my photo didn't come out good enough to read of that side of the board, if you have one let me know please>

Resources

<also couldn't read this, silly iPhone camera>

In summary, my big take away from this week is that we are right at the tipping point where the market is latching on to some of the concepts of Cloud Computing and seeing the potential.  My guess is that in the the next 3-5 years Cloud Computing will be as pervasive as Social Networking is today probably.

Designing, launching, and integrating community platforms that leverage trends in modern computing such as Cloud Computing and Virtualization well can be a tremendous strategic benefit for a company. However, much of those benefits can be quickly erased by operational and development issues surrounding doing that successfully are often marginalized. This has fiscal, operational, and morale impacts over time.

ROI no matter who you might calculate it will include the total actual costs (TCO - Total Costs of Ownership) of running or renting the platforms that community sites run upon.

Will your system stand up to Digg/Slashdot/TechCrunch/Reddit or exponential user base growth? Does it have a clear scalability path? These are questions that development, operations, and business staff need to discuss periodically throughout any community development project and long after when the project is live.

I saw a case study recently about a facebook community app that went from zero to a billion page views per MONTH in just a short time. I’m sure they thought about scalability and operations upfront at least a little or they would not still be in business. Do you know what you would do if your community caught fire like that? Would you get a dreaded “Site Not Available” message at your time of greatest success?

If you plan your communities technology platform correctly from the start you have a much better chance of succeeding and fully leveraging your investments in your businesses technology over time.

Here are some things you might want to think about and questions to ask. They pretty much all break down to how you spend time and money. Since time and money are often inexorably related.

Time. How much time is needed to upgrade, maintain, and manage the underlying infrastructure as the site grows?

Time. How long will it take you to build the infrastructure out from development through production with proper environments and capacity?

Money. What are your up front capital costs?

Money. What are your on-going maintenance costs for infrastructure and operational support?

Money. How much operational support will you need to keep things running and scaling?

Money. What is the total cost of ownership of this community including the underlying infrastructure, software, and people?

Ask those questions and you will lay the groundwork for better understanding of your project and a better opportunity for success.

Note:  This is a Re-Post of my Blog entry for blog.solutionset.com

We are awash in a world of new and disruptive technology products and services.  One of the most prevalent is Cloud Computing.  The topic of this article is primarily Private Cloud Computing and thoughts on what it will mean for Enterprise IT.  Unfortunately it's a bit of a ramble but I'm not much for spending more hours tweaking and editing this one.

What is private cloud computing?  To make a non-technical analogy, Private Cloud Computing is a little like owning your own car instead of using a rental car that you share with others others and that someone else owns for your automobile and transportation needs.  Rental cars haven't completely replaced personal automobile ownership for many obvious reasons.  Public Cloud Services will not likely replace dedicated private servers either and will likely drive adoption of private cloud computing.

Security is a serious issue with public cloud computing in any form.  Would you be willing to run a critical corporate financial application on a shared cloud computing platform when the terms of service state something like, if we get a court order we'll hand your data over without question or battle and there is nothing you can do about that fact because by using our service you implicitly give us that right as a means of protecting ourselves?  In fact, entire countries are already becoming embroiled in this very debate.   In a recent article in Network World there was discussion about Cloud Storage and international law.  The question was asked, "What does it mean if your data is stored in the cloud and some foreign government entity might have access to it?" That is in fact a really interesting question!

When you have a few minutes and a law degree take a look at seciont 5.4.2 of Amazon Web Services Terms of Service. I am not an attorney but to me it says, you data/information can and will be turned over if we feel like it.  Other parts of that document limit liability and such for Amazon as well.  Now, were I Amazon I'd do exactly the same thing I suppose.  This isn't an article about Amazon though.  It's an article about why private cloud computing is a must.  You can't necessary trust your sensitive data, even in transient states, to public service providers that are required to write Terms of Service agreements like that.

The first stages have already blown right by us in the form of Virtualization technologies like VMWare, Xen, and others that have been commercialized over the last several years and themselves become enablers of the current wave of products and services.  Terms like VM (virtual machine), and Virtualization (which my spell checker still doesn't even recognize) are common now in IT departments and CxO suites everywhere.  Xen is an underlying piece of Amazon Web Services EC2 platform.  VMWare underlies the virtualization and cloud offerings of some newer companies.  Then, there are even companies creating management tools to better manage Xen, VMWare, EC2, and other platforms better than the companies that created the software in the first place.  There are still laggards that haven't effectively used or even explored the earlier rounds of such technologies yet for strategic and tactical business purposes.  We all know what happened to the dinosaurs when things changed.

Today there are very few companies that have the internal knowledge and the resources to create and effectively manage true Cloud Computing infrastructures.  These are well known names like Google, Yahoo, and Amazon.  It is these companies that have dramatically leveraged their internal and originally Private Cloud Computing infrastructures to significant economic benefit.  But, this fact is changing very quickly.

There are many up and coming competitors as well that can make significant contributions in the private cloud computing space over time like Nirvanix, XCalibre, 3Tera, Enomolism, and Joyent.  Then, there are companies like Gigaspaces, 3Tera, and Enomolism that are already rolling out private cloud related services and products that you can use in your own data center.  While some have relatively mature feature sets, the genre as a whole is still very young and deeply segmented.  But, what is happening is a fast and significant shift towards the ubiquitous availability of the technology required to install and manage your own personal cloud computing infrastructures.

As with so many things, this is very much about business economics.  The companies, like Google, that have pioneered and used their private cloud computing platforms effectively today and over the last few years can and do generate staggering economic values.  The economic benefits should, if the project is properly executed,  translate directly to the bottom line in a variety of ways due to increased technology operations effectiveness.

Existing corporations across many industries can benefit similarly to various degrees by leveraging both public cloud computing services and also building private clouds behind the firewall when it's necessary.  The New York Times is a fairly dramatic and recent example of this trend.  They've used modern shared cloud computing to generate economic value from old archived content.

What does all this mean for Corporate IT?  The trend in IT that will materialize more clearly over time is a shift toward internal software and hardware changes that will make the corporate infrastructures resemble current commercial cloud computing infrastructures.  Enterprises need and will want these technologies.  But, the necessary trade-offs regarding security and privacy inherent in public cloud computing will likely cause private clouds will flourish.  Even though the public services themselves are technically secure; it's simply a domestic and international legal matter no matter what country your businesses home office resides.  Just have your attorney read Amazon's terms of service and let us know what they have to say.  So, we'll probably see white label private cloud computing products and services spawn across the industry and settle right in behind the corporate firewall.  Companies will find this economically desirable, managerially pragmatic, cost effective, and very disruptive to their IT Departments.

Nati Shalom again has posted a thoughtful and excellent blog post

Twitter as a scalability case study

 

So, I just put my comment right there on his blog and it's probably easier to read it there in-line.  Here is a re-post of my response though since it was practically a blog post anyway.  It doesn't really stand totally on it's own so you should check out Nati's post first.

A wise article. Thank you. My thoughts after reading it some directly or tangentially related...

Slowly but surely people are catching on that what can generally be referred to as loosely coupled asynchronous capable systems architecture stacks and software architectures are critical to building truly scalable systems. Twitter even knew this early on and tried to adjust many times but for reasons unknown by me still made/makes some rather odd choices about their systems and software architecture; their mysql usage for example. Funny thing is, the body of work exists to build these sites already. People just keep focusing on the wrong things like language wars or putting the individual problems into overly broad problem domains or applying the wrong solutions all together.

Today's languages and frameworks can take some of the sting out of developing and scaling an application to a point but once an application moves beyond any significant traffic level problems inevitably arrive that lay bare all the bad choices that followed before. The people who really know how to address and fix those problems are few and far between. The people who know how to avoid those problems from the very beginning based on their experience are even more rare. The companies who have those rare people on staff and actually listen to them almost don't exist at all.

I think, in business, the definition of insanity is doing the same thing over and over and expecting a different result. If so, we're seeing industry insanity around the concepts of designing web based applications that will scale. People just keep making the same mistakes again and again.

I've been thinking about all this in terms of site traffic for the average implementation of a systems and software architecture underlying a web application that might grow on todays terms. I think the page views per month to description looks loosely like this.

0 - 100,000 page views = micro site
100,000 - 1,000,000 = small site - troubles start here
1,000,000 - 100,000,000 = large site - troubles magnify dramatically here. OMG Rewrite!
100,000,000 - 1 billion = very large site - nothing works that used to work because it just can't because your system died a tragic death

Each of these requires certain skills and knowledge to build to. But all of them can be handled if planned for well up front. It's commonly held that premature scaling is the root of all evil ( and cost over runs). It's just not true. Designing the systems and software architecture of your site to be a very large site doesn't require the CAPEX expense to do so any more. It's just an technology architecture problem and it requires a broad range of skills to solve.

Anyway, I've officially rambled on and I want ice cream so bye!

Thanks for reading ProductionScale!

 

If you know me then you know I really like grids, clusters, and large scale computing technologies.  So, I was happy to get the opportunity to attend the Hadoop Meeting up in San Francisco a little over a week ago.  They said it filled up in 1-2 hours so I was lucky to get a spot it would seem.  I'm a fan of the Hadoop project and I've been thinking about it for a while.  What follows are some of my observations and thoughts about the event.

The Realities

I asked what the technical barriers were causing a 2000 node limit.  I got back an answer that the namenode for the HDFS cluster simply can't scale beyond a couple of thousands nodes since it's a single point of failure non-horizontally scalable system.  That's not exactly the way they put it but that's what they meant I do believe.  That's a surprising weakness really.  The namenode does two primary things.  It manages file system namespace, including the replication factor, how many replicas of any single file there are in the total system, and regulates access to files by clients.  For better or worse, I found myself wondering what needs to be done is to port FSImage and EditLog, the two main components of the namenode, to a distributed database format like CouchDB. Everything in the name node seems to be just a name/value pair that would benefit from using a distributed columnar DBMS.

I also asked the admittedly odd question of what is Hadoop very bad at doing.  I really didn't get a good answer on that overall.  But, I'll interpret the entire evening into saying that it's bad at being highly available and massively scalable due to currently existing single points of failure.  This is one area where Hadoop and all it's related components shows it's youth.  Of course, what I really meant was, what can Yahoo NOT port to Hadoop for competitive advantage because it's just not in the Hadoop map-reduce problem domain.  Unfortunately, I didn't ask the question well enough.  In general, there is as much or more sometimes to be learned from failed mistakes and experiments than from successes.  But, nobody wants to talk about that much usually.

I also observed a trend in the statements made throughout the talk in that the code could be significantly optimized to run better on fewer nodes.  In short, that would certainly be a money saver and help push the system with it's current architecture a little further.

They also mentioned that in no way are they trying to do the Globus like thing of distributing work over a WAN (Wide Area Network).  The excellent Globus project is a long standing Grid framework well worth examining.  This is a serious limitation of the system overall in terms of it's overall scalability in my opinion.  It was also contrary to an earlier statement that said, "if you are going to be scalable then you must be distributed."  But, to be fair, they reminded us many times that the software is young and there is still a lot left to do.  In particular, it sounds like they are working very hard on their advanced job scheduler which will likely have some relevance in this area some day.

I learned that in no way does Yahoo consider Hadoop to be truly "real-time" production ready.  Although they are using it and actively porting applications to use it, they seemed very hesitant about proclaiming it production ready.  I think this reticence is primarily related to performance and scalability issues and the difficulty in job scheduling in complex multi-tenant environments.

Another piece of software and knowledge I picked up that is worth mentioning was regarding HBase and Varnish.  I have no idea why I didn't think of this but you can front HBase with Varnish (or any reverse cache; aka any CDN probably) to scale read performance of the dataset/documents in document data stores.  I need to email Jan from the CouchDB project and ask him if he's ever tried this for accelerating/caching CouchDB reads.  I use varnish plenty for other projects but I hadn't thought to use varnish this way.  So, I found myself wondering why not front HBase with Akamai for a globally distributed read farm?  Anyway, those were the kinds of technical gems I was hoping for at the meetup.

I would have liked a little more tech and a case study type of information as opposed to the Hadoop history lesson.  I knew the history going into the event so I was looking for more readily useful information.

The Take Away

Hadoop is a very nice piece of work and it, combined cloud computing initiatives it is well on the way to creating some extremely powerful tools for many uses.  There is a great deal to learn from what the Hadoop project has accomplished thus far and it seems like they are on the path to greater things.  I certainly got the message that they are working hard on the scalability and availability issues.  I think this just adds more validity to my feeling that we live in an very exciting time technologically and that we have really only just scratched he surface of what is possible.  I walked away with some new ideas and a good checkpoint on where the technology is today.  Thanks to GigaOM for sponsoring.