ROI no matter who you might calculate it will include the total actual costs (TCO - Total Costs of Ownership) of running or renting the platforms that community sites run upon.

Will your system stand up to Digg/Slashdot/TechCrunch/Reddit or exponential user base growth? Does it have a clear scalability path? These are questions that development, operations, and business staff need to discuss periodically throughout any community development project and long after when the project is live.

I saw a case study recently about a facebook community app that went from zero to a billion page views per MONTH in just a short time. I’m sure they thought about scalability and operations upfront at least a little or they would not still be in business. Do you know what you would do if your community caught fire like that? Would you get a dreaded “Site Not Available” message at your time of greatest success?

If you plan your communities technology platform correctly from the start you have a much better chance of succeeding and fully leveraging your investments in your businesses technology over time.

Here are some things you might want to think about and questions to ask. They pretty much all break down to how you spend time and money. Since time and money are often inexorably related.

Time. How much time is needed to upgrade, maintain, and manage the underlying infrastructure as the site grows?

Time. How long will it take you to build the infrastructure out from development through production with proper environments and capacity?

Money. What are your up front capital costs?

Money. What are your on-going maintenance costs for infrastructure and operational support?

Money. How much operational support will you need to keep things running and scaling?

Money. What is the total cost of ownership of this community including the underlying infrastructure, software, and people?

Ask those questions and you will lay the groundwork for better understanding of your project and a better opportunity for success.

Note:  This is a Re-Post of my Blog entry for blog.solutionset.com

Twitter as a scalability case study

So, I just put my comment right there on his blog and it's probably easier to read it there in-line.  Here is a re-post of my response though since it was practically a blog post anyway.  It doesn't really stand totally on it's own so you should check out Nati's post first.

A wise article. Thank you. My thoughts after reading it some directly or tangentially related...

Slowly but surely people are catching on that what can generally be referred to as loosely coupled asynchronous capable systems architecture stacks and software architectures are critical to building truly scalable systems. Twitter even knew this early on and tried to adjust many times but for reasons unknown by me still made/makes some rather odd choices about their systems and software architecture; their mysql usage for example. Funny thing is, the body of work exists to build these sites already. People just keep focusing on the wrong things like language wars or putting the individual problems into overly broad problem domains or applying the wrong solutions all together.

Today's languages and frameworks can take some of the sting out of developing and scaling an application to a point but once an application moves beyond any significant traffic level problems inevitably arrive that lay bare all the bad choices that followed before. The people who really know how to address and fix those problems are few and far between. The people who know how to avoid those problems from the very beginning based on their experience are even more rare. The companies who have those rare people on staff and actually listen to them almost don't exist at all.

I think, in business, the definition of insanity is doing the same thing over and over and expecting a different result. If so, we're seeing industry insanity around the concepts of designing web based applications that will scale. People just keep making the same mistakes again and again.

I've been thinking about all this in terms of site traffic for the average implementation of a systems and software architecture underlying a web application that might grow on todays terms. I think the page views per month to description looks loosely like this.

0 - 100,000 page views = micro site
100,000 - 1,000,000 = small site - troubles start here
1,000,000 - 100,000,000 = large site - troubles magnify dramatically here. OMG Rewrite!
100,000,000 - 1 billion = very large site - nothing works that used to work because it just can't because your system died a tragic death

Each of these requires certain skills and knowledge to build to. But all of them can be handled if planned for well up front. It's commonly held that premature scaling is the root of all evil ( and cost over runs). It's just not true. Designing the systems and software architecture of your site to be a very large site doesn't require the CAPEX expense to do so any more. It's just an technology architecture problem and it requires a broad range of skills to solve.

Anyway, I've officially rambled on and I want ice cream so bye!

Thanks for reading ProductionScale!

Cloud Computing (Figure 1.0) is a commercial extension of computing resources like computation cycles and storage offered as a metered service similar to a physical public utility like electricity, water, natural gas, or telephone network. It enables a computing system to acquire or release computing resources on demand in a manner such that the loss of any one component of the system will not cause total system failure. Cloud computing also allows the deployment of software applications into an environment running the necessary technology stack for the purposes of development, staging, or production of a software application. It does all this in a way that minimizes the necessary interaction with the underlying layers of the technology stack. In this way cloud computing obfuscates much of the complexity that underlies Software as a Service (SaaS) or batch computing software applications. To explain better though, let's simplify that and break it down this definition to it's constituent parts.

More compactly stated, cloud computing is a commercial extension of utility computing that enables scalable, elastic, highly available deployment of software applications while minimizing the level of detailed interaction with the underlying technology stack itself.

Definitions of Necessary Terms to clarify and define cloud computing:

Utility Computing - The combination of computing resources as a metered service in a way similar to a physical public utility.

Scalability - The ability of a computing system to grow relatively easily in response to increased demand

Elasticity - The ability of a system to dynamically acquire or release compute resources on-demand

Highly Available - Systems designed such that the loss of any one component of a system will not result in system failure

Deployment - Placing your software application into a technology stack in a running environment for development, testing, or production

Software Application - An arrangement of programming code designed to achieve some specific purpose

Technology Stack - The Hardware and Software layers underlying a given software application. Figure 1.1.

Now we know what cloud computing is and that's great. But, I'll let you in on a little secret. For most people, it just doesn't matter! Remember, we're talking about cloud computing in the context of business. We're not talking about buzz words, technobabble, or hope. Let's get to the really important question. How can all this cloud computing alphabet soup help my business or my idea flourish? Starting again with that question and keeping it in mind at all times you and your technology team can make pragmatic decisions over time. You may have heard it before, and it bears repeating, that there is no place in day to day businesses for technology for technologies sake alone in most companies. Technology, cloud related or otherwise, has to make good business sense to make any sense at all.

To decide if cloud computing can help your idea or business flourish you have to decide what type of service provider you need. There are three distinct sub-areas of cloud computing. They are IaaS, PaaS, and SaaS. We will define and discuss these in turn.

Infrastructure as a Service (IaaS)

IaaS clouds make it very easy and affordable to provision resources such as servers, connections, storage, and related tools necessary to build an application environment from scratch on-demand. IaaS clouds are the underlying infrastructure of PaaS and SaaS clouds. A common characteristic of IaaS clouds is that they are more complex to work with but with that complexity comes a high degree of flexibility. So, these are generally lower level services in the grand scheme of things; not in a derogatory sense of course. You'll be dealing with virtual machines, operating systems, patches, and various other issues. You'll likely require some specialized help to make it all work well.

Some current examples of these types of services are (some of these are hybrids too but I put them where the most belong in my opinon).

Amazon Web Services - Extremely flexible Build your own w/ many add-ons
VMWare - Build your own
Elastra - Up an comer build and manage your own IaaS
3Tera - Sexy GUI based IaaS/PaaS building tools
Xen - Build your own
XCalibre - Very interesting and can do Linux or Windows
Nirvanix - All about cloud storage, very interesting subset similar to Amazon S3
EngineYard - Rails only Build your own
Joyent - Build your own on Solaris w/ Java/PHP/Rails/Python

The number one benefit of such services is rapid provisioning. You will not have to wait days, weeks, or months for new servers. In some cases, you can have them in minutes! In fact, it's so easy to provision that it's easier to just throw away "broken" servers and replace them with new instances in most cases. All the details of provisioning, racking, stacking, cabling, and more are completely abstracted away from you.

The developers of applications for such systems will often need to adjust things to accommodate for the IaaS cloud. It can also be somewhat difficult to move from one cloud to another in some cases. But, less so with IaaS clouds than the PaaS clouds we will discuss next.

Billing for these services is usually incremental by use and can get complex with tiered on-demand pricing that can be difficult to track in real time. Pricing is usually well defined but can be rather difficult to forecast in some cases. It can vary to the minute depending on levels of use, tiers of service, and other interesting combinations. Now, on to the second type of cloud computing model that's important in the context of this article.

Platform as a Service (PaaS)

PaaS clouds are designed, often within IaaS Clouds by experts to make the deployment and scalability of your application trivial and your costs incremental and reasonably predictable. Here are a few of the choice Application Stack Cloud Providers (ASCP) in this space today.

Mosso, PHP, .NET, Java, Rails, Python, other?
Google App Engine, Python
SalesForce - Proprietary
Morph - Ruby on Rails
Heroku - Ruby on Rails

There are more and more PaaS clouds sprouting up constantly and rapidly. The number one benefit of such a service is that for very little money, none in some cases, you can launch your application with little effort beyond having developing and and possibly some porting work if it's an existing application. Additionally, there will be a large degree of scalability built into your PaaS choice by design as it is a cloud as defined earlier in the article. Finally, you will not need to hire a professional systems administrator more than likely as they are part of the service itself. If you are trying to keep your operations staff lean this can be a useful path to follow assuming your application will capitulate.

The number one down side of choosing an PaaS Cloud provider is that all such services come with various restrictions or trade-offs that may be a non-starter for your project. This is especially true of you already have a pre-existing application that might need to be ported to the PaaS solution you choose. You will need to plan on some porting development time costs and it might not be trivial. For example, a particular PHP extension, rails gem, or operating system tool may be unavailable that your application needs and you'll have to code around these types of issues as your ability to add this to an Application Platform Cloud will be limited unless it's a custom PaaS.

Billing for these services varies. It can be by the hour, request, CPU cycle, or other creative ways. Some even help you do pass through billing for your customers; like Mosso. But, the defining factor in pricing of Application Platform Clouds is that they generally strive to be robust, simple, and easy to load your application into when you are ready.

Software as a Service (SaaS)

Software as a Service has been around for a while now and actually precedes the newer term Cloud Computing. What's interesting, and the reason to include SaaS in this article is that Cloud Computing is breathing ever more life into the SaaS model by reducing the costs associated with producing a SaaS application. A couple of well known examples of SaaS are GMail or Salesforce.

SaaS is not really the ultimate goal of Cloud Computing per say but it is an important, relevant, and right now step along the evolution of compute resource management and allocation.

In summary, understanding cloud computing requires some base knowledge and historical review to know where it came from. This will enlighten people that it's not exactly new but that there is a new excitement now due to technology and business convergence. Once you have that base, which you now do, explicitly learning to use the cloud is the next step. Then, you can almost certainly implement your ideas faster, cheaper, and more profitably than people have ever been able to do before. It's a very exciting time to be in computing and in fact, to be on this planet. Welcome to the clouds!

They had 25,000 members on Monday, 50,000 on Tuesday, and 250,000 on Thursday. Their EC2 usage grew as well. For the last month or so they had been using between 50 and 100 instances. On Tuesday their usage peaked at around 400, Wednesday it was 900, and then 3400 instances as of Friday morning.

While I am very impressed by the scale and scalability I am also very interested to know if they can actual monetize at a level that provides a good return on a $300 to $500 / hr investment + bandwidth.  They built in elasticity so it can shrink as well as grow but still, the average number of AMI's this month is going to be up there!  Also, I wondered what happens to Animoto when they max out the the credit card on their AWS account?

Sources:

Animoto Blog: http://blog.animoto.com/
Amazon Blog: http://aws.typepad.com/aws/2008/04/animoto---scali.html

Scalr describes itself as a fully redundant, self-curing, self-hosting EC2 environment.  This is reminiscent of something called WeoCEO that wrote about briefly a while back.

I twittered today that it must definitely be the year of the cloud and not the year of the Rat.  I think I might be right.  The cloud news releases are pretty much non-stop these days and the rate of innovation is accelerating.  Exciting times and it's about time.

Since it's late and I'm not in a summary typing kind of mood here is the text from the Scalr page as printed and with full credit to the URL just below so you can go see for yourself.

Scalr is a fully redundant, self-curing and self-scaling hosting environment utilizing Amazon's EC2.

It allows you to create server farms through a web-based interface using prebuilt AMI's for load balancers (pound or nginx), app servers (apache, others), databases (mysql master-slave, others), and a generic AMI to build on top of.

The health of the farm is continuously monitored and maintained. When the Load Average on a type of node goes above a configurable threshold a new node is inserted into the farm to spread the load and the cluster is reconfigured. When a node crashes a new machine of that type is inserted into the farm to replace it.

4 AMI's are provided for load balancers, mysql databases, application servers, and a generic base image to customize. Scalr allows you to further customize each image, bundle the image and use that for future nodes that are inserted into the farm. You can make changes to one machine and use that for a specific type of node. New machines of this type will be brought online to meet current levels and the old machines are terminated one by one.

The project is still very young, but we're hoping that by open sourcing it the AWS development community can turn this into a robust hosting platform and give users an alternative to the current fee based services available.

 SOURCE:  http://code.google.com/p/scalr/

 I need a fully staffed lab to keep up with loading, deploying, testing, and applying all the things coming onto the market lately.  If anyone is ready to volunteer to fund the Cloud Computing Innovation and Business Application Labs ^TM (CCIBAL) just let me know and we'll get right on it.

Resources:

Scaling Out MySQL by Nati Shalom
http://natishalom.typepad.com/nati_shaloms_blog/2008/03/scaling-out-mys.html

Vertica
http://www.vertica.com/

Memcached
http://www.productionscale.com/display/Search?searchQuery=memcached&moduleId=1481658
http://www.danga.com/memcached/

Oracle Coherence
http://www.oracle.com/technology/products/coherence/index.html

My initial tests show that the related content links provided are quite relevant which is impressive. 

-Kent

This is a recurring theme that I hear time and time again from well researched web systems implementers.  In my experience something like 80% or more, of the problems companies will have operationally when trying to scale a system efficiently will trace back to the work and care taken by the developers, software, and systems architects very early on in the projects life cycle.  This dramatically effects TCO, Total Cost of Ownership, of a system over time.

I'm not saying scale early necessarily because you don't have to over engineer things like software and infrastructure up front.   But, you should be thinking about scalability early in the overall design and systems implementation or you will suffer later.  It's just good sense and I think the article written by Kendall Miller and the subsequent comments to the article echoes this as well.

If you like this article and would like to read more by this author you can subscribe to this sites RSS feed with your favorite reader and/or follow the author's Twitter feed for related posts and information.

Resources: 

Original Article:  http://kendall.srellim.org/development/technology-is-not-scalable

Nirvanix has created an API accessiblem globally distributed, highly available storage area network.  When properly integrated in any of a variety of ways to an application or work flow might be very useful and cost effective indeed.  Some of the uses they list on their site are CDN Origin Storage, Digital Lockers, Online Archiving, Backups, extending storage services to managed services and SaaS, etc.  However, the problem I ran into when looking at the service wasn't the service itself, but the pricing. 

I looked at their pricing and projecting costs month over month at scale to use that service with any degree of certainty would be difficult at the higher levels of complexity.  I'd have to estimate and add up bandwidth, media, primary storage itself, geographic co-location instances, level of extended service, methods of access, support, etc.  You get the idea. It looks like they've built a service reasonably resembling a service but then gone far from a utility pricing model which makes it more difficult to cost than necessary.  I just want to use the service and have the utility pricing match the utility service.

Pricing should not be this complicated.  Relative to Amazon Web Services for projects that use several AWS services this one isn't so bad.  But, both fail in my opinion to create a reasonable utility pricing model to go along with their utility services.  Ideally, I'd prefer to see a more simple tiered rate structure based on some overall system transaction/unit of work usage metric (SPU? Storage Processing Unit?).  I would tend to expect the rate to decline as my usage volume goes up to a point then probably plateau or even go up again at very high levels.  Within that model I'd be able to use any of the services provided if I am paying or a reasonable subset if I'm using a starter or free level of service for a while.  The point is that in just the way the cloud abstracts away some of the complexities of infrastructure it should also strive to abstract away the complexities of infrastructure pricing and offer an easily understandable model.

If you like this article you can subscribe to the sites RSS feed with your favorite reader and/or even follow the author's Twitter feed for related posts. 

Resources:

Nirvanix Pricing: http://www.nirvanix.com/gettingStarted.aspx

AWS Pricing: http://www.amazon.com/aws (click on various links in the infrastructure services side bar)

3Tera: http://www.3tera.com

"There is a time and place for hardware upgrades. The problem is that most people find hardware easier and just aren’t aware of how expensive their app logic really is. I find tuning easier and more cost effective."

The article explains an example of a database query and data model that brings a 10 node Oracle RAC cluster on a SAN to it's knees.  After properly tuning and indexing the need for the 10 node cluster disappeared and it was replaced with a mere two nodes.  I'd say that saved someone a few hundred thousand dollars a year.

Resources:

Good Database Design is Mightier than Hardware

A quote that got my attention, "When continuous improvement is an integral part of daily work, the need for catchy slogans to inspire the workforce and heroic problem solving greatly diminishes."

One example of that, in the context of launching web sites, that I often tell colleagues is that if we do this site launch right, then this launch will be the most boring web site launch you've ever experienced.  Oh how I love a boring site launch with no emergencies, late hours, heroic coding marathons, or 72 hour system rebuilds followed by dozens and dozens of trouble tickets over the next 4 weeks while living (so they say) on RockStar and Energy Bars.  Yes, I love a boring launch.

Anyway, if you are into Technology Operations, Development, or managing one or both this article is worth a few minutes of your time to read in my opinion.

And lastly, to quote liberally, because it touches a special place in my heart,

The notion that business strategy and IT strategy should be aligned and, therefore, that business users should be involved in the design of enterprise systems has been widely accepted. However, doing this has proven fiendishly difficult, for several reasons. For one thing, IT leaders struggle to truly understand the business context. What’s more, business leaders do not invest the time required to appreciate the power and the challenges of technology and tend to treat the IT staff as second-class service providers. Even when the two groups meet to discuss a project, those occasions tend to be isolated, onetime events, rather than part of an ongoing discussion. Like it or not, however, information systems are an integral part of business strategy in almost all industries today. If business leaders view the IT staff as an ancillary player rather than a partner, then knowledge transfer between the two groups will suffer, resulting in missed opportunities and suboptimal performance.

So, enjoy the article and let me know what you think.

Resources:

The Article Link