ProductionScale

Designing, launching, and integrating community platforms that leverage trends in modern computing such as Cloud Computing and Virtualization well can be a tremendous strategic benefit for a company. However, much of those benefits can be quickly erased by operational and development issues surrounding doing that successfully are often marginalized. This has fiscal, operational, and morale impacts over time.

ROI no matter who you might calculate it will include the total actual costs (TCO - Total Costs of Ownership) of running or renting the platforms that community sites run upon.

Will your system stand up to Digg/Slashdot/TechCrunch/Reddit or exponential user base growth? Does it have a clear scalability path? These are questions that development, operations, and business staff need to discuss periodically throughout any community development project and long after when the project is live.

I saw a case study recently about a facebook community app that went from zero to a billion page views per MONTH in just a short time. I’m sure they thought about scalability and operations upfront at least a little or they would not still be in business. Do you know what you would do if your community caught fire like that? Would you get a dreaded “Site Not Available” message at your time of greatest success?

If you plan your communities technology platform correctly from the start you have a much better chance of succeeding and fully leveraging your investments in your businesses technology over time.

Here are some things you might want to think about and questions to ask. They pretty much all break down to how you spend time and money. Since time and money are often inexorably related.

Time. How much time is needed to upgrade, maintain, and manage the underlying infrastructure as the site grows?

Time. How long will it take you to build the infrastructure out from development through production with proper environments and capacity?

Money. What are your up front capital costs?

Money. What are your on-going maintenance costs for infrastructure and operational support?

Money. How much operational support will you need to keep things running and scaling?

Money. What is the total cost of ownership of this community including the underlying infrastructure, software, and people?

Ask those questions and you will lay the groundwork for better understanding of your project and a better opportunity for success.

Note:  This is a Re-Post of my Blog entry for blog.solutionset.com

If, like pretty much everyone else, you are thinking of launching your web site/service in the "cloud" on Amazon Web Services?  One of the things people don't often realize is that they will not be able to successfully send email from an EC2 AMI's.  Why?  You need a few things these days to be able to successfully send mail from a server.  In most cases you need reverse DNS, SPF record, and a static IP address.  At AWS you do not have a static IP or a reverse DNS entry.  That is a non-starter.  You will need to use an external to AWS relay server or service.  Following are 3 services you might want to review depending on your needs.

http://www.authsmtp.com
Subscription service by email volume in numbers of emails, tiered pricing.

http://www.easydns.com
Email forwarding by subscription.

http://www.dnsmadeeasy.com
Email forwarding by subscription.

http://www.rackspace.com good location to build your own SMTP server if you are so inclined and install either open source (postfix) or commercial software.  That choice will likely come down to your needs for tracking the data your emails generate.  It's difficult, but not impossible, to mirror the functionality of a product like Port25's PowerMTA (http://www.powermta.com/) when it comes to tracking and reporting on email sending and activity.

How to choose?  It's all about business needs, volume, and what technical skills you have available to you to manage your services.  So, just take the time to look at your options.

I am a few pages away from finishing the book Release It! by Michael Nygard.  I honestly can't really say enough to recommend this book to anyone who actually does care about creating web applications that work well and fulfill their purpose.  The book is a bit Java centric but that is not an issue.  The concepts, examples, and stories that often made me laugh remembering similar situations just lay it all out there.  The patterns/anti-patterns idea is an interesting way to organize and present the ideas that works well.  Most of all, this is a well written and readable yet technical book.  A well written book is a breath of fresh air and I see it often when reading the books in the Pragmatic Programmers line of products.

Again, I can't recommend this book enough for anyone responsible for all or part of building a software application.  I think this is especially true of people in a position of technical leadership and decision making.

At work we've been having some good fun w/ monitoring packages as work.  Recently testing and then installing Monit and Munin on a variety of servers for various reasons.  If you need to collect data on the performance of a server(munin) and have some degree of alerting and automatic watching and handling of processes (monit) check them out.  They are both very flexible and useful tools that are easier to install that most others in their class.

In a post at oreilley.net Tim O'Reilly give a nice tip of the hat to the folks that slave away trying to shared knowledge about building web sites that are available, scalable, reliable, and performant.  He mentions a new book that just made their "internet best seller" list recently.  The book was "High Performance Web Sites" by Steve Souder.
 
Since I read the book in its pre-release PDF form I'll do a little mini-review here.  I can pretty safely say that if you are a programmer or administrator embroiled in building web sites that you would like to actually work if they get popular you should read this book.  I'd also recommend it for managers of development teams.  The way the book is broken down is easily digestible and good for getting ideas across to your team.  In summary, buy it and read it.  It's not revolutionary in most cases but it's just good sense all around and it's the stuff you just scratch your head about because you figured your devs and admins just knew this stuff anyway.  But, they just might not.

Oh, there is another link out in the article to the "High Performance MySQL" book.  Don't buy that one.  It's in the process of being re-written right now and is SERIOUSLY in need of such a thing.  It's a great book, but the content is just a bit to dated for my liking.  Here's the blog where you can follow the re-writing of that book.  They are even taking feedback to incorporate.  Pretty cool and shows you publishing really has changed a quite a bit in the last few years.  Here are some of the recent posts to that blog about the writing of the new edition.

http://www.xaprb.com/blog/2007/10/07/high-performance-mysql-second-edition-query-performance-optimization/
http://www.xaprb.com/blog/2007/10/05/high-performance-mysql-second-edition-advanced-sql-functionality/
http://www.xaprb.com/blog/2007/10/02/progress-on-high-performance-mysql-backup-and-recovery-chapter/

I got a chuckle from an email I received recently for an invitation to a Webinar tomorrow.  I would like to go to the webinar so this post is no ill reflection on the actual event itself.  Here is the excerpt from the email.

"Getting 30-year old OLTP database technology to handle today's data warehouse and business intelligence (BI) workloads, is like trying to put a square peg in a round hole. As a result, DBAs are required to force databases to perform unnatural acts at high cost, or replace them with expensive and proprietary data warehouse hardware."

I believe that I can say with great confidence that I don't want to see most of the DBA's I've ever known "being forced to perform unnatural acts at high cost..."

Thanks for the chuckle Vertica.  I find the promise of the Vertica technology interesting although I'm less interested in it for data warehousing than I am for large scale web application possibilities.   If you would like to follow up on the webinar, here is the information.  

Good Things Come in Small Packages
The advantages of compression in column databases

What if your database could store 20TB of user data in just 2TB of disk space? Not only would that save you hundreds of thousands of dollars on storage hardware, it can help boost database query performance by over 50x.

Attend this complimentary Webinar to hear MIT Associate Professor Dr. Sam Madden describe how column-oriented relational databases are able to aggressively compress data and as a result, revolutionize the performance and economics of data warehouses and data marts, so that more people can query more data faster and more affordably.

Date: Wednesday, October 10
Time: 12:00PM Eastern time, 9AM Pacific
Duration: 1 hour
Featured presenter: Dr. Sam Madden, MIT Computer Science Professor and Vertica Advisor

Agenda:

Column-oriented RDBMS architecture overview
How the columnar Vertica Database is able to:
Compress data by 90%
Query it in compressed form
Use compression to enable other DBMS break-throughs such as support for grid computing and ad-hoc query-handling
Benefits of compression:
50x-200x faster query performance
70% lower storage costs and greater resource utilization
Handle ad-hoc queries without special tuning
Customer benchmarks: compression in practice in Call Detail Record (CDR) databases
Live demonstration of Vertica compression and query performance
Getting 30-year old OLTP database technology to handle today's data warehouse and business intelligence (BI) workloads, is like trying to put a square peg in a round hole. As a result, DBAs are required to force databases to perform unnatural acts at high cost, or replace them with expensive and proprietary data warehouse hardware.

Register Today!