Automating SaaS Operations

"The first rule of any technology used in a business is that automation applied to an efficient operation will magnify the efficiency. The second is that automation applied to an inefficient operation will magnify the inefficiency" - Bill Gates

This is an article which I posted at the Nolio blog site as a guest blogger back in 2009. Not only is is relevant today, but perhaps even more so...

The Next Killer App
If I had a great idea for the next killer app (I have, actually) and if I had unlimited funds (I don’t, actually) I would have built the software as an on-demand offering.

I would have spent half my funds on building the operational support systems – provisioning, billing, retention policy, self-service, report generator, etc. The other half would be invested in building instrumentation, redundancy, automation, integration, application level monitoring, silent upgrades, customer notifications, and so on.

The rest of the money (you may wonder about my math, but hey, I’ve got unlimited funds) would go towards building the actual application.

Most SaaS vendors out there (and they are growing fast) have chosen the predictable path of building the application first, and worrying about serviceability later. This is the fastest way of getting to market with low costs. The next step is choosing some viable hosting solution and off we go, offering the world our ever better CRM.

Growth
Many months and dozens of customers later, reality hits with all the issues of servicing the software, rapid growth and dealing with labor intensive tasks that are the humdrum of daily life in a SaaS operation. Provisioning/de-provisioning, configuration changes, customized reports, and the most dreaded – upgrades, task the team as a whole, especially when the product is successful and the number of customers is growing daily.

It is not that SaaS executives, architects and engineers are lacking in any way. On the contrary, they are mostly smart, inventive, and creative and have a deep understanding of their customers’ needs in the specific domain. The problem is that they are product people, not service people. Practically none of them come from IT and cannot envision the life of a service operations engineer.

At this point, automation becomes crucial to the survival of the business.

Whether it is built into the next version (many architectures make this quite difficult) or done externally, automation is needed to reduce costs, physical labor, frustration and mainly, error-prone manual procedures. Repeatability, which is a derivative of automation, is also crucial.

Automation is needed across the board. Be it in setting up a new server, or building a new application instance. It could be a manual procedure regarding provisioning of application resources, or building a seamless upgrade procedure.

Outages happen. How quickly can you recover from a service disruption and ensure that the recovery does not create it own problems? Automation not only provides the routines for quick recovery, but instills a discipline of thinking out the necessary steps, discovering dependencies and planning ahead. An added benefit of automation is that it documents the process so you can go back and review the best and worst of your procedures.

In my next post I will take a closer look at the SaaS Upgrade Nightmare.

2 x E-cube = S-cube - Simple math for SaaS Scalability Success

"The circumstances of human society are too complicated to be submitted to the rigor of mathematical calculation" (Marquis De Custine)

Recently I posted a discussion on the link between success with SaaS and scalability and how, therefore, the Service Operations needs to be geared to handle scale and deal with fast growth.

On a recent consultation engagement, I gave a presentation to the board members of a SaaS company that decided to penetrate the SMB, following their hardships in securing big deals in the enterprise market.

As part of the thought process and brainstorming session I came up with a marketing catchphrase to make a point. At the time I did not think much about it, but as I was working on the board presentation, I realized that there it was much deeper than I had originally thought.

I would like to share this with you.

The First Cube

What I drew on the white-board was roughly this artistic creation:

Announcing: “E-cube is the winning formula!”

• Easy to Buy
• Easy to Implement
• Easy to Use

Easy to Buy - Their product is very easy to implement, and easy to on-board a new organization, (although it is a complex product). There is a technical issue that involves an architectural change, so that became part of the plan.

Easy to Use - the product has a great user interface and intuitive flow. They need to add tutorial videos.

Easy to Buy – That was lacking and that was what we decided to focus our efforts on. It included a lead generation program, a no-touch free trial and an improved landing site.

The Second Cube

As I was working on what it would take from the company’s end to deliver scalability, I realized there was another E-cube involved:

• Easy to Sell
• Easy to Scale
• Easy to Maintain

It is all nice and well if you can bring thousands of leads to your site with the E-cube formula, but if you cannot convert these leads into paying customers and then give them the best service on a controlled budget, you will have not achieved your scalability goals.

So, Easy to Sell means - a simple pricing model and a top-notch insides-sales team (not easy to come by though) backed up by funnel management software. 

Easy to Scale means - a solid, configurable, multi-tenant architecture, self service features and automated procedures.

Easy to Maintain means – a full featured Operations Support System, Service Operations practices and the discipline to enforce them.

So with two E-cube guidelines one can achieve SaaS Scalability Success.

Ergo: 2 x E³ = S³

Q.E.D.

SaaS, Scalability and the Three Little Pigs

“We have the wolf by the ears, and we can neither hold him, nor safely let him go” - Thomas Jefferson

A few days ago I gave a talk at a SaaS Business Challenges conference and I would like to share the main theme with my readers.

It’s the Service, Dummy!

When SaaS burst upon the scene about ten years ago, the first consumers were of two types: Those that used on-demand as an ideology, having foreseen the cloud revolution early on, and those that had no choice, because the other option was an expensive, time consuming and complex solution.

Back then, there were few SaaS application or service choices, so customers had to be very forgiving about the service levels and were willing to put up with reduced functionality, outages and low response times. SLAs, if they even existed, were non- binding and lacked both depth and breadth.

These days, however, when there are dozens of SaaS applications for every need, the differentiator is no longer the functionality. Most SaaS applications offer a similar set of capabilities, and as applications change on a bi-weekly or monthly basis, features are added on an on-going basis.

Even if one comes up with a revolutionary solution and provides the only SaaS offering of its kind, it is safe to say that within a year, three new SaaS companies will offer the same, or an improved set of capabilities.

So what differentiates one SaaS offering from another? It’s the Service, dummy!

Features don’t make a loyal customer – outstanding service does.

Scalability

There aren’t many SaaS companies out there that service a few dozen Fortune 200 companies while keeping the profitability high. The general rule is that margins are small and that profitability is achieved through hundreds or thousands of customers.

Giving great service to 20 or 30 customers is a no-brainer. Just throw more bodies at the problem and you will achieve a highly satisfied customer base.

But what happens when these numbers multiply rapidly? You’ll soon find out that what worked for a few dozens might collapse at the next order of magnitude.

When I approach SaaS companies at the stage when they already have a growing customer base and warn them about the perils of scaling up, the usual response I get is:

“I wish I will have to deal with that problem”, meaning that they would love to have 200 hundred customers that cause strain on the infrastructure and operations and deal with those ‘good’ problems then. It is only human to postpone these issues when they are not burning your behind.

To those CEOs I would say: “Pack your stuff, return the money to your investors and go look for another job elsewhere”.

Your investors did not give you their hard earned dollars for a proof of concept.

They invested in you because they believed that you will bring in thousands of customers.

My best clients are SaaS companies that come to me when they are in pain. They start loosing customers because they did not build an operation capable of handling the scale which they had wished for.

As I have stated time and again – SaaS companies usually consist of outstanding, creative developers that build great technology, but they don’t have the IT and/or operational experience. They lack the know-how and especially the methodology for building a successful, scalable operation.

Building Operations for Scalability

I have written much about these issues and they can all be found in previous blog postings.

Suffice to say that the setup you need to build for a scalable service operations includes:

· Methodology – the framework of practices, templates, workflows and tools

· Operations Support Systems – everything else your engineering team left out of the product.

· Executive buy-in and awareness -define the metrics, and provide tools to capture and analyze those metrics.

Can a Huff and Puff Blow Your House Down?

So what has all this to do with the Three Little Pigs?

The first little pig built a house of straw, since he didn't have time to invest in operational infrastructure and wanted to make a quick exit. He ended his career as wolf poop.

The second little pig invested in a better infrastructure but did not pay attention to the practices and processes. He lasted longer in his wooden structure but was huffed and puffed and blown away by the competition.

The third little pig took his time and invested executive attention in doing it right from the start. He lived happily ever after.

So even if you cannot afford a brick house from day one, you should have the blueprints ready and the determination to add the bricks when they become available.

By the time the Scalability Wolf arrives, you should have a sturdy enough structure and react quickly to threats, to survive and prosper.

Reducing SaaS Operational Costs (II)

In the last post Reducing SaaS Operational Costs I discussed how utilizing two strategies, Automate and Delegate can enable economies of scale, so that the cost of adding new customers is marginal.

Following is a review of some of the areas in which these strategies will prove to be cost effective over the long run, and reduce the probability of the operation collapsing under pressure.

Professional Services
Remember - “customization – OUT; configuration - IN”. The software should be designed to allow maximum configuration without altering the code. This includes branding the software and, more importantly, allowing the customer to define names and custom fields to various entities of the product.
Build the architecture to support Web Service for painless integration with in-house systems, then Let your customers do the integration.
Train the trainer. Find the champion of your solution at the customer, and grow her. Help her promote the product and build a well trained team that will take the burden off your shoulders, including some of the sales and marketing efforts.

Operations
Provisioning should be a seamless process with no (or minimal) human involvement. A customer should be able to sign up to the service and the resources should be provided automatically. A customer-centric application is essential and administration should be delegated to the customer with multiple-hierarchy supported.
Client side components should be downloadable with self-installation. If the application supports an on-premise agent, it is essential that the backend application is version-backward compatible, to avoid the upgrade nightmare.
In order to ensure that the application is up, available and performing to the SLA requirements automation should be used to the maximum. This begins with monitoring the network, resources and the application. A dashboard should depict the status of all resources with the ability to drilldown to sub systems and components. Alerts should be used and crossed referenced in order not to create false alarms. Use automation scripts for self recovery whenever possible and keep extensive logging for postmortem of downtimes, because they are going to happen no matter how good you are.
And while we’re at it, make sure that R&D provides well-tested, easy-to-use data migration tools for version upgrades. Utilize change management and patch management tools and processes to lower costly human errors.
Maximize self help systems – FAQs & online knowledge base - to reduce customer calls.
Add new services seamlessly – build the capability into the application management systems.

Sales
Naturally, a SaaS offering reduces greatly the need for direct marketing and sales. Cyber sales will take most of the burden and referrals will become an important sales tool. Design a good interactive demo site and allow customers to test the application for a limited time as a proof of concept. This process should not involve human intervention on the ISV side.
If your service is becoming a commodity, allow for self registration and on-line payments. This will greatly reduce the sales cycle and the number of people involved.
Automate the metering and thus, the billing process.

Customer Support
Remember that a SaaS customer service rep replaces some of the functions of the customer organization's HelpDesk. That means that your reps will be getting a LOT more traffic than your average packaged software company. And when bad things happen, everybody will be screaming bloody murder and bog down your communication lines. Therefore have an updated status page where customers can view the responsiveness and availability of your production system(s).
A self service portal will go a long way to reduce the call volume. A rich and well maintained knowledge base will do that as well (check out all the Freemium sites - they will not invest in a non-paying customer, yet they want happy customers as bad as you do). User forums are also a great way to harness the goodwill and knowledge of the community to save on training and problem resolution resources.

In conclusion
The OSS (operations support system) should at least support provisioning, access, self admin, metering, and report tools. There are a number of available platforms for purchase that provide the OSS. They may be a good solution for your operation.

I have touched upon many areas and providing for full automation and the technology enabling delegation may not be available from day one. It is therefore imperative for the Ops team to work closely with product marketing, R&D and QA and to participate in product discussions and development planning.

Reducing SaaS Operational Costs

"Say, Tom, let me whitewash a little." (Tom Sawyer, Mark Twain)

Remember that quote of Ben from the timeless Tom Sawyer, who gladly gave up his apple to have the opportunity to do Tom’s work?

Cute, but how does that relate to reducing operational costs in a SaaS house? Well, at least fifty percent of the magic formula “Automate and Delegate”

In my numerous interactions with pure SaaS startups as well as with established ISVs transitioning to on-demand, I have encountered time and again the lack of planning for a robust Operations Support System (OSS).
Whenever I bring up the potential hazard of non-scalability I hear the same response “we wish that were our problem” or “we’ll deal with it when that becomes an issue”, secretly wishing that it will become an issue fast enough.
The logic being - let’s make sure our software works as a service and that we get traction. When we’ll have more customers than we can currently support efficiently that would be a success milestone.

I’m sorry to break the news but when that happens you might find yourself with a catastrophe waiting to happen (see Chronicle of a Death Foretold).

The profitable SaaS operation utilizes economies of scale when the cost of deploying, say, 100 customers is just marginally more expensive than a single customer. In an ideal world, the hardware and software infrastructures costs should remain the same; bandwidth utilization should not change dramatically; storage needs will expand more or less proportionally to the number of new customers, while the support staff should remain steady and perhaps increase slightly.

So what costs are involved with the growing SaaS business? There is the hardware, bandwidth, software licenses (e.g. databases), customer care, and the inevitable marketing and sales. Human resources are still the greatest expenditure for a software vendor.

Careful planning – ‘doing things right the first time around’ can make the difference between a profitable operation and one that is bleeding the company.

In order to maximize your margins, you must reduce human intervention in every aspect possible. There are two strategies, that when applied in concert, can do just that. They are Automate and Delegate.
Automate means let technology do whatever task that would otherwise require manual work.
Delegate means enable your customer to do whatever task that would otherwise require your team to handle. Remember Tom Sawyer?
Obviously these stategies will necessitate a technological infrastructure to support it.
By carefully planning your application and operational environment you may achieve a high level of automation and delegation.

In my next post, I will review the areas in which these principles can play a major role.