“'Tis not enough to help the feeble up, but to support them after” - William Shakespeare
Every SaaS company needs to deal with numerous functions that are not necessarily part of the technological stack that originally came with the application, namely the Operations Support Systems.
If a SaaS start-up had unlimited time and funds to plan and build the perfect solution, they would probably all be in the Caribbean islands doing what people with unlimited time and funds do.
Just Do It!
Because of the nature of monetizing SaaS, companies try to get to market as soon as possible, getting subscriptions fees streaming in. Sometimes, they even launch with a half baked solution that will provide added value to the customers at the expense of future operational headaches.
The logic behind this is that dealing with a scalability problem, is a good thing. In other words - who doesn’t want to reach the stage when too many customers are taxing the team? We’ll deal with that when it becomes a problem.
These days, numerous PaaS offerings or SaaS frameworks offer built-in operational support systems functionality, but many of the necessary features are not supported.
Most of the SaaS application in the market were not built with these frameworks for various reasons. The most prevalent are that they were not available a few years ago, and that engineers have a tendency to build everything from scratch, or using frameworks (e.g. LAMP, Java, .Net, RoR) that they are familiar with.
When the typical SaaS service is launched, it lacks functionality that would support the scalability of the service operation. It is left to the Service Operations team to deal with all the 'maturity' functionality that the product lacks.
Let us examine some of the Operations Support Systems functions that are typically not handled by the application.
On-boarding new customers
One would expect most SaaS systems to have automatic provisioning. While that is true in many cases, a lot of the systems allow the customer to define users, but creating a new customer entity is left to the Support or Operations team. That may require generating a new database or schema, or setting up storage, etc. If the company is adding one customer a week, it may be manageable, but at a higher rate this is a taxing job and error prone.
De-provisioning
While it is expected that some level of automatic provisioning is provided with the product, very few SaaS applications provide a simple (never mind automatic) mechanism for removing existing customers. Very few SaaS developers design with of the prospect of loosing a customer in mind. Beyond the task of removing dependencies from the database, there are issues of releasing resources, and removing customizations.
Billing
Such a basic function, (one would think, for a company that lives or dies by subscriptions) is typically lacking from most SaaS infrastructures. While the situation is improving dramatically with the advent of PaaS and SaaS development frameworks, many systems still start out with excel sheets and a lot of manual work. Customer data must be extracted by some ad-hoc solution form the application database or is duplicated in the CRM, which causes endless synchronization errors. Ad-hoc solutions are implemented (either in-house or SaaS billing solutions) as the billing becomes ever more complex but until that occurs the brunt of the work falls upon the Operations team.
Metering is a whole new layer of complexity, if the billing is more complex than a fixed amount per seat per month.
Retention Policy
Some SaaS companies plan ahead for resource consumption by their customers, bur many start dealing with the issue only when they begin running out of storage, or when their storage costs are starting to hurt. Most customers want to retain their data forever on the provider’s disks, but that is impossible. So a retention policy must be defined and followed (such as delete files that are older than X months or larger than Y gigabytes).
The problem is that the application does not support that, so manual work is requried or ad-hoc solutions have to be built around the product.
Failover and Backup
Automatic recovery and failover are rarely built into the initial solution and are usually managed by the Operations team via building complex solutions with networking boxes.
This is true for backup and recovery mechanisms as well. The Operations team frequently has to build mechanisms around the production to support backup and recovery and most often they require much manual labor.
Application Monitoring
Well designed software is packed with instrumentation that is easy to turn on and off and easy to monitor and interpret. Not all SaaS systems have that built-in capability and the Operations team has to create an application-specific, monitoring infrastructure that can detect and react quickly to service degradation.
Seamless Upgrade
Upgrades are the recurring nightmare of any Operations team, especially on applications that have a demanding uptime SLA. Few SaaS systems are designed with that goal in mind. It usually takes a level of maturity and a sizeable customer base to get Engineering to consider revising the code to allow some sub-systems a no-downtime upgrade.
The problem is that is usually requires major revisions of the code if the system was not designed a-priori to handle a silent upgrade.
SLA Management
Whether you are using automated SLM or not (chances are you are not), you need to take into consideration the various aspects of your service that need to be metered, tracked and compared against a set of Service Level Objectives.
End User Broadcasting
Sometimes it is necessary to communicate with all of you users, or a sub group of them in real time. What a better option is there that to pop up a message on the end-user’s browser that you can compose on the spot or pull from a list of canned messages? The operations team need an integrated solution with the product to be able to do that.
Operations Console
And to tie it all up, a separate application that controls all the operational aspects of the service is needed. Included should be: provisioning/de-provisioning customers, password management, real-time login view of current users, real-time view of application usage, customer communication console, production environment control, etc.
There are many other operational features that are typically not found in a SaaS application as it leaves the factory floor. Just to mention a few: Integration, Reporting engine and reporting Database, Security, Scale-up and Scale-out capabilities, Sandbox, Status Page.
Development and Product management Experience Required
The VP Operations (or whatever title the job carries) is required to be a product manager of sorts, and a background in software development is almost a must. The Operations manager should be highly involved in the product roadmap and insist on having a say in defining future releases. There will be a contention between investing in Serviceability versus Functionality. Since the paying customers require more functionality, it is usually an uphill battle to gain service upgrades to the product.
While building an organic set of solutions into the product may practically take years, Operations cannot continue to throw bodies at solving scalability and downtime issues. So beyond influencing the product group on the direction in which the application should be developed, the Operations manager needs to build a set of tools addressing the Operations Support Systems needs as stated above. One option is to nurture a relationship with VP Engineering and get resources from her group to build well defined solutions that could each be completed in a couple of weeks of work. This is especially true in early stage companies where the Operations team is small and the engineers take the brunt of many of the operations’ tasks. The VP Engineering would appreciate the need as members of her team are feeling the pain as well.
In a more established company, the VP Operations must make sure that there are coding/scripting capabilities in the team, so simple projects and tools could be developed within the team, with minimal aid from the Engineering group.
