Organizational Culture and Company DNA – What Makes a Successful SaaS Company.

“No people come into possession of a culture without having paid a heavy price for it” (James A. Baldwin)

It was always clear to me that success with SaaS was not about technology, but about execution. This week I got a clear reminder.

The SaaS CEO Forum
For the past few months I have been running a SaaS CEO Forum that meets every six weeks or so, each time hosted at a different SaaS company, by a member of the Forum . The forum consists of a select group of successful SaaS companies, that have been selling their service for a number of years and are dealing with issues such as growth, operations, sales, marketing and customer satisfaction. Every meeting has a theme such as running a fabulous inside sales teams, SaaS Service Operations, knowledge-as-a-service, etc.. Yesterday the forum was hosted by Avinoam Nowogrodski, founder and CEO of Clarizen, a fast growing, market leader on collaborative project management. Beyond the very interesting review of the company’s clockwork marketing and sales operation, Avinoam gave a presentation on what makes a SaaS company successful.

Successful SaaS Company
No one can argue with the success of Clarizen, having grown 400% year over year, with an ever-growing community of happy customers, so it was worthwhile listening to Avinoam’s credo.

Clarizen’s CEO was talking about managing a company where execution is paramount and where Customer Success always comes first. He has been careful selecting an executive team that he regards as ‘A’ players and nurturing a culture of Respect, Modesty, Openness, and Accountability.

Among the factors Avinoam mentioned was “checking your ego at the door”, willingness to take risks, and acceptance of mistakes as part of the ever changing environment and conditions. Delegating Authority, Hands-on in your domain, Transparency and above all – Measure, Measure, Measure every aspect of the company’s operation; sales, marketing, conversion between each stage of the pipeline, responsiveness, costs.

The SaaS Angle – Fast Forward
While I agree wholeheartedly with all the above criteria being critical for a successful company, I asked for the SaaS angle. The answer was obvious even before I finished asking the question: “Pace”. In a SaaS company everything is fast-forwarded. The cycles in almost every aspect shorten and therefore the margins of error are ever so narrow. In a company that caters for the SMB in a low-touch model, the sales cycles are measured in days, not quarters, the software releases are shortened to weeks. The discovery of bugs usually occur within hours (or minutes) after a new version is introduced. Hence, Openness and Transparency are paramount and there is no time for ego games or controlling vital information (I have written about these aspects in the past: Transparency & Communications).

To reiterate – in a fast-pace, ever-changing, 24X7 environment, the need to feel the operational pulse, the need for responsiveness and open communications, the need for a listening ability and accountability are vital for success. Time spent on BS, on analysis-paralysis, on political games, on territorial squabbles, is time spent away from making sure your customers are successful, and that will be evident on the company’s bottom line and eventually, on the quarterly bonuses.

System Integrators’ Cloud Strategies – React or Lead?

“A leader takes people where they want to go. A great leader takes people where they don’t necessarily want to go, but ought to be.” (Rosalynn Carter)

Recently a number of medium-sized System Integrators (SIs) have approached me to help them either define their SaaS/Cloud strategy, validate their strategy or help them with the realization of their strategy.
It seems that, the shoe has dropped. It took two years of relentless Cloud hype for the System Integrators to finally understand that major changes are occurring which will impact their customers and therefore – them.

New Breed of System Integrators
There is a new breed of Cloud System Integrators (up to a year ago they were called “SaaS System Integrators”) including companies such as Appirio, Astadia, Bluewolf & Model Metrics, that are dedicated to providing SaaS based services. Being newcomers, they are quite small compared to the large players, but their growth rates are phenomenal (e.g. Appirio has been growing in triple digits since 2008). So far they are have not been big enough to pose a visible threat to the old timers, but I suspect that the Dinosaurs are starting to feel a bit uncomfortable with the quick mammals that are infiltrating their territories.

Lack of Strategy – Tactical Approach
Having studied approaches of the major players in the market I have come to some interesting observations:

1. There are around 15 Cloud services that SIs could offer their customers. Examples are: billing/integration/SSO services, PaaS development (Azure, Google Apps Engine, Force.com), SaaS-oriented testing, training, 24X7 NOC, etc. 
2. Judging by my engagement with some of the SIs, most are not even aware of what those services are, and are therefore concentrating on a few obvious choices.
3. Most of the SIs are offering only one or two Cloud services from the list, and therefore:
4. Most large SIs do not have a SaaS/Cloud strategy, or, their strategy is to wait and see how the market develops.

A number of large players have chosen to become a single application integrator. Examples are:
Accenture is a Salesforce.com partner.  Ernst & Young are helping implement EmployU and HumanWave. Deloitte is with Workday and Genpact is a Netsuite implementor.
Capgemini is working with AWS to provide a Cloud Computing COE.

Considering the size and nature of the business of these players, it seems that these services are not high on the priority lists of the big SIs. It is quite obvious (to me) that these giants have not defined a SaaS strategy, rather they are reacting in an opportunistic manner to the market - akin to a “me too” tactic, just to have something 'Cloudy' or 'SaaSy' on their web site.

Cloud Strategy – React or Lead
As mentioned, many SIs have adopted tactical approaches, at the Cloud Service level, rather than a strategic approach. Further analysis led us to the conclusion that by zooming out, grouping and mapping the above Cloud Services. we can define five Cloud Strategies for System Integrators:

1. SaaS Aggregator – Provide Applications on the cloud
2. SaaS One-stop-Shop for Software Vendors
3. Cloud/SaaS Adoption for IT
4. Private cloud technologies
5. BPasS - Could Integration

Note that Strategies 4 and 5 are "smaller" in scope and could either be included in one of the first three. Each Strategy includes between four to twelve of the above mentioned Cloud Services.

Using this prism, we could say that a number of players have chosen a strategic path:
IBM’s GBS and Wipro are offering services that we define as SaaS One-stop-shop for ISVs (strategy 3).  KPMG is offering Roadmap for SaaS Adoption (strategy 5) while Infosys is acting as a SaaS Aggregator for enterprise IT (strategy 1).  Smaller players such as iProcess are offering BpassS (Strategy 2)

Even though adoption rates are growing very fast, the hype of Cloud Computing is a couple of years ahead of reality, especially in the larger organizations that are the natural customers of the leading SIs. Some SIs have chosen to play it safe: not invest up front in new technologies and methodologies and watch the market carefully. Since they will still be milking many fat cows for a few years to come, the decision not to decide could be considered a smart, conservative strategy.

Others, though, have taken leadership positions, risking investments without a clear date on the ROI. They have the advantage of defining the market trends and have a better chance of emerging as the de-facto leaders in a few years when all the players, that are currently sitting on the fence, will have to define their strategy and see what bones are left to pick.

Automating SaaS Operations

"The first rule of any technology used in a business is that automation applied to an efficient operation will magnify the efficiency. The second is that automation applied to an inefficient operation will magnify the inefficiency" - Bill Gates

This is an article which I posted at the Nolio blog site as a guest blogger back in 2009. Not only is is relevant today, but perhaps even more so...

The Next Killer App
If I had a great idea for the next killer app (I have, actually) and if I had unlimited funds (I don’t, actually) I would have built the software as an on-demand offering.

I would have spent half my funds on building the operational support systems – provisioning, billing, retention policy, self-service, report generator, etc. The other half would be invested in building instrumentation, redundancy, automation, integration, application level monitoring, silent upgrades, customer notifications, and so on.

The rest of the money (you may wonder about my math, but hey, I’ve got unlimited funds) would go towards building the actual application.

Most SaaS vendors out there (and they are growing fast) have chosen the predictable path of building the application first, and worrying about serviceability later. This is the fastest way of getting to market with low costs. The next step is choosing some viable hosting solution and off we go, offering the world our ever better CRM.

Growth
Many months and dozens of customers later, reality hits with all the issues of servicing the software, rapid growth and dealing with labor intensive tasks that are the humdrum of daily life in a SaaS operation. Provisioning/de-provisioning, configuration changes, customized reports, and the most dreaded – upgrades, task the team as a whole, especially when the product is successful and the number of customers is growing daily.

It is not that SaaS executives, architects and engineers are lacking in any way. On the contrary, they are mostly smart, inventive, and creative and have a deep understanding of their customers’ needs in the specific domain. The problem is that they are product people, not service people. Practically none of them come from IT and cannot envision the life of a service operations engineer.

At this point, automation becomes crucial to the survival of the business.

Whether it is built into the next version (many architectures make this quite difficult) or done externally, automation is needed to reduce costs, physical labor, frustration and mainly, error-prone manual procedures. Repeatability, which is a derivative of automation, is also crucial.

Automation is needed across the board. Be it in setting up a new server, or building a new application instance. It could be a manual procedure regarding provisioning of application resources, or building a seamless upgrade procedure.

Outages happen. How quickly can you recover from a service disruption and ensure that the recovery does not create it own problems? Automation not only provides the routines for quick recovery, but instills a discipline of thinking out the necessary steps, discovering dependencies and planning ahead. An added benefit of automation is that it documents the process so you can go back and review the best and worst of your procedures.

In my next post I will take a closer look at the SaaS Upgrade Nightmare.

2 x E-cube = S-cube - Simple math for SaaS Scalability Success

"The circumstances of human society are too complicated to be submitted to the rigor of mathematical calculation" (Marquis De Custine)

Recently I posted a discussion on the link between success with SaaS and scalability and how, therefore, the Service Operations needs to be geared to handle scale and deal with fast growth.

On a recent consultation engagement, I gave a presentation to the board members of a SaaS company that decided to penetrate the SMB, following their hardships in securing big deals in the enterprise market.

As part of the thought process and brainstorming session I came up with a marketing catchphrase to make a point. At the time I did not think much about it, but as I was working on the board presentation, I realized that there it was much deeper than I had originally thought.

I would like to share this with you.

The First Cube

What I drew on the white-board was roughly this artistic creation:

Announcing: “E-cube is the winning formula!”

• Easy to Buy
• Easy to Implement
• Easy to Use

Easy to Buy - Their product is very easy to implement, and easy to on-board a new organization, (although it is a complex product). There is a technical issue that involves an architectural change, so that became part of the plan.

Easy to Use - the product has a great user interface and intuitive flow. They need to add tutorial videos.

Easy to Buy – That was lacking and that was what we decided to focus our efforts on. It included a lead generation program, a no-touch free trial and an improved landing site.

The Second Cube

As I was working on what it would take from the company’s end to deliver scalability, I realized there was another E-cube involved:

• Easy to Sell
• Easy to Scale
• Easy to Maintain

It is all nice and well if you can bring thousands of leads to your site with the E-cube formula, but if you cannot convert these leads into paying customers and then give them the best service on a controlled budget, you will have not achieved your scalability goals.

So, Easy to Sell means - a simple pricing model and a top-notch insides-sales team (not easy to come by though) backed up by funnel management software. 

Easy to Scale means - a solid, configurable, multi-tenant architecture, self service features and automated procedures.

Easy to Maintain means – a full featured Operations Support System, Service Operations practices and the discipline to enforce them.

So with two E-cube guidelines one can achieve SaaS Scalability Success.

Ergo: 2 x E³ = S³

Q.E.D.

The SaaS Consumer’s Point of View – Negotiating an Agreement

“The food was superb, the atmosphere was great, the service was outstanding; it was those goddamned customers that had to ruin it all”. (Morris Green, restaurateur, NYC, 1987)

This short article is an introduction to an interesting blog post that was published recently by Derek Singleton from ERP Software Advice, but before I let you go, I want to take this opportunity to talk about the customer’s perspective.

My writings, presentations and webinars have been mostly dedicated to the point of view of the software provider, whether SaaS or SaaS-to-be. I would like to present a different point of view.

Throughout the years of being on the software provider’s side I have learned a thing two on what makes a happy customer and that is the guiding light I have been trying to follow for years.

I believe that if we understand the customer’s perspective we have a better chance of providing a good service and creating a happy, loyal customer base.

Who is the customer?

SaaS adoption has been mostly done in a haphazard fashion throughout the years. Many of the early adopters were business managers at the department level in the enterprise.

Even within smaller companies, the decision to consume SaaS was usually a point solution, for a particular issue to handle and not as part of a well thought process and methodology.

In many cases the IT department and CIOs were kept out of the loop in defining needs, selecting the service, negotiating the deals and the process of provisioning and de-provisioning. In the extreme, IT found out about their company consuming a web application only when a user would call the help desk and ask for support.

As more CIOs are ‘getting it’, as more IT departments are becoming cloud-oriented they are becoming that target customers, rather than the end users. They are usually better equipped (once their fears are neutralized) to judge the provider, the application, the integration and to negotiate a better deal for the organization.

These IT professionals should be planning a Roadmap for SaaS so that the consumed applications become part of a coherent plan rather than something the cat dragged in.

We should start focusing on this new generation of SaaS customers.

An interesting article covering the negotiations with a provider is therefore presented.

SaaS, Scalability and the Three Little Pigs

“We have the wolf by the ears, and we can neither hold him, nor safely let him go” - Thomas Jefferson

A few days ago I gave a talk at a SaaS Business Challenges conference and I would like to share the main theme with my readers.

It’s the Service, Dummy!

When SaaS burst upon the scene about ten years ago, the first consumers were of two types: Those that used on-demand as an ideology, having foreseen the cloud revolution early on, and those that had no choice, because the other option was an expensive, time consuming and complex solution.

Back then, there were few SaaS application or service choices, so customers had to be very forgiving about the service levels and were willing to put up with reduced functionality, outages and low response times. SLAs, if they even existed, were non- binding and lacked both depth and breadth.

These days, however, when there are dozens of SaaS applications for every need, the differentiator is no longer the functionality. Most SaaS applications offer a similar set of capabilities, and as applications change on a bi-weekly or monthly basis, features are added on an on-going basis.

Even if one comes up with a revolutionary solution and provides the only SaaS offering of its kind, it is safe to say that within a year, three new SaaS companies will offer the same, or an improved set of capabilities.

So what differentiates one SaaS offering from another? It’s the Service, dummy!

Features don’t make a loyal customer – outstanding service does.

Scalability

There aren’t many SaaS companies out there that service a few dozen Fortune 200 companies while keeping the profitability high. The general rule is that margins are small and that profitability is achieved through hundreds or thousands of customers.

Giving great service to 20 or 30 customers is a no-brainer. Just throw more bodies at the problem and you will achieve a highly satisfied customer base.

But what happens when these numbers multiply rapidly? You’ll soon find out that what worked for a few dozens might collapse at the next order of magnitude.

When I approach SaaS companies at the stage when they already have a growing customer base and warn them about the perils of scaling up, the usual response I get is:

“I wish I will have to deal with that problem”, meaning that they would love to have 200 hundred customers that cause strain on the infrastructure and operations and deal with those ‘good’ problems then. It is only human to postpone these issues when they are not burning your behind.

To those CEOs I would say: “Pack your stuff, return the money to your investors and go look for another job elsewhere”.

Your investors did not give you their hard earned dollars for a proof of concept.

They invested in you because they believed that you will bring in thousands of customers.

My best clients are SaaS companies that come to me when they are in pain. They start loosing customers because they did not build an operation capable of handling the scale which they had wished for.

As I have stated time and again – SaaS companies usually consist of outstanding, creative developers that build great technology, but they don’t have the IT and/or operational experience. They lack the know-how and especially the methodology for building a successful, scalable operation.

Building Operations for Scalability

I have written much about these issues and they can all be found in previous blog postings.

Suffice to say that the setup you need to build for a scalable service operations includes:

· Methodology – the framework of practices, templates, workflows and tools

· Operations Support Systems – everything else your engineering team left out of the product.

· Executive buy-in and awareness -define the metrics, and provide tools to capture and analyze those metrics.

Can a Huff and Puff Blow Your House Down?

So what has all this to do with the Three Little Pigs?

The first little pig built a house of straw, since he didn't have time to invest in operational infrastructure and wanted to make a quick exit. He ended his career as wolf poop.

The second little pig invested in a better infrastructure but did not pay attention to the practices and processes. He lasted longer in his wooden structure but was huffed and puffed and blown away by the competition.

The third little pig took his time and invested executive attention in doing it right from the start. He lived happily ever after.

So even if you cannot afford a brick house from day one, you should have the blueprints ready and the determination to add the bricks when they become available.

By the time the Scalability Wolf arrives, you should have a sturdy enough structure and react quickly to threats, to survive and prosper.

Private Clouds – What’s in a Name?

“Happiness is like a cloud, if you stare at it long enough, it evaporates” - Sarah McLachlan

Remember when all of our secretaries and stewardesses turned into office managers and flight attendants overnight? Remember when all the co-los and server-hosting companies became cloud providers overnight?

(To alleviate all suspicions – as my readers know, I am an ardent advocate for Cloud computing and SaaS in particular, so this post is not about arguing the merits of this constructive and disruptive trend).

Cloud means different things to different people – but mainly it means “a cool way to market my same old, tired stuff”. I worked with a co-lo provider a couple of years back. A few months ago I went to their site to check prices and lo and behold: They became a Cloud Provider! They had clouds splashed all over their site and every solution they sold was a Cloud solution. I called up a sales person and asked about their elasticity and time units. Turns out they were elastic in one direction – you could always order more servers – and you only had to commit to one year in advance. On a geological time scale that is quite flexible. When I laughed and asked what was ‘cloudy’ about their offering, the guy got confused and said that his manager will get back to me.

So what is all this newspeak about Private Clouds?

Guess what is happening to the good ol’ data center? As David Linthicum aptly puts it in his latest blog post: “the reality is that ‘private cloud’ is just another term for on-premise systems”.

Point number one is that I find the term an Oxymoron:
Cloud means that it is ‘somewhere out there’; location is transparent. Cloud means sharing; resources are transparent. The physical server I used this morning might be used by someone else this afternoon.
Private means it is in my back yard and only I get to play in the sandbox.

Point number two is that even if we apply the Cloud concepts to the enterprise, it will be relevant to a very small number of very big players. Those enterprises that are truly global and distributed. They could take advantage of the peaks and troughs, of the "follow the sun" model, of the large numbers and justify the investment of a Cloud player.

And what about the rest of us mortals? Let’s assume that a certain percentage of our IT services will not transition to the cloud – be it regulation, compliance, perceived loss of control, or the illusion of maintaining job security. We should be using Cloud-enabling technologies, to make a smarter use of our resources and data centers. That means virtualization, automation, orchestration and auto-provisioning technologies. We only get the silver lining – not the cloud.

So the Private Cloud, in essence, is a wonderful opportunity for the big vendors out there to sell to the enterprises - new equipment, new systems and new services.

The SaaS VP Operations as Product Manager

SaaS Operations Support Systems

“'Tis not enough to help the feeble up, but to support them after” - William Shakespeare

Every SaaS company needs to deal with numerous functions that are not necessarily part of the technological stack that originally came with the application, namely the Operations Support Systems.

If a SaaS start-up had unlimited time and funds to plan and build the perfect solution, they would probably all be in the Caribbean islands doing what people with unlimited time and funds do.

Just Do It!
Because of the nature of monetizing SaaS, companies try to get to market as soon as possible, getting subscriptions fees streaming in. Sometimes, they even launch with a half baked solution that will provide added value to the customers at the expense of future operational headaches.
The logic behind this is that dealing with a scalability problem, is a good thing. In other words - who doesn’t want to reach the stage when too many customers are taxing the team? We’ll deal with that when it becomes a problem.

These days, numerous PaaS offerings or SaaS frameworks offer built-in operational support systems functionality, but many of the necessary features are not supported.
Most of the SaaS application in the market were not built with these frameworks for various reasons. The most prevalent are that they were not available a few years ago, and that engineers have a tendency to build everything from scratch, or using frameworks (e.g. LAMP, Java, .Net, RoR) that they are familiar with.

When the typical SaaS service is launched, it lacks functionality that would support the scalability of the service operation. It is left to the Service Operations team to deal with all the 'maturity' functionality that the product lacks.

Let us examine some of the Operations Support Systems functions that are typically not handled by the application.

On-boarding new customers
One would expect most SaaS systems to have automatic provisioning. While that is true in many cases, a lot of the systems allow the customer to define users, but creating a new customer entity is left to the Support or Operations team. That may require generating a new database or schema, or setting up storage, etc. If the company is adding one customer a week, it may be manageable, but at a higher rate this is a taxing job and error prone.

De-provisioning
While it is expected that some level of automatic provisioning is provided with the product, very few SaaS applications provide a simple (never mind automatic) mechanism for removing existing customers. Very few SaaS developers design with of the prospect of loosing a customer in mind. Beyond the task of removing dependencies from the database, there are issues of releasing resources, and removing customizations.

Billing
Such a basic function, (one would think, for a company that lives or dies by subscriptions) is typically lacking from most SaaS infrastructures. While the situation is improving dramatically with the advent of PaaS and SaaS development frameworks, many systems still start out with excel sheets and a lot of manual work. Customer data must be extracted by some ad-hoc solution form the application database or is duplicated in the CRM, which causes endless synchronization errors. Ad-hoc solutions are implemented (either in-house or SaaS billing solutions) as the billing becomes ever more complex but until that occurs the brunt of the work falls upon the Operations team.
Metering is a whole new layer of complexity, if the billing is more complex than a fixed amount per seat per month.

Retention Policy
Some SaaS companies plan ahead for resource consumption by their customers, bur many start dealing with the issue only when they begin running out of storage, or when their storage costs are starting to hurt. Most customers want to retain their data forever on the provider’s disks, but that is impossible. So a retention policy must be defined and followed (such as delete files that are older than X months or larger than Y gigabytes).
The problem is that the application does not support that, so manual work is requried or ad-hoc solutions have to be built around the product.

Failover and Backup
Automatic recovery and failover are rarely built into the initial solution and are usually managed by the Operations team via building complex solutions with networking boxes.
This is true for backup and recovery mechanisms as well. The Operations team frequently has to build mechanisms around the production to support backup and recovery and most often they require much manual labor.

Application Monitoring
Well designed software is packed with instrumentation that is easy to turn on and off and easy to monitor and interpret. Not all SaaS systems have that built-in capability and the Operations team has to create an application-specific, monitoring infrastructure that can detect and react quickly to service degradation.

Seamless Upgrade
Upgrades are the recurring nightmare of any Operations team, especially on applications that have a demanding uptime SLA. Few SaaS systems are designed with that goal in mind. It usually takes a level of maturity and a sizeable customer base to get Engineering to consider revising the code to allow some sub-systems a no-downtime upgrade.
The problem is that is usually requires major revisions of the code if the system was not designed a-priori to handle a silent upgrade.

SLA Management
Whether you are using automated SLM or not (chances are you are not), you need to take into consideration the various aspects of your service that need to be metered, tracked and compared against a set of Service Level Objectives.

End User Broadcasting
Sometimes it is necessary to communicate with all of you users, or a sub group of them in real time. What a better option is there that to pop up a message on the end-user’s browser that you can compose on the spot or pull from a list of canned messages? The operations team need an integrated solution with the product to be able to do that.

Operations Console
And to tie it all up, a separate application that controls all the operational aspects of the service is needed. Included should be: provisioning/de-provisioning customers, password management, real-time login view of current users, real-time view of application usage, customer communication console, production environment control, etc.

There are many other operational features that are typically not found in a SaaS application as it leaves the factory floor. Just to mention a few: Integration, Reporting engine and reporting Database, Security, Scale-up and Scale-out capabilities, Sandbox, Status Page.


Development and Product management Experience Required

The VP Operations (or whatever title the job carries) is required to be a product manager of sorts, and a background in software development is almost a must. The Operations manager should be highly involved in the product roadmap and insist on having a say in defining future releases. There will be a contention between investing in Serviceability versus Functionality. Since the paying customers require more functionality, it is usually an uphill battle to gain service upgrades to the product.

While building an organic set of solutions into the product may practically take years, Operations cannot continue to throw bodies at solving scalability and downtime issues. So beyond influencing the product group on the direction in which the application should be developed, the Operations manager needs to build a set of tools addressing the Operations Support Systems needs as stated above. One option is to nurture a relationship with VP Engineering and get resources from her group to build well defined solutions that could each be completed in a couple of weeks of work. This is especially true in early stage companies where the Operations team is small and the engineers take the brunt of many of the operations’ tasks. The VP Engineering would appreciate the need as members of her team are feeling the pain as well.
In a more established company, the VP Operations must make sure that there are coding/scripting capabilities in the team, so simple projects and tools could be developed within the team, with minimal aid from the Engineering group.

Change Management and the Sanctity of Production

“Most people are afraid of change. We love it!” (Sign of a beggar on a street in San Francisco, 2004)

(Note: This article is part of the STORM™ methodology)

Change is the greatest cause for service interruption in any IT operations. Period. Stop. Exclamation mark.
In the extreme, one might argue that change is the cause for every service interruption if one counts hardware or software malfunctions as a change as well.
SaaS operations tend to suffer more from a lack of proper change management for two reasons: First, the consequences of a service outage for a company whose entire existence depends on its service, is dire. Second, SaaS engineers, as I have written in numerous posts, lack the discipline that is more inherent in IT departments.

And yet, my experience has been that in most SaaS companies, changes are unsupervised, undocumented, unauthorized, unplanned, (sometimes unnecessary), underestimated and un_____ (fill in the blanks).

The importance of Change Management cannot be overstated and it is the first practice that I have implemented at companies that I worked at (or for). I wince when I recall the casualness which I have witnessed at various SaaS companies about making changes in the production system. I can quote my former boss, Mansur Salame, CEO of Contactual, saying that “production should be treated as sacred, with utmost respect appropriate to holy places” (or something of the sort). And, boy, were we sacrilegious back in those days!

In the Chapter on Change Management in the upcoming book, I will present my STORM™ adaptation with much detail.

In this post I will outline some guidelines for sane change management.

Sixty Seconds on Change Management
Below are listed the objects that comprise a comprehensive Change Management practice.

RFC – Request for Change document. Must initiate the process, regardless how small or major the changes are. It should include the what, the why, the when, the risk, the potential impact (on customers or components), and a checklist of notifications and tests that should/should not be done.

A Change Window must be defined, clearly notating what type of changes to what subsystems are allowed at which days, during what hours.

Change Calendar – which might be implemented in a number of static or automatic formats, must represent the ‘Change Window’, and depict all planned changes by the company, service providers and customers, and must be part of the RFC process.

Change Advisory Board or CAB is the pre-determined group or people who scrutinize and approve the RFC. The CAB may be large or small but it should include at least one person who is not involved in the request and planning process. The CAB may meet on a recurring schedule or as needed.

A Change Record is a record describing a change that occurred in production or the eco-system. It could be implemented as a database or excel or within a ticketing system. It should include the what, when and impact (on customers or components). Much important information could be derived from this data store that pertains to the Incident and Availability Management practices.

The Maintenance Plan is a detailed document defining the pre-requisite tasks, the maintenance tasks, rollback tasks and post-maintenance tasks. Each task should have a description, an owner, a time and duration. In most cases the plan must be scrutinized to the lowest detail level, and practiced in a Pre-Production environment that should mimic the production environment as much as possible.

The sixty seconds are over. This was just a teaser. Obviously there are templates, workflows and a sleuth of details that tie all of these objects into a well-oiled practice. The book will expand on the details and include the workflows, the templates and methods for automation.
To summarize; as the market matures and competition thrives, the big differentiator will be the second ‘S’ in SaaS and customers will become less and less forgiving. If a SaaS company does not practice a robust Change Management practice it will end up paying in a frustrated staff and customer churn.

SLA Consequences to Service Operations

"What, me worry?" (Alfred E. Neuman)

In my previous post I discussed some basic concepts about SLAs, SLOs and penalties. As promised, I am addressing the ‘who cares?’ question.

From a Service Operations point of view, you may shrug your shoulders and claim that these are issues with the Legal and Finance departments. Most likely you were brought on board later in the game and never viewed an SLA until your were forced to do so.

As the person responsible for keeping all the services up and running, it may be best to keep the SLA to a minimum. After all, a document containing vague language, with little commitment and liability would be hard to wave in front of your face when the service levels drop.

I will argue that vagueness will play against you. A tough SLA will require the company to adhere to the high service levels they are committed to, and yes, pay the penalties for breaching these agreements. Keep in mind that if your service level drops one time too many, the legalese you will be hiding behind will not save your butt when customers drop from the service or simply do not renew.

I would take it even one step further. I advocate that the Service Ops managers bonuses are tied to achieving those SLOs that will keep a smile on the customers’ faces. (typically up-time and response time, but in some cases there are other objectives that are crucial to the customers). The carrot and the stick should work nicely to assure that you are doing the utmost to live up to the agreements.

Commitments
Another issue that concerns you (Service Operations) is that Sales are making commitments that you are suppose to keep, usually without you ever knowing about it. Operations needs to initiate a fact finding effort to learn what is there. You need to know what you are capable of providing. Everybody likes to state that they are five nines (99.999% uptime) but how many companies out there really are? You need to monitor and test your service over a substantial period of time before you commit to those numbers.

Another point in favor of having a good grasp of your SLA is that you, as a consumer of services would be conscience of your requirements vis-à-vis your service providers.

That will include the hosting services, your ISP, your communications provider, and whatever cloud services you are using. In my past positions as VP Service Operations I have been appalled by the contracts that my predecessors have signed with service providers. Some of them had no consequences to service level degradation. Others had ridiculous clauses such as 'for every hour of downtime, the credit would be for one hour prorated service cost' which meant that there was no real penalty. Another contract stated that we could get out of the agreement if for three months in a row(!) the service provided was available for less that 75% of the time.
We would have been out of business by then.

Where are those damn SLAs?
As we have seen, SLAs that are broad and meaningful will be complex. Add to that various service levels such as Standard, Gold and Platinum and the fact that some customers have negotiated special terms for themselves, and you are dealing with a mean, slimy problem.

To compound that problem, nine times out of ten, these documents are sitting on someone’s laptop in a PDF format with perhaps a hard copy in a dusty folder, in the cabinet below the espresso machine.

Imagine the exercise of figuring out if an SLA was breached for a particular customer, and if that breach carries a penalty.

I have painfully gone through that exercise too many times, and believe you me - I had much better things to attend to following a service outage. The process was extremely slow, finding the various documents, looking up the terms and comparing the events with them.

Then a calculation was needed as to how much credit was due. And all this was done for a single customer. Multiply that by the number of customers that may have been affected and you have just wasted many good hours of Solitaire.

SLA Management Tools
There are multiple tools out there (some are offered as SaaS) to manage your SLAs. Many of them provide a full cycle of defining SLOs, creating SLAs, generating the documents, monitoring performances against obligations, computing compensation and generating reports. I have not used any of them (although I used to work at an SLM ISV), so am not about to promote any single one, but there are very slick solutions available.

If you are at an early stage, it would be hard sell for you to justify to management that you need to start paying for a service that possibly no one in the company comprehends.

Typically, when a SaaS company starts out there are very simple, non-abiding, fixed SLAs, so there is very little attention paid to this aspect of the business.

As with any aspect of SaaS Service Operations, scalability issues hit you when you least expect them.

As most (all?) SaaS companies do not start with Service Level Management software, by the time it becomes a burden they will have many dozens, or hundreds of such SLAs. The effort of converting them to an automated system is daunting.

Therefore, you can start structuring your existing and future SLAs into a simple excel, or DB so that they are easily accessible, and comparable.

An example of a typical SLA would be stored in a table such as below.
The values for the various SLOs in the table were automatically populated from the definitions in the pre-defined Platinum and Gold tables (which state the default values for these SLAs). They may be overridden by specific values, following negotiations for a particular customer.

Cust.	Calia	…	Google
Cust ID	123	…	213
SLSLA	Gold	…	Platinum
Uptime	99.9	…	99.99
Response time	under 6 sec	…	under 4 sec
Support Response time	2 hours	…	30 min
Support Avail.	12X6	…	24x7
Major Outage Resolution	1 hour	…	30 min
Partial outage resolution	4 hours	…	2 hours
Minor Outage Resolution	12 hours	…	6 hours
Maint. Notification	10 days	…	2 weeks
FTP	12 hrs	…	6 hours
Outage Notif.	Email 1 hours	…	email + call 30 min

In the book I will elaborate on the structures and the tools and how to automate the compensation computations.

SLA Management for SaaS

“God does not ask about our ability, but our availability.” (Source unknown)

(Yet another chapter in the book - keep the feedback coming!)

As the second ‘S’ of SaaS indicates, the on-demand company is all about providing a service and therefore one would expect Service Level Agreements to be well defined and understood in this industry, but the facts tell another story. Few SaaS companies pay much attention to the SLAs, few companies really invest in it and most customers are quite clueless about it as well.

SLAs are tricky. Every SaaS provider is supposed to adhere to its service level commitments but on the whole, it is a document that most providers tend to keep out of the limelight and out of the conversation with customers. Judging from my experience, many SaaS companies use a single, non-abiding, standard SLA for all customers, keeping to a minimum their commitments and consequences.

An SLA, as its name suggests, is an agreement between the service provider and the consumers, consisting of sections regarding the various commitments to service levels that will be matched or exceeded.
Each section is defined as a Service Level Objective (SLO).

A typical SaaS SLA should have the following SLOs:

• Service Availability – define the availability of the service represented in percentage (e.g. 99.95% uptime)
• System Response Time – define response time of various transactions represented in seconds. (e.g. login should not take more than 9 seconds)
• Customer Service Response Time – a response on customer enquiries should take no more than an allotted time for various services (e.g. enabling a service for a new group should take less than two business days)
• Customer Service Availability – hours of availability of customer service represented in a ‘hours per day’ notation. (e.g. 11X5 for regular customers, 24X7 for platinum customers)
• Service Outage Resolution Time – the times it takes to restore a service after an outage has been reported. Represented in minutes and hours (e.g. 30 minutes for a full system outage)
• Failover Window For Disaster Recovery - how long will it take to restore the service in a disaster recovery site, if disaster disables the main datacenter.
• Reclaiming Customer Data – a commitment to transfer all (agreed) data in an agreed format in case the customer leaves the service.
• Maintenance Notification – the advance notice that the provider will notify customers of planned service outages, represented in days. (e.g. a planned downtime that will take more than one hour requires 10 business days notification)
• Proactive Service Outage Notification - the time it takes for the provider to inform the customer that there are service issues, represented in minutes.
• RFO (Reason for Outage) – a report to customers following a service outage explaining the circumstance, the incident and steps taken to remedy the problem. (For more information see the chapter on Incident Management). Some customers require an RFO automatically; in some SLAs it is written that an RFO will be generated only following a specific customer request. Usually the company commits to three business days following the service disruption.

Note the emphasis on should when referring to the SLOs of the document. The SLA provided by most on-demand companies consists of two or three paragraphs at most, regarding uptime, customer service availability and perhaps another one of the items above.
Many providers have additional services such as daily reports, daily data aggregations, or FTP services. Each one of these services merits an SLO that should be part of the document.

Some SLOs override others. In the example of an service outage, the Availability SLO takes precedence over the Response Time SLO, as you would not expect the performance of the system to be up to par when the system is down. On the other hand, this will kick start other SLOs such as Outage Notification, Resolution Time and Support Response Time.

Customer Expectations
Not all SaaS companies are created equal. They will vary by maturity, by the vertical they are serving, by the company size they cater for and, of course, by the type of application.
Some applications are core and some are peripheral. Some applications are used around the clock, like metering or call centers and the customers have zero tolerance for downtime. Other applications are rarely used outside of office hours, (e.g. payroll, talent management) and if the system is down, the price is a handful of irritated end-users that will need to take a coffee break earlier than they planned.
Larger customers tend to have more rigorous demands while lower paying customers will usually be more tolerant of the system’s performance and support availability.
Therefore, your SLA should reflect the relative position of your service along the following three vectors:

1. Customer size (reflecting subscription [potential] size)
2. Core vs. periphery
3. Downtime tolerance

So if you are providing a mission critical application to a large customer, whose downtime will cost the customer real dollars, your SLA should be taken very seriously.

Service Level Breaches and Penalties
We have seen the promises that come with the SLAs, but many of these agreements fail to state the consequences to the provider of not meeting the terms.
Each SLO should also define the penalties for breaching the service level commitment.
Penalties are typically specified as a prorated credit for the following month’s subscription fees.
From the customers’ point of view, the penalties should not be flat rated but increase as the service deteriorates, so that the second outage will carry a heavier penalty than the first outage. It is rare that customers insist on this point but those that do will need to negotiate these terms separately.

There is typically a maximum. It is unusual that accumulated penalties will top the monthly subscription costs. There is a catch here. As an extreme example, if your service was down for the duration of the whole month, the customer will be exempt from paying a full month’s service fee – but this is ridiculous of course. The damage to you customers is typically orders of magnitude higher than the subscription costs.
Many SaaS customers commit up front to a year or more of service, for a reduced subscription price. A good SLA will include a section that allows the customer to breach the extended commitment if the provider failed to adhere to the service levels for, say, three consecutive months.

The next chapter will outline what all of this means to the Service Operations group and why should you care about issues that initially seem to be in the domain of Sales, Legal and Finance.

Inter-department Communications

(Yet another chapter in my upcoming book on SaaS Service Operations - Your feedback has been great so far; thanx and keep it coming.)

"The Problem with Communication is the illusion that it has been accomplished" - George Bernard Shaw

While it is true of any institution, communications between the various silos of the organization is particularly vital for the successful operations of a SaaS company.

The reason are that things happen much faster in an on-demand company, customers are in constant contact with the company and expectations are high for a fast turn around.

At a product company, when bad things happen to the application, nine times out of ten, the software company doesn’t even know about it, and the customer’s IT deals with it. The end user is rarely in touch with the product provider. The product salespeople tend to ‘shoot and forget’ once the commission has been paid. If things go bad, the customer can mostly blame itself for not deploying or maintaining the software correctly or for not doing its due diligence.

Multiple channel interaction
At a service company, on the other hand, a typical customer will interact through multiple channels continuously. The CIO may have a direct line to the SaaS CEO. The IT department may be in touch with professional services, and managers of the service on the customer end could be speaking with the Program Management group. Members of the Operations group will inevitably be in touch with supervisors or IT managers on the customer side, and Sales will have developed personal relationships with managers on the customer’s side, as they nurture the relationship to expand the sales in-house. And, of course, the end users might be in daily contact with Customer Support.

Customers, naturally, will be irritated when things aren’t going smoothly regarding any one of multiple scenarios. It may concern a delayed service initialization, an undelivered bug fix, an incomplete customization, an unsatisfactory report, or (ouch) a service outage. Part of the allure of on-demand service is a much faster turn around time in every aspect. The customers believe it and expect it.
Imagine the customer’s frustration when they call in any one of their contacts within the company to inquire about unresolved issues, and that person has no idea what they are talking about.

Disconnect between the groups
Typically, a SaaS company will be using a CRM that serves Sales and Customer Service. In many organizations the Sales view is radically different from the Support view and information available to one is not available to the other.
It is rare that other members have access to the CRM. Operations, Engineering, Professional Services and Program Management keep their own records in different systems for various reasons and are not trained in using a CRM.
Not surprisingly, the different silos do not have much knowledge of what each department is doing, and I have seen continuous tension between various groups and quite a lot of finger pointing when bad things happen.
It is also typical to see a startup company, where everybody occupies a single open space office, yet where so little communication takes place between the groups and political affiliations begin to form.
Resources are always limited and the demands are constantly growing; how does one prioritize the tasks and attention to a particular customer?

Service Outage and Communication
To illustrate through an acute, but none too rare, example: Many a time I had experienced a service outage that, for obvious reasons, took everybody’s focus and energy. A couple of offices down the hall sat the Sales team and across the continent were various regional Sales reps. They were not informed of the outage since they play no role in detecting, classifying or resolving the issue, and all those that knew about it were busy trying to fix it, or taking customer calls. Often the customers, especially the senior members who have established a close relationship with the sales reps, would call Sales or Program Management immediately asking for updates. The uninitiated sales rep would answer that they are not aware of any outage and perhaps the problem is local to the customer (This would usually trigger a nasty remark about the incompetency of the provider). The experienced sales rep would mutter something in embarrassment and then storm over to the Ops group demanding an explanation why, once again, Sales was not notified of the outage. Not only does the company look bad, but it also raises unnecessary tension between the groups
(This issue will be addressed in the chapter on Incident Management)

Recurring Mandated Meetings
Inter-department communication is the answer. If the managers of the different departments talk to each other on a regular and formal basis, issues can be addressed before they get out of control, plans can be communicated and a deeper understanding of the challenges of each department can be better understood.
Since Operations is at the center of it all at the end of the day, and since Operations will take the blame for whatever incident that occurs, VP Ops group should initiate these meetings. This initiative and meetings will also serve as an important PR tool for the service operations group.
Following are the inter-department sessions that should be standard in a SaaS organization to improve communication and visibility and to help prioritize tasks and address issues before they boil over.

Name: Daily Operations Sync
Frequency: Daily (15-20 min)
Suggested Time: Late afternoon
Participants: Operations, Support, Program Mgmt
Agenda: Burning issues, Service outages, Planned maintenance, Delayed deliveries, Staffing

Name: Customer Success
Frequency: Weekly
Suggested Time: Monday
Participants: Sales, Program Mgmt, Support, Operations, Professional Services, R&D
Agenda: Customer Success Score sheet, Updates, Delays, Priorities. Address Red and Orange flags

Name: Operations-Engineering Sync
Frequency: Bi-Weekly
Suggested Time: Anytime
Participants: Operations, Engineering, QA
Agenda: Requirements, Releases, Known issues, Bugs, Dev/staging environment


Name: Company Fridays
Frequency: Bi-Weekly
Suggested Time: Friday Afternoon
Participants: All employees + food & beer
Agenda: Announcements, updates and department presentations


Name: SPOF Analysis
Frequency: Quarterly
Suggested Time: Anytime
Participants: Operations, Engineering, QA, Product, Support
Agenda: Single Point of  Failure Analysis(In the book, these meeting would be discussed in more detail)

I cannot emphasize enough the importance of these meetings. Not only do they facilitate the smooth operations of the company, but they also foster better relations between the company’s groups.

Introduction to the book on SaaS Service Operations

"You can't handle the truth!" (Col. Jessep in 'A Few Good Men')

(Note: This article is part of the STORM™ methodology)

As I have mentioned in a previous post, I am working my way through writing a book on SaaS Service Operations. Using the web as a collaborative tool, I have decided to share my work, bit by bit (three chapters, so far) to test it within the community and get live feedback from those who matter, potentially those that would read and recommend it.
Following is the (draft) introduction chapter. I would dearly appreciate your feedback on content, style, typos, grammar and whether you might find such a book an interesting read.
My initial thoughts about the title are along the lines of 'Survival Guide' or 'A day in a SaaS Emergency Room'.
I am not fishing for compliments - it will beat the purpose, and yes, I can handle the truth.
Many thanx,
Dani

Introduction – or Why am I Writing This Book.

Well, someone has to write it. Numerous words have been exhausted over the years on matters SaaS, but I have seen very little being written about SaaS Service Operations, and there are no books on this subject that I am aware of.

As SaaS is becoming mainstream, it has also become the most visible and mature service in the Cloud stack. Consumer expectations have elevated such that they are demanding fast response times and a service that delivers on the availability slogan of ‘anytime-anywhere’. These expectations do not refer only to the application; but also it is expected of the customer and professional services as well. SaaS companies often excel when it relates to the first ‘S’ of SaaS, i.e. Software, but fair quite poorly with regards to the second ‘S’ – Service.

What started as an experiment of the few and the brave, will soon become the major force in the software market, and what will differentiate one company from the rest is no longer the on-demand allure or the feature set, but the level of service it provides.

I am a war veteran in this respect and have many scars to parade. There are probably very few mistakes that I have not made. Being a descendant of Homo Sapiens Sapiens, I like to think of myself as one who has learned from his mistakes and taken steps to remedy them.

‘Operational Fatigue’ is a term I coined after the umpteenth time I was awoken in the wee hours of the morning to handle an outage that occurred yet once again, after having seemingly fixed the problem two weeks prior. I could have just as well created this phrase after the two hour scheduled downtime to upgrade the service. The upgrade turned into a nine hour nightmare that was finally resolved (a couple of minutes before our major customers started their workday) by some engineering heroics. As always, these were followed by heart wrenching phone calls to the CEOs of our customers to explain what went wrong (again) and why it would not repeat.
No wonder I grind my teeth at night.

Throughout my years of practice in this space I have discovered a number of traits across the industry:

• Most SaaS companies are structured and behave in a similar fashion
• Most SaaS companies lack the discipline, the tools and the practices to provide an efficient and effective service operation
• Most SaaS companies, therefore, end up paying the price of not meeting their SLAs, which leads to customer dissatisfaction, customer churn and ‘Operational Fatigue’

The intended audience for this book is whomever is responsible for the quality of customer service. That includes the CEO, the CTO, VP Engineering, VP-Director-Manager of Operations and VP-Director-Manager of customer service. All of these functions must work in unison to ensure a smooth operation both outwardly and internally.

This book is divided into four sections:

1. The first section introduces concepts about SaaS, the evolution of the market and why the model is here to stay. Enough has been written about the subject so I will stick to some of my observations without going into a long dissertation.
2. The second section contains insights on service operations in an SaaS company. It includes various posts published on my blog (‘Dani’s Perspective on SaaS’), over the past year. It discusses typical SaaS operations, discipline, transparency, outsourcing in the Cloud, metrics, inter-department communications, etc.
3. The third section covers Operational Support Systems that might or might not be supported by the product. They include: Billing, On-Boarding, De-provisioning, Integration, Retention Policy, Communication and more.
4. The final section is instructional and lays out the principles of my adaptation of ITIL for SaaS Service Operations™ . It explains what ITIL is and why I chose ITIL as a basis for defining the practices of running an efficient and effective service operation. It covers six practices that I have developed and refined throughout the years at various companies with whom I worked either as an employee or as a consultant.

By following the practices, following the workflows and deploying the tools outlined in this book, SaaS companies can instill the discipline needed to reap the benefits in a surprisingly short time.

It is not complicated, it is not expensive, nor is there sorcery involved - it only requires awareness and leadership.

Cloud IaaS: Sorry, not very Interesting

“There is an incessant influx of novelty into the world, and yet we tolerate incredible dullness” – Henry David Thoreau

Don’t get me wrong. Infrastructure-as-a-Service is a wonderful, useful and logical development. I do not need to sing the praise of it here. I believe in it and I am sure that it will provide a growing, significant percentage of computing needs around the globe.

But, it is just not very interesting, although it is the rage in all IT circles and hype generators. The technologies that enable it are basically: high speed bandwidth, virtualization and sophisticated management software. Now, I do not belittle these technologies. They are the product of years of development of ingenious engineers and some fast acting companies that had the ability to put one and one together and come up with the offering. And kudos to Amazon Web Services on leadership, ideas and execution.

Still, I believe that it is the domain of the few, and although every datacenter and ISP out there is starting to offer a ‘cloud’ solution, the end result will be a few very large companies that are big enough to invest in a model that makes economic sense and are sophisticated enough to pull it through.
So what does that say for technological companies that are thinking of providing IaaS-enabling software or hardware? There will survive only a handful of those companies, since they will be competing in such a small market.

So why is it such a hype, and why is it burning like a bushfire in the Kalahari savannah, while it took almost a decade for SaaS to become mainstream? Because the idea of IaaS is very simple and straightforward. IT gets it. Any old CIO can understand the concept, because hardware is a commodity and has been for a long time. Because many enterprises have been hosting in co-los for decades, acting as if their hardware is in their datacenter.
Once you get over the fear of losing control and get through the blah-blah of security, the idea of IaaS is very simple, and therefore, not interesting.

SaaS on the other hand is all about Applications. And applications are not perceived as a commodity (although many of the non-core applications are beginning to assume that role – and that’s a good thing). Therefore, once the hype will run its course and the dust Clouds will settle, IaaS will become mainstream. Every enterprise will choose how much of its infrastructure will lay outside of its firewalls and to what extent it will use the flexibility of the solution. SaaS will still be the interesting item, since every ISV will offer an on-demand solution, and the competition will continue to generate innovation and breakthroughs.

SaaS 70 – Nextgen Certification for On-demand companies

“A certified lunatic is certified nonetheless” (Dani, 2009).

I was asked by one of my readers (note the plural) to include a chapter on SAS 70 in my upcoming book on SaaS Service Operations. I must admit that I was not sure if he was advocating SAS 70 or he wanted me to discuss certifications for SaaS, since I am not a fan of the former but a promoter of the latter.

Confusion is defining the SaaS market when it comes to certification.
Enterprise IT personnel certainly do not know what questions to ask, so they generate these long RFPs that are very similar to the on-premise RFPs, and they slap on top of it security questions that make their CSO officer feel important with a multitude of acronyms that are either relevant or not. Most on-demand ISVs wouldn’t know how to define a ‘certified’ SaaS either.

The good news is that the customer base is demanding assurances. While a few years back, the concerns were mostly security and mostly compared to on-premise solutions, the market is maturing and now there are a myriad on-demand solutions for every vertical or horizontal aspect of applications.
So how does an IT professional distinguish between the good and better solutions? How can she judge whether the SaaS provider will stand up to its SLAs, whether the data is secured and operational procedures exist and are followed?

SAS 70
The truth is, there are no authoritative answers to these questions nowadays. With a glowing lack of SaaS certification the only default out there is SAS 70.

Statement on Auditing Standards No.70 (SAS 70) is an internationally recognized auditing standard developed by the American Institute of Certified Public Accountants (AICPA) in 1992. It is used to report on the "processing of transactions by service organizations", which can be done by completing either a Type I or a Type II audit. A SAS 70 Type I is known as "reporting on controls placed in operation", while a SAS 70 Type II is known as "reporting on controls placed in operation" and "tests of operating effectiveness" (http://www.sas70.us.com/what-is/definition-of-sas70.php)

(Disclosure: I have not undergone a SAS 70 audit in the companies I worked for. My knowledge is based on reading and sharing other companies’ experiences)

What’s good about SAS 70
The fact that SaaS companies want to take the extra (expensive) step to distinguish themselves from the rest of the pack, shows a level of maturity and seriousness about their business. SAS 70 requires that you have a set of practices and that you are following them.
This in itself is a big step forward for most SaaS companies – they actually have a set of defined practices.
Sorry, only two short paragraphs on the benefits.

The shortcomings of SAS 70
This audit was not defined for SaaS. It was developed in 1992, years before even ASPs were in vogue. It is a general audit for service organizations and covers a wide range of businesses, from credit processing, to medical insurance and data processing.
There are no specifics for an on-demand software company. Heck, there are no specifics for a software company either.

Please note the language “A SAS 70 audit helps companies meet regulatory compliance…”, and “a SAS 70 audit provides an additional layer of accountability…”
Nowhere does is state that it certifies the company at any level, other than the fact that the audit was done.
It reminds me of cosmetic advertizing “makes your skin feel younger” – how very scientific.
There are no recommendations, no standards to meet, no right or wrong. It merely states that you have practices (good or bad) in place, and that you are following them.

As mentioned, the mere fact that there are defined practices exhibits a level of maturity, so I do not belittle the exercise, but there are no provisions in SAS 70 to avoid documenting your bad practices and following them through.

SaaS 70
There is a dire need for a certification program for SaaS companies as the domain matures and SaaS becomes a major component of IT.
IT wants to know that you are a competent service operator, that you are running a tight shop and that the service will be around next Thanksgiving.
I am suggesting a certification program, currently named SaaS 70 (to demonstrate my famous wit), which includes three elements:

• Service Operational Maturity – Has the company defined and implemented practices and procedures for running a robust operation, to ensure that SLAs are met? This would include Change Mgmt, Release Mgmt, Incident mgmt, Event Mgmt, Availability Mgmt, On-boarding, de-provisioning, integration, data retention, etc.

• Security – covering all aspects of password policies, data separation, vulnerability testing, virus protection, privacy, etc.

• Service Continuity – examining the financial viability of the company and what plans are in place to continue providing the service even if the ISV goes belly-up.

Within each component the company will score a level of maturity beyond a pass/fail that comprises coverage, depth, documentation, and tools. And, of course, the report will include recommendations for improvement and scaling up the maturity ladder.

Only with such a specific, SaaS-centric, verifiable and accountable program, will the consumer of these on-demand services know that a company can or cannot meet their expectations.

Discipline (or lack thereof) and Operational Fatigue

“Half of life is luck; the other half is discipline - and that’s the important half, for without discipline you wouldn’t know what to do with luck”- Carl Zuckmeyer

Creative and nonconformists
SaaS companies are mostly composed of a group of highly capable software engineers. These techies are, by nature, creative, imaginative, out-of-the-box engineers, inventing new ideas or new ways of achieving better results. They tend to adopt the latest and greatest technologies and are always looking forward to the next best thing.
Naturally, these engineers are nonconformists and not inclined to follow rules or to stick to routine.
Almost always, they do not come from an enterprise IT environment, where rules and regulations are stricter and operational practices are followed almost religiously.
With the nascent state of SaaS, if the engineers have prior experience, it would mostly come from on-premise, product companies that emphasize features, versatility and usability.
They rarely had to deal with customers, and bugs that were found were handled according to their priority to be fixed in the next release (which could be months away).
Therefore, typical SaaS engineers lack the necessary discipline to run a 24X7 service, and are usually hostile to restrictions imposed on them.

What, me worry?
The lack of discipline manifests itself mainly in Change Management and consequently in Asset Management and, then, consequently in Incident Management.
This refers to what changes are allowed to be done when (‘hey, just to let you guys know, I installed the new patch during lunch break’), how are they approved and communicated (‘yeah, no prob, I tested the code on my laptop – it is foolproof, just a small change in the parsing engine’) how they are recorded and rolled back if necessary (‘don’t worry, I keep all changes in a dedicated notepad on my machine’).
There usually are no rules about touching production. Typically, every engineer has full SUDO access to all servers in the data center, using a single super-user login, so that activities cannot be traced to any specific person.
One-offs can be installed on a particular server and not be documented. Months later when a new version is installed or a server replaced, things fail to work and it may take hours for someone to remember that a special component is not functioning any more.
Lack of a fully functional staging environment may cause an engineer to ‘temporarily test’ some feature on a production machine that either causes service disruption or is forgotten until the fan turns brown.


Operational Fatigue
Operational Fatigue is a term I coined after years in the trenches, of waking up at 3:00 AM to deal with the same problem that hit us three weeks ago; of the stress of dealing with an incident at peak time when Management is hysterical, when Sales are complaining, when Support is overwhelmed with frustrated customers; of making the calls to the high profile customers, explaining, apologizing, promising; of having to explain to the Board why we lost so many customers this quarter.
It gets to you. You discover new gray hair and develop a fear of answering the phone.

The point is – it is avoidable. Instilling the practices and discipline can make life so much easier and allow the ops team to plan and improve instead of fighting fires all the time.

Educating the young
Like toddlers, engineers crave for guidance and discipline, but as most parents would testify, they will make every attempt to break the rules and stretch the envelope to test the boundaries of their environment. Experienced parents will tell you that the young children feel much more secure when they know the rules and when the rules are being enforced. It has been my experience that when I introduced a new set of regulations such as in Change Management, there is always an initial push-back, mumbling about bureaucracy and attempts to circumvent the rules in the beginning. But I have always seen a quick adoption of the new regulations, followed by a realization that life would be so much better if we only stick to the rules – these guys are smart, you know. Many a disaster was avoided by playing the game by the new rules and I found out how quickly the engineers embraced the discipline and started devising ways to improve on and automate the processes.

Just do it!
I recently participated in a round table hosted by HP on the subject of Change Management. Most of the participants were from large IT shops and were talking about adapting to new Change Management processes in terms of six to twelve months. I was astonished. I concede that my background has been with much smaller groups, and I had the full backing of the executive management, but twelve months? Jeez!

The process in my experience was:
· Prepare the documents, templates and work-flows.
· Make a compelling Power Point presentation.
· Present to the Engineering, Ops and Support groups.
· Emphasize the consequences of not following the practice (genitalia hanging at high altitude)
And Voila - It works! A few weeks later you have a spiritual following of admirers, because the fruits of the labor are so obvious in a very short time.