Bring Ops back into DevOps

“Ownership is yet another of the endless forms of arrogance engaged in by the lower self.”
― Bryant McGill

Two items caught my attention in the past week.
One was an invitation for a DevOps event in Tel Aviv. Lots of flair, keynote speakers, demos, you name it. On the bottom, underlined, it was written “registration for developers only”.
What? Really?

A few days later I saw a recruitment ad for Director of Operations for an upcoming SaaS company, in Silicon Valley, no less. “… the Director of Operations will report to the VP of Engineering”.
I even read an article (don’t remember where – I should have bookmarked it) that DevOps is great since it returns control over production to Engineering, like in “the good old days”.

It felt like a trip in a time machine that landed me in the early 2000s.

Looking up DevOps  in a Google search, it is all about configuration automation for Release Management, scripting languages as Ruby, courses for developers  and such. No wonder Engineering feels so much at home.

When I took my first job as VP Service Operations in a SaaS company, I had to wrestle Production out of the hands of Engineering. I had to change the passwords to the production servers and wean the developers off the habit of changing the environment when they had a ‘safe and quick’ fix. They went home to their families in the evening. They were not on the 24x7 call list.
It took some time for the VP Engineering to come to terms with the new reality; imagine having to report to him while doing the job.

And yes, we used configuration automation scripts even back then, but they lacked the aura of ‘DevOps’. It was all about discipline, Change Management, Asset Management and careful planning, as covered in my previous post.

Chef, Puppet, Docker and the other mechanics are great tools, but they should enhance DevOps, not replace it. The first principle of DevOps is a culture of collaboration / cooperation / communication. Making the Ops team feel like an outsider is not going to help.

At the end of the day, Ops is left with task of keeping the light on, responding to outages 24x7, and taking the rap for service degradation.

So, just as SaaS is more about the Service than the Software, so is DevOps about the Operations more than Development.

Let the cowboys do what cowboys do best, and bring the ranchers back into the farm.

Hey developers – you’re missing the point of DevOps

DevOps and SaaS Service Ops

“Besides black art, there is only automation and mechanization” - Federico García Lorca

DevOps is not just Automated Configuration
If you Google “DevOps”, it seems synonymous with automated configuration tools. Almost everywhere DevOps is driven by the Development team, not the Operations team. While these tools are an important means of achieving Continuous Integration, Continuous Deployment and Continuous Delivery, and they provide a higher level of software quality that gets delivered to the production environment; they do not ensure the other 90% of the DevOps principles.

DevOps is all about breaking barriers, mutual respect, inter department communications, building operations-able software and working as one team with a common goal. In the world of corporate IT, this sounds almost as fictional as “they lived happily ever after” but in the world of SaaS it is a do-or-die principle.

DevOps tries to bridge between the maverick, out-of-the-box, change-or-die attitude of the Development team and the conservative, no-change policy, if-it-ain’t-broke-don’t-fix-it approach of the Operations team. (see also)

DevOps is not a new thing, just a new term. I have been practicing it since 2000 when I was hired by Mercury Managed Services (MMS) to build their US operations group
(As an aside, MMS, now the “HP-SaaS” division following the acquisition of Mercury by HP, was one of the largest SaaS providers at the time, if not the biggest. It was selling around $100M worth of ‘hosted/managed/multi-tenant’ services annually [2004], quietly, without the noise that SFDC was generating at the time)
Back then it meant a highly collaborative effort between R&D and Ops, and a respect by Engineering for the operations team, their practices and the automated tools they developed.
These principles were further forged when I became VP Service Ops in Contactual and then Transera Communications (both SaaS companies in the call center arena) where downtime and degradation of service were devastating for the business.

The DevOps Components
I have written enough about the various aspects of SaaS Service Operations and I don’t intend to repeat it in this post, but I would like to point out the aspects that relate to DevOps

• Communications & respect

Engineering, remember: You have a single customer, and that is the Operations group, so start treating Ops as you would treat a customer; listen to their needs and requests. Schedule a bi-weekly meeting between R&D and Ops at the highest level and discuss issues and plans. Have QA and Support represented as well.
An important aspect of the success of a functioning DevOps undertaking is to build the initial and leading members of the Ops team Ops team from ex software engineers who can speak with R&D at eye level. This will ensure simpler communications and the needed respect from the developers. (see also)

• Configuration Management and Virtualization

Automation is one of the important principles of a smooth SaaS operation, in order to reduce human error, which is the number one cause of downtime. And replicating environments is a crucial factor to ensure that code that was running smoothly in Dev, in Testing, in Staging will run the same in Production.
So, yes, one does need the virtual servers and the configuration automation tools. There are many of those and different developers swear by their favorite tools. Capistrano, CFEngine, Chef, Puppet, Vagrant, Docker, to name a few, all play an important role in ensuring a smooth transition between environments and reduce human error.
But guess what – things still break down in the production environment, and when that happens, the Operations guys are driving the process and managing the crisis, not the über smart R&D guys.

• Sanctity of the production and Change Management

Developers - Lay off! You have no business nor privileges to get near Production. Respect the sanctity of it and respect the Ops teams when they call the shots.
Perhaps the most important aspect of all practices – if you ensure a robust CM process, the probability of downtime diminishes substantially and the recovery should be fast. The development team has to partake in CM. (see also)

• Incident Management

Without detailing the practice, Engineering must be an integral part of the response team, and should have a representative in the real or virtual 'war room'.

• Product features and architecture

Perhaps it is obvious these days that a SaaS product must be built as a highly available and scalable system. And perhaps automatic on-boarding of customers is supported. But what about the other operational-ready features that always get pushed back in favor of the user driven product requests? The list is long, but features should include Instrumentation, Self-serve, De-provisioning Ops-Console, Billing, Dashboards, Customer Notification, Retention policy and Data migration to name a few. (see also)

To conclude, the principles of DevOps should be driving the success of delivering a useful service to the company’s customers, and they are Automation, Communications and Respect.

Automating SaaS Operations

"The first rule of any technology used in a business is that automation applied to an efficient operation will magnify the efficiency. The second is that automation applied to an inefficient operation will magnify the inefficiency" - Bill Gates

This is an article which I posted at the Nolio blog site as a guest blogger back in 2009. Not only is is relevant today, but perhaps even more so...

The Next Killer App
If I had a great idea for the next killer app (I have, actually) and if I had unlimited funds (I don’t, actually) I would have built the software as an on-demand offering.

I would have spent half my funds on building the operational support systems – provisioning, billing, retention policy, self-service, report generator, etc. The other half would be invested in building instrumentation, redundancy, automation, integration, application level monitoring, silent upgrades, customer notifications, and so on.

The rest of the money (you may wonder about my math, but hey, I’ve got unlimited funds) would go towards building the actual application.

Most SaaS vendors out there (and they are growing fast) have chosen the predictable path of building the application first, and worrying about serviceability later. This is the fastest way of getting to market with low costs. The next step is choosing some viable hosting solution and off we go, offering the world our ever better CRM.

Growth
Many months and dozens of customers later, reality hits with all the issues of servicing the software, rapid growth and dealing with labor intensive tasks that are the humdrum of daily life in a SaaS operation. Provisioning/de-provisioning, configuration changes, customized reports, and the most dreaded – upgrades, task the team as a whole, especially when the product is successful and the number of customers is growing daily.

It is not that SaaS executives, architects and engineers are lacking in any way. On the contrary, they are mostly smart, inventive, and creative and have a deep understanding of their customers’ needs in the specific domain. The problem is that they are product people, not service people. Practically none of them come from IT and cannot envision the life of a service operations engineer.

At this point, automation becomes crucial to the survival of the business.

Whether it is built into the next version (many architectures make this quite difficult) or done externally, automation is needed to reduce costs, physical labor, frustration and mainly, error-prone manual procedures. Repeatability, which is a derivative of automation, is also crucial.

Automation is needed across the board. Be it in setting up a new server, or building a new application instance. It could be a manual procedure regarding provisioning of application resources, or building a seamless upgrade procedure.

Outages happen. How quickly can you recover from a service disruption and ensure that the recovery does not create it own problems? Automation not only provides the routines for quick recovery, but instills a discipline of thinking out the necessary steps, discovering dependencies and planning ahead. An added benefit of automation is that it documents the process so you can go back and review the best and worst of your procedures.

In my next post I will take a closer look at the SaaS Upgrade Nightmare.

SaaS and Automated Application Management

A quick blog this time.

I have been asked by a great new company Nolio, to write a few blog posts for their new blog site.

Nolio automates all key processes needed to service and manage applications across your data center, improving application uptime and quality, while streamlining operations for immediate productivity gains.

I have seen their product and was impressed to the point that I am hooking them up with a number of SaaS companies.

Please read the two blog posts. A third is on the way.