Hey developers – you’re missing the point of DevOps

DevOps and SaaS Service Ops

“Besides black art, there is only automation and mechanization” - Federico García Lorca

DevOps is not just Automated Configuration
If you Google “DevOps”, it seems synonymous with automated configuration tools. Almost everywhere DevOps is driven by the Development team, not the Operations team. While these tools are an important means of achieving Continuous Integration, Continuous Deployment and Continuous Delivery, and they provide a higher level of software quality that gets delivered to the production environment; they do not ensure the other 90% of the DevOps principles.

DevOps is all about breaking barriers, mutual respect, inter department communications, building operations-able software and working as one team with a common goal. In the world of corporate IT, this sounds almost as fictional as “they lived happily ever after” but in the world of SaaS it is a do-or-die principle.

DevOps tries to bridge between the maverick, out-of-the-box, change-or-die attitude of the Development team and the conservative, no-change policy, if-it-ain’t-broke-don’t-fix-it approach of the Operations team. (see also)

DevOps is not a new thing, just a new term. I have been practicing it since 2000 when I was hired by Mercury Managed Services (MMS) to build their US operations group
(As an aside, MMS, now the “HP-SaaS” division following the acquisition of Mercury by HP, was one of the largest SaaS providers at the time, if not the biggest. It was selling around $100M worth of ‘hosted/managed/multi-tenant’ services annually [2004], quietly, without the noise that SFDC was generating at the time)
Back then it meant a highly collaborative effort between R&D and Ops, and a respect by Engineering for the operations team, their practices and the automated tools they developed.
These principles were further forged when I became VP Service Ops in Contactual and then Transera Communications (both SaaS companies in the call center arena) where downtime and degradation of service were devastating for the business.

The DevOps Components
I have written enough about the various aspects of SaaS Service Operations and I don’t intend to repeat it in this post, but I would like to point out the aspects that relate to DevOps

• Communications & respect

Engineering, remember: You have a single customer, and that is the Operations group, so start treating Ops as you would treat a customer; listen to their needs and requests. Schedule a bi-weekly meeting between R&D and Ops at the highest level and discuss issues and plans. Have QA and Support represented as well.
An important aspect of the success of a functioning DevOps undertaking is to build the initial and leading members of the Ops team Ops team from ex software engineers who can speak with R&D at eye level. This will ensure simpler communications and the needed respect from the developers. (see also)

• Configuration Management and Virtualization

Automation is one of the important principles of a smooth SaaS operation, in order to reduce human error, which is the number one cause of downtime. And replicating environments is a crucial factor to ensure that code that was running smoothly in Dev, in Testing, in Staging will run the same in Production.
So, yes, one does need the virtual servers and the configuration automation tools. There are many of those and different developers swear by their favorite tools. Capistrano, CFEngine, Chef, Puppet, Vagrant, Docker, to name a few, all play an important role in ensuring a smooth transition between environments and reduce human error.
But guess what – things still break down in the production environment, and when that happens, the Operations guys are driving the process and managing the crisis, not the über smart R&D guys.

• Sanctity of the production and Change Management

Developers - Lay off! You have no business nor privileges to get near Production. Respect the sanctity of it and respect the Ops teams when they call the shots.
Perhaps the most important aspect of all practices – if you ensure a robust CM process, the probability of downtime diminishes substantially and the recovery should be fast. The development team has to partake in CM. (see also)

• Incident Management

Without detailing the practice, Engineering must be an integral part of the response team, and should have a representative in the real or virtual 'war room'.

• Product features and architecture

Perhaps it is obvious these days that a SaaS product must be built as a highly available and scalable system. And perhaps automatic on-boarding of customers is supported. But what about the other operational-ready features that always get pushed back in favor of the user driven product requests? The list is long, but features should include Instrumentation, Self-serve, De-provisioning Ops-Console, Billing, Dashboards, Customer Notification, Retention policy and Data migration to name a few. (see also)

To conclude, the principles of DevOps should be driving the success of delivering a useful service to the company’s customers, and they are Automation, Communications and Respect.