Zen and the Art of Automation Continuance

The new year is a good time to start thinking about automation continuance. Most of us initiate automation projects with a focus on a handful of areas – such as provisioning servers or databases, automating the patch process, and so on. While this kind of focus is helpful in ensuring a successful outcome for that specific project, it also has the effect of reducing overall ROI for the company – because once the project is complete, people move on to other day-to-day work patterns (relying on their usual manual methods), instead of continuing to identify, streamline and automate other repetitive and complex activities.

Just recently I was asked by a customer (a senior manager at a Fortune 500 company that has been using data center automation technologies, including HP/Opsware and Stratavia's Data Palette for almost a year) "how do I influence my DBAs to truly change their behavior? I thought they had tasted blood with automation, but they keep falling back to reactive work. How do I move my team closer to spending majority of their time on proactive work items such as architecture, performance planning, providing service level dashboards, etc.?” Sure enough, their DBA team started out impressively automating over half-a-dozen tasks such as database installs, startup/shutdown processes, cloning, tablespace management, etc., however during the course of the year, their overall reactive workload seems to have relapsed.

Indeed, it can seem an art to keep IT admins motivated towards continuing automation.

A good friend of mine in the Oracle DBA community, Gaja Krishna Vaidyanatha coined the phrase “compulsive tuning disorder” to describe DBA behavior that involves spending a lot of time tweaking parameters, statistics and such in the database almost to the point of negative returns. A dirty little secret in the DBA world is that this affliction frequently extends to areas beyond performance tuning and can be referred to as “compulsive repetitive work disorder”. Most DBAs I work with are aware of their malady, but do not know how to break the cycle. They see repetitive work as something that regularly falls on their plate and they have no option but to carry out. Some of those activities may be partially automated, but overall, the nature of their work doesn’t change. In fact, they don’t know how to change, nor are they incented or empowered to figure it out.

Given this scenario, it’s almost unreasonable for managers to expect DBAs to change just because a new technology has been introduced in the organization. It almost requires a different management model, laden with heaps of work redefinition, coaching, oversight, re-training and to cement the behavior modification, a different compensation model. In other words, the surest way to bring about change in people is to change the way they are paid. Leaving DBAs to their own devices and expecting change is not being fair to them. Many DBAs are unsure how any work pattern changes will impact their users’ experience with the databases, and whether that change will cost them their jobs even. It’s just too easy to fall back to their familiar ways of reactive work patterns. After all, the typical long hours of reactive work shows one as a hardworking individual, providing a sense of being needed and fosters notions of job security.

In these tough economic times however, sheer hardwork doesn’t necessarily translate to job security. Managers are seeking individuals that can come up with smart and innovative ways for non-linear growth. In other words, they are looking to do more with the same team - without killing off that team with super long hours, or having critical work items slip through the cracks.

Automation is the biggest enabler of non-linear growth. With the arrival of the new year, it is a good time to be talking about models that advocate changes to work patterns and corresponding compensation structures. Hopefully you can use the suggestions below to guide and motivate your team to get out of the mundane rut and continue down the path of more automation (assuming of course, that you have invested in broader application/database automation platforms such as Data Palette that are capable of accommodating your path).

1. Establish a DBA workbook with weights assigned to different types of activity. For instance, “mundane activity” may get a weight of say 30, whereas “strategic work” (whatever that may be for your environment) may be assigned a weight of 70. Now break down both work categories into specific activities that that are required in your environment. Make streamlining and automating repetitive task patterns an intrinsic part of the strategic work category. Check your ticketing system to identify accurate and granular work items. Poll your entire DBA team to fill in any gaps (especially if you don’t have usable ticketing data). As a starting point, here’s a DBA workbook template that I had outlined in a prior blog.

2. Introduce a variable compensation portion to the DBAs’ total compensation package (if you don’t already have one) and link that to the DBA workbook - specifically to the corresponding weights. Obviously, this will require you to verify whether DBAs are indeed living up to the activity in the workbook by having a method to evaluate that. Make sure that there are activity IDs and cause codes for each work pattern (whether it’s an incident, service request or whatever). Get maniacal about having DBAs create a ticket for everything they do and properly categorize that activity. Also integrate your automation platform with your ticketing system so you can also measure what kind of mundane activity are being carried out in a lights-out manner. For instance, many Stratavia customers establish ITIL-based run books for repetitive DBA activities within Data Palette. As part of these automated run-books, tickets get auto-created/auto-updated/auto-closed. That in turn will ensure that automated activities, as well as manual activities get properly logged and relevant data is available for end-of month (or quarterly or even annual) reconciliation of work goals and results – prior to paying out the bonuses.

If possible, pay out the bonuses at least quarterly. Getting them (or not!) will be a frequent reminder to the team regarding work expected of them versus the work they actually do. If there are situations that truly require the DBAs to stick to mundane work patterns, identify them and get the DBAs to streamline, standardize and automate them in the near future so they no longer pose a distraction from preferred work patterns.

Many companies already have bonus plans for their DBAs and other IT admins. However they link those plans to areas such as company sales, profits or EBITDA levels. Get away from that! Those areas are not “real” to DBAs. IT admins rarely have direct control on company revenue or spending levels. Such linking, while safer for the company of course (i.e., no revenue/profits, no bonuses), does not serve it well in the long run. It does not influence employee behavior other than telling them to do “whatever it takes” to keep business users happy and the cash register ringing, which in turn promotes reactive work patterns. There is no motivation or time for IT admins to step back and think strategically. But changing bonuses and variable compensation criteria to areas that IT admins can explicitly control – such as sticking to a specific workbook with more onus on strategic behavior – brings about the positive change all managers can revel in, and in turn, better profits for the company.

Happy 2009!

PS => I do have a more formal white-paper on this subject titled “5 Steps to Changing DBA Behavior”. If you are interested, drop me a note at “dbafeedback at stratavia dot com”. Cheers!