State of database tools is appalling!

When one thinks of “database tools”, an overly crowded image comprising lots and lots of GUI utilities and point solutions comes to mind. Given all these tools, one would think the database tools market is a mature one such that all that is to be invented in this area is already invented and available. Right? On the contrary, the truth was never further away.

Almost all of the tools in the market today focus on just two things:
1. Monitoring and alerting
2. Administrative GUIs to carry out ad-hoc/one-off tasks.

There are a handful of large companies out there that specialize in this area and they produce all kinds of tools – utilities to point and click and create users, compare schemas, add space, roll out releases, tune SQL, produce wait event graphs and other such performance metrics, etc. But at the end of the day, all of these tools belong to one or both of the above categories in various permutations and combinations. (You name a tool and I will tell you which of the above category it focuses on.) There is no earth-shattering innovation here. Add to this the fact that many DBAs prefer the flexibility of the command-line anyway for the two categories above and many of these tools eventually morph into shelfware grossly under-delivering on the investment made in them.

Don’t get me wrong – these large tools providers do their fair share of R&D (approx. 18 to 28% of their overall revenue, as per last year’s SEC filings of heavyweights BMC, Quest Software and Embarcadero… whoo hoo!). But it looks like most if not all of their R&D is relegated to figuring out better and nicer looking GUIs for monitoring. That’s great! I’m a big fan of usability improvements that result in nice intuitive interfaces, however think of this: if I want to create a user in one database, I might as well use the command-line (unless I’m too lazy to memorize the syntax or look it up in the online SQL Reference guide). If I’m dealing with say, half a dozen databases where I need to create the same user, I would love to use these fancy GUI tools and point and click my way to glory. But what happens if I need to propagate this same new user and privileges to a couple dozen databases, or even, hold your breath - a 100 databases? That would be a lot of pointing and clicking, no matter how easy to use that tool is. Now add to these 100 databases, a variety of mundane/repeatable tasks – such as installs, configs, refreshes and cloning, maintenance plans, patches, upgrades, migrations, managing load processes, DDL releases, and so on. That makes for a very busy DBA team working around the clock and no amount of point solutions and nice GUIs can change that. None of these mainstream tools are even architected for that kind of functionality and task replication.

Now some of these tools vendors are getting smart. They are introducing newer "automation" features. Yaay! But guess what, much of these automation features are built on top of the monitoring functionality they previously had. And they are restricted to certain vanilla automation or allowing a script to be kicked off whenever an alert fires off. In the latter case, the script still has to be written and tested by a DBA. DBAs in complex, heterogeneous environments are already super busy. They don’t have the time to write these complex scripts. The most they may do is look on Google for a particular script and if found, pick it up. But again, they don’t have the bandwidth to test these recently downloaded scripts in all the database environments they support. And oh by the way, the DBA that wrote or downloaded a particular script may be well versed with how it works, but none of the other DBAs in the team are likely to be familiar with it. So now each DBA starts building her own toolbox of scripts. Most of these scripts are not documented in a central location and there is no pre-defined (commonly agreed upon) methodology regarding which script to use in what circumstances. Worse yet, each of these scripts usually need to reside locally on each of the (100) database servers. Even if one line of code needs to change in one script to accommodate a change in the environment, someone needs to manually log into each of the DB servers and make the change. Not a bad approach to follow, if fat-fingering and typos were non-existent!!!

Now you begin to see what many veteran DBAs and IT managers have been seeing for years – the DBA tools industry is in a state of despair without any light at the end of the tunnel. All that’s coming into the tunnel is more monitoring tools – they have become more light-weight, they now carry out extra audit functions, etc. But at the end of the day, all they do is monitoring and ad-hoc administration.

In the meantime, some of the DBMS vendors have been introducing automation capabilities into their core products in recent years. However the bulk of that automation is also vanilla in nature and hence fairly limited in its capability. Let's understand what this "vanilla automation" means. Let's take for instance, Oracle's datafile auto-extend capability. Oracle has had this feature for some time now. However the feature is quite set in its way such that it just makes the datafile grow until it reaches a pre-set limit or until the underlying mount-point/drive gets full. It does not accommodate sys admin policies prevalent in most organizations such as implicit quotas on different mount-point (due to diverse applications and databases sharing the same storage devices as the DBMS) and gracefully growing onto the next appropriate mount-point. In other words, vanilla automation capabilities built into DBMS products and 3rd party tools do not necessarily accommodate custom IT policies and environmental rules. Any kind of custom policies need a script to be built by the DBA and that approach runs into the script-related problems mentioned above. Worse yet, some of these tools require the DBA to learn a proprietary scripting language before they can build any custom automation capability (a notable example is BMC Patrol). Yeah, like they have the time to do that...

Some of the DBMS vendors have also begun to provide DBA tools separate from their core DBMS product. But they are not exactly stellar either! For instance, Oracle has been dutifully releasing its Enterprise Manager product since version 7. It’s “intelligent agent” technology has been difficult to install, was unreliable and would fail often. At times, it would suck up more system resources than the database itself (a la BMC Patrol or an HP Openview DBSPI). I know of an old school DBA that has been burnt by OEM more than once on a production environment. He's sworn never to ever touch OEM with a barge pole no matter how much it evolves! It's just lost all credibility with him and others like him. He feels Oracle should stick to building databases rather than trying to do it all! Anyway, the latest incarnation of OEM, Grid Control is certainly more reliable and powerful than its predecessors. However its power mostly applies just to Oracle and that too, the latest versions of Oracle. For instance, one of the most powerful capabilities of Grid Control is dynamic RAC node provision and configuration for Oracle10g, but that doesn’t work for Oracle8i and Oracle9i. OEM claims to support SQL Server and IBM DB2 as well, but only for monitoring (no automation features are included) and that too for a hefty surcharge! Well, that’s no surprise right? Think about it, why would Oracle make it easy for customers to manage its competitors’ products? A lot of my customers (though they use Oracle databases for some of their applications) think that Oracle, being the crafty company that it always has been, is in the business of selling databases and applications and along the way, cutting off the oxygen for competing products while ostensibly supporting them.

So what’s a customer to do? Continue to participate in the great monitoring rollout from these behemoths and hire human DBAs by the truckloads to do the real work when the pagers go off? Sure, you need high quality DBAs. But merely adding head-count doesn’t scale. Talk to the companies that have 100s of DBAs. They have already tried that approach and are consumed in their own hodgepodge of problems…

There is still hope yet. There are some smaller innovative companies that are rising up to the challenge – namely, companies like Opalis, Appworx, Realops, Opsware (this one isn’t so small any more!) and StrataVia (warning, implicit marketing push, since this is the company I work for) with its Data Palette product. All of these companies are beginning to address the above problems in their particular area of focus – application administration, systems administration, database administration, et al. Take a look at what they have to offer and see if it’s suitable for your organization. Maybe it is, may it is not. But you owe it to yourself and your shareholders to see if you can use their newer offerings to go beyond mere monitoring and ad-hoc administration to actually improve the operating efficiency of your IT environment.

The coming years should be interesting to see which of these innovative companies and products the market adopts and embraces and which ones wither away. But more than anything, it would be intriguing to see how long the 800-pound DBMS tools behemoths continue to survive focusing on improving monitoring and ad-hoc administration alone.