Being Innovative Everyday

One of the most common problems that I see technical professionals are faced with everyday is what I call the dual standard of many businesses. On the one hand businesses expect technical professionals to perform the tasks assigned to them, yet on the other hand they expect their employees to go beyond what is assigned and do more.

Generally speaking this is what every employee wants to do, but in many cases businesses and business leaders fail to deliver for a variety of reasons. However the challenge to technical professionals is more significant mostly because as with many specialist professions, they have one foot in a world that makes little sense to the business person. In fact many of the more intriguing topics of today are mostly still not well understood by many organizations such as: semantic technology, RDF, NoSQL solutions, cloud acronyms (i.e. PaaS, SaaS, IaaS), search technology, and many others. Often times speaking the native techno-lingo justifies in the minds of business how generally unhelpful and expensive technology groups are. However in truth there is real value to an organization that can be found in the technologies that capture the attention of technical professionals. It is the practical application and benefits that tend to be lost in the discussion.

Most technical folks that I speak to mention everything that cannot be done within their budgets or their means for an organization. In truth there are many opportunities to be had. For example in the midst of the bad economic times, the entire technical staff was ordered to cut back so much so that even paper supplies were under scrutiny. Attempting to acquire hardware and software was pretty much impossible. Hence along comes the solution to use open source technologies to create environments on our existing workstations to create solutions. In my case I created a performance metrics dashboard and infrastructure using nothing more than my desktop, Ubuntu Linux, MySQL, PHP Lucene/Solr, and Java. In essence I created a separate partition on the harddrive, so that I had my corporate desktop/operating system on one partition and my Ubuntu partition on another. With the Ubuntu partition I created a small instance whose purpose was to basically reach out to all the running infrastructure and collect performance data (i.e. memory, CPU, network, disk, logs, event dumps, web logs, database statistics, application information, etc.) and analyze and store them for reporting and decision making. The approach was fairly simple: gather the data in raw form, convert it to semantic information based on rules for general data, write a fairly simple program to store the interesting facts for presentation and search, and create a simplistic data presentation stream that could be exported to Excel (which was the tool of choice for everyone in the company) and at the same time a simple web page people could get to to review the data. Over time as the demands on the application grew, I worked with others on the team to create a small grid using existing workstations so that the data could be accessible without stressing out a single workstation and that I could distribute reasonably well to other workstations in other buildings and regions in case of power failures. Interestingly enough the application ran for several years in this manner.

So while this may of course sound interesting to the technical folks what did this all accomplish to the business? Well for minimal costs, the application provided critical answers to mission critical applications within seconds for the purposes of uptime, problem discovery, capacity and performance for a multi-billion dollar product line serving some 30 million customers. Additionally the application also provided metrics in a manner that could be easily exported to company analysts in their spreadsheets within minutes as opposed to the days to weeks it had previously taken and had far more facts and data than any other existing resource within the company at the time. It became the source with which several thousand employees used to make determinations about products and services for their customers. All for about four thousand dollars in labor.

The upside to both myself and the technical folks that I worked with was that we learned quite a lot about many cutting edge approaches such as primitive dynamic resource allocation, distributed computing, semantic technologies, search engines, distributed data collation and analysis, and many others.

So in the end it was a win-win for both the technical professionals and the business.

Now many professionals that hear this event think that I had the blessing of management or that the internal business processes made this possible. In truth all of what we accomplished were technically not approved by anyone even management but kept in a stealth mode until release. The reason was fairly simple: most business personnel are managers not leaders. So another key set of skills we learned happened to be doing very risky things with no permission. The old saying of "rather ask for forgiveness, rather than permission" worked pretty well in this particular setting.

In the end being innovative is not so much about what you do know as opposed to knowing what you have to do. In my experience, talented individuals will always find a problem to solve that interests them for the benefit of everyone. Even in spite of any hurdle that a company might through in their way.