Yes, Virginia, There is a Role for IT in Big Data

Much like cloud and other technology that is used by IT and only later given a catchy, marketable name, IT has been involved in collecting, analyzing, and making decisions based on big data for, well, ever.

Whether it has focused on quality of service, bandwidth management, or performance, IT has collected big operational data and used it to make decisions that improve the quality and security of service delivery. So it was somewhat bemusing to read Joe McKendrick’s blog on big data and discover that most organizations consider data analysis a business function, not an IT function.

Today, 95% of businesses do not consider data analysts a part of their IT staff. Instead, companies are now distributing that expertise to line-of-business groups throughout the company. The majority of respondents (58%) report data management is now embedded throughout their business as a dedicated function.

— Yes, There is ROI in All That Big Data: Survey 

While we certainly don’t give the title “data analyst” to the folks in operations who can glance at a CPU utilization chart and immediately deduce that a memory leak is occurring and causing the system to overprovision resources as a compensatory measure, that’s exactly what these seasoned IT operators are doing – data analysis. It just so happens that they’re focused on operational data, not business data. They’re aggregating across the whole of the data center (and increasingly the cloud) and analyzing operational trends in order to address operational issues like performance, resource consumption, and security as quickly as possible.

Big Operational Data, as it were, has been the foundation of improving operations for as long as log aggregators and inline monitoring solutions have been deployed in the data center. Much in the way we marvel at the ability of old school assembly programmers to peruse a Matrix-like screen full of hex codes and point at two instructions as the root cause of a problem, so do operators today analyze logs and output and charts produced by a menagerie of infrastructure services and rapidly deduce through their analysis from whence a problem originates.

The Convergence of BIG Business and Operational Data

Big operational data has the potential to add significant value to the business. In a world where micro-seconds of delay translate to lost revenue and customers, it is imperative that operations be able to quickly track down the causes of delay and redress. That means analyzing operational data and making decisions to reroute traffic, tweak policies, or turn on services that will improve performance, resource utilization or security.

Additionally, some big data is equally valuable to both IT and the business. Browser, location, identity, device form factor. These pieces of data are pertinent to operations as it enables the codification of policies regarding access and security and to business to understand the habits and preferences of its (potential) customers. This data exists across a variety of systems in the data center (and without) and must be used by both operational and business data analysts alike if organizations are to fully take advantage of big data.

There are plenty of ways in which IT can take advantage of big operational data – from more tailored delivery policies to improving the bottom line through better provisioning of resources and optimizing performance. Doing so, however, requires analysis of the data, either formally or informally. Data analysis, whether of operational or business data, is still analysis.

There is a role for IT operations in big data, and big data has a role in IT operations.