According to ESG research, 28% of enterprise organizations collect, process and analyze substantially more security data then they did two years ago while another 49% collect, process, and analyze somewhat more data than they did in the past (note: I am an ESG employee).
What’s happening here? Well first of all, organizations are collecting more data from traditional sources – system logs, vulnerability scans, network flows, etc. They are also grabbing security data from supplementary security sources like EDR tools, behavioral analytics systems, threat intelligence feeds, etc. Oh, and over the last few years, enterprises started gathering data from IoT devices, public cloud services, SaaS, etc. It all adds up to a growing pile of terabytes of security data.
In theory, all this data acquisition is a good thing. Armed with gobs of security telemetry, security analysts can assess events, prioritize actions, and make decisions based upon real world activities.
Unfortunately, many CISOs I speak with are running face first into a security data scalability wall. Collecting, processing, and storing terabytes of data costs a lot of money while managing growing volumes of data isn’t easy. Security professionals know that the answers to their questions are out there, but they lack the skills, processes, and infrastructure to find them.
In my humble opinion, too many organizations made the tactical decision that more security data was a good thing and subsequently piled more and more data on the SOC. Okay, but now security operations teams are buried in this very data. Ironically, this results in longer threat detection and decision cycles.
What can be done here? Leading organizations are addressing growing security data volumes by:
- Building a security operations and analytics platform architecture (SOAPA). The foundation of SOAPA is a distributed data management layer; a common security data repository built for massive scale, that provides data access to security analytics tools. Distributed data management is well understood by technology vendors like Oracle and SAP but it is only starting to appear in cybersecurity. This is one reason why enterprise data analytics pros like the SAS Institute have been pushing their way into cybersecurity.
- Aligning data sources with processes. Rather than try to plough through piles of data, many organizations are letting security operations processes guide them to what’s most important. This is where security operations automation and orchestration tools play a starring role as they can be used to align workflows with data curation, contextualization, and distribution.
- Moving security data to the cloud. Yeah, I know that security data is sensitive causing many security pros to eschew cloud-based solutions. Well, everything in life is a tradeoff so it may be worthwhile to consider moving security data to the cloud in lieu of the capital and operating cost and complexity of keeping in onsite. SumoLogic has long offered a SIEM in the cloud solution while IBM and Splunk can be deployed in the cloud now as well. Meanwhile, CrowdStrike, FireEye, Kenna Security, and Palo Alto Networks all utilize the cloud as part of their security operations offerings.
- Letting machines do the work for them. Let’s face it, security data volumes have grown well beyond human ability to consume them. This is one reason why AI, cognitive computing, and machine learning are an inextricable part of security operation’s future.
I agree that we need to cybersecurity should be based upon data-drive decisions, but few organizations have the resources or chops of Facebook, Goldman Sachs, or NSA to cope with a growing security data tsunami. Collecting more data was the easy part, now we must become much smarter about how we manage all this security data more effectively.