Storage requirements for security event data

Aidan Dewey

To combat the threat of global, sophisticated cyber-attacks, government security professionals deploy a dizzying array of security technologies. These technologies provide information about network and system events to Security Information and Event Management (SIEM) software that collects and analyzes log data to detect attacks.

These systems can generate hundreds of thousands or millions of log items per day. Attackers, meanwhile, can use automated systems to launch assaults with millions of events, which instantly create even more data that must be stored for analysis. This flood of information can expose a little-known Achilles heel in government security systems -- the lack of a powerful, fast and robust data storage system to handle all of the data that must be analyzed.

Storage challenges

The main challenge for these security storage systems is the sheer volume of data they must handle, and the speed with which data can be created. Some agencies may employ thousands of IT devices, which together generate so many individual events that they create very high demands for “input-output operations per second,” known as IOPS, on the storage system. Driven by automated systems throughout an agency’s network, this flow can continue 24 hours a day, seven days a week, with few significant lulls.

An enterprise-scale security event and monitoring system requires a storage infrastructure that is:

• Highly responsive -- Able to predictably support 200,000-plus IOPS (depending on the agency’s specific needs);

• Highly scalable -- Able to grow performance, connectivity and capacity in a linear fashion, as data retention requirements grow;

• Highly flexible -- Able to support the rapidly changing environment of today’s security world;

• Highly reliable -- Able to remain available for constant security monitoring;  and,

• Highly cost-effective.

Maintaining a repository of event information for review and historical analysis requires both high capacity and high performance. Achieving this combination requires selecting the proper storage technologies.

For example, one agency was moving to an information and event management system running on an Oracle database that required roughly 100,000 IOPS. Internal testing showed that the agency’s existing storage vendor’s mid-range storage solution would not meet those requirements. A second vendor proposed a similar, dual-controller storage architecture. While this appeared to have enough disk capacity to meet present demand, the agency determined that it would soon need to buy additional arrays, meaning more cost, more devices to manage, and more time and staff to maintain them. Furthermore, the agency was unsure if that array could process the high IOPS if they added faster drives to it.

Wide striping, virtualization yield better results

Government security planners need storage technologies that can provide the requisite speed without compromising data storage utilization, as many older technologies do. One approach that is getting increasing attention is massive wide striping of the log data across every drive within an array. 

Wide striping avoids the high cost, inefficiencies and bottlenecks found in other attempts to meet the performance needs of government security systems. In order to improve the poor I/O performance of legacy storage arrays, some storage architects will employ “short stroking,” which uses only a small amount of the capacity on a disk drive in order to increase its I/O responsiveness. Because this leaves so much unused space, short stroking forces the agency to buy many more drives than it would otherwise need.

In 3PAR’s implementation of wide striping, data is written instantaneously to all drives at the same time, creating ultra-high, predictable performance for security applications. This provides much higher IOPS without sacrificing utilization, thus providing a lower cost and balancing I/O across multiple disks for the best responsiveness and throughput.

In addition, the use of virtualized storage can create efficiencies that let government agencies eliminate waste and provide “greener” computing. One such technology is thin provisioning, which allocates disk space only when an application actually needs it. While almost every vendor claims to have this technology, there are worlds of difference in various implementations. A robust thin provisioning technology will easily support the scalability, responsiveness, flexibility and cost-effectiveness required by security and information monitoring applications. Truly virtualized storage can also provide significantly better high availability, fault tolerance and load balancing than traditional approaches.

Another flawed yet common practice is the use of relatively low-cost, but low-performance SATA disk storage combined with massive caching. The flaw here is that the cache may never empty and thus become a bottleneck itself. In contrast, the employment of wide striping and autonomic tiering makes effective use of new, faster solid state disks (SSD) with inherently faster I/O rates, in combination with SATA disks. This drives storage costs below traditional fiber channel disks while maintaining the performance that IO-intensive applications need.

Do your research and understand the subtle, but significant differences in the underlying technology. And whichever storage architecture you choose, remember that you can only fight the cyber-attacks you can see, and that finding those attacks means sifting through a constant stream of event data. If you build a storage infrastructure that makes ultra-fast analysis possible -- and insist that your vendor provide third-party benchmarks to back up its claims -- you’ll be able to handle the flood of data created by an orchestrated, malicious attack.