How to Monitor Cloud Workload Deployments with NetFlow
Gartner estimates that the market for public cloud services surpassed $208B in 2016. This represented about 17% growth year over year. Not bad considering the on-going concerns most cloud customers still have regarding data security. Another particularly interesting Gartner finding is the common practice by cloud customers to contract services to multiple public cloud providers. According to Gartner “[m]ost organizations are already using a combination of cloud services from different cloud providers” (http://www.gartner.com/newsroom/id/3443517). While the business rationale for the use of multiple vendors is sound (e.g., avoiding vendor lock in), the practice does create additional complexity in monitoring activity across an organization’s increasingly dispersed IT landscape.
While some providers support better visibility than others (for example, AWS CloudTrail can monitor API calls across the AWS infrastructure) organizations need to understand and address the visibility problems associated with moving to the cloud regardless of the cloud provider or providers they work with. Unfortunately, the ability to monitor user and application activity, and networking communications from each VM or endpoint in the cloud is limited.
Irrespective of where computing resources reside, organizations must answer the questions of “Which users, machines, and applications are communicating with each other?” Organizations need visibility across the infrastructure in order to:
- • Quickly identify and prioritize issues
- • Speed root cause analysis and identification
- • Lower the mean time to fix problems for end users
- • Quickly identify and eliminate security threats, reducing overall dwell times
Conversely, poor visibility or poor access to visibility data can reduce the effectiveness of existing security and management tools. Organizations that are used to the maturity, ease, and relative low cost of monitoring physical data centers are apt to be disappointed with their public cloud options. What has been missing is a simple, ubiquitous, and elegant solution like NetFlow for public cloud infrastructure.
NetFlow, of course, has had 20 years or so to become a de facto standard for network visibility. A typical deployment involves the monitoring of traffic and aggregation of flows at network chokepoints, the collection and storage of flow data from multiple collection points, and the analysis of this flow data. Flows consist of a basic set of source and destination IP addresses and port and protocol data that is typically collected from a router or switch. Netflow data is relatively cheap and easy to collect and provides nearly ubiquitous network visibility and allows for actionable analysis for both network monitoring and performance management applications. Most IT staffs, especially networking and some security teams are extremely comfortable with the technology.
But NetFlow was created for solving what has become a rather limited problem in the sense that it only collects network data and does so at a limited number of potential locations. To make better use of NetFlow, two key changes are necessary.
- NetFlow at the Edge: First, we need to expand the useful deployment scenarios for NetFlow. Instead of only collecting NetFlow at networking choke points, let’s expand flow collection to the network edge (clients, cloud, and servers). This would greatly expand the overall view that any NetFlow analytics provide. This would allow organizations to augment and leverage existing NetFlow analytics tools to eliminate the growing blind spot of visibility into public cloud activity.
- Rich, contextual NetFlow: Second, we need to use NetFlow for more than simple network visibility. Instead, let’s use an extended version of NetFlow and include information on the user, device, application, and binary responsible for each monitored network connection. That would allow us to quickly associate every network connection back to its source.
In fact, these two changes to NetFlow, are exactly what Ziften has accomplished with ZFlow. ZFlow provides an expanded version of NetFlow that can be deployed at the network edge, including as part of a VM or container image, and the resulting data collection can be consumed and analyzed with existing NetFlow analysis tools. In addition to traditional NetFlow Internet Protocol Flow Information eXport (IPFIX) networking visibility, ZFlow provides greater visibility with the addition of information on user, device, application and binary for each network connection.
Ultimately, this allows Ziften ZFlow to deliver end-to-end visibility between any two endpoints, physical or virtual, eliminating traditional blind spots like east-west traffic in data centers and enterprise cloud deployments.