[AWS security]How to query and analyze VPC flow logs in AWS
overview
I will explain about VPC flow logs in AWS. Then, I will show how to query and analyze VPC flow logs in Cloudwatch Logs. This is the real-world experience working with VPC flow logs that are stored in CloudWatch logs.
VPC Flow Logs
- gives security engineers a history of high-level network traffic flows within entire VPCs, subnets, or specific network interfaces (ENIs)
- outside path of network traffic
- agentless
- logged to AWS CloudWatch Logs
use cases
- security group rule analysis
- monitoring network traffic
- determining traffic direction and patterns on network interfaces
Where?
- For an entire VPC
- For a VPC subnet
- For a network interface in the VPC subnet
What type of resources?
- type of traffic — ALL |REJECTED|ACCEPTED
- Publishing destinations => S3 or CloudWatch logs
Available fields
Traffics that are not captured
- Queries to Amazon DNS
- DHCP traffic
- Windows License Activation traffic
- Metadata and TimeSync traffic
- Traffic to reserved IP address for the VPC router
DEMO
Analyze Flow Logs with CloudWatch Logs Insights
Situation
- launched a set of EC2 instances
- You need to take a look at the VPC Flow Logs to assess
1:On the CloudWatch console, click on the Insights menu item on the left hand column menu, under the Logs section.
2:Click into the search box with the grayed out text Select log group(s) and then select the FlowLogs log group.
3:Click outside of the log group selection area and then click the Run query button, using the default query code, which should be:
fields @timestamp, @message
| sort @timestamp desc
| limit 20
After a few seconds of Fetching data… a table will be populated below. In the table below, CloudWatch Log Insights will show you the latest 20 Log events from your FlowLogs Log group. Notice that you’ll also see a bar chart graph that illustrates data from hundreds or thousands of log entries over a timeline. At the top, it should say Showing 20 of <many-more> records matched.
4:Update the query code to stats count(*) as records by srcAddr | sort records desc and then click the Run query button again. You should now see a list of source addresses with corresponding record counts, sorted in descending order. This shows you which IP addresses are producing the most Flow Log records, and therefore, producing the most network activity, on your VPC.
5: On the right-hand column menu, click on the Queries menu item. Under Sample queries, expand VPC Flow Logs, and then click the Top 10 byte transfers by source and destination IP addresses item. This will show you some sample query code that you can run to help determine the network sources that are utilizing the most bandwidth. Click the blue Apply text under the query code to populate your query text box with this code. The query code above the Run query button should have changed to the following:
stats sum(bytes) as bytesTransferred by srcAddr, dstAddr
| sort bytesTransferred desc
| limit 10
Click the Run query button. After a few seconds of Fetching data… the table below should populate.
7: You now have data on the sources and destinations using the most bandwidth (bytesTransferred) within your VPC. If you click on the top result, it will likely contain a srcAddr that is an Amazon owned public IP address, and a dstAddr that is a VPC private IP address for one of your EC2 instances.