CloudWatch Log Analytics
Analyzing VPC Flow Logs with Contributor Insights
1. Overview ▾
- Enabling Custom VPC Flow logs on NLB ENI's can provide more visibility into performance issues.
- To identify targets emitted TCP RST's include the
tcp-flagsfield. - In situations where the original source and destination IP addresses are obscured or translated, consider including
pkt-srcaddrorpkt-dstaddrfields. - To allow for consistent analysis across VPCs and accounts, it is recommended that the same VPC Flow Log format be deployed to all resources.
- To identify targets emitted TCP RST's include the
Depending on the custom VPC Flow Logs format, the field positions may need to be updated to obtain meaningful results.
2. Contributor Insights ▾
Fig 1: Using Contributor Insights with Log Analyzer
Contributor Insights returns the top talkers within logs. This can provide a coarse overview of network performance via VPC Flow Logs.
8 Sample Contributor Insights are listed here.
- Accepted Flows by Destination Port & Subnet
- Accept/Rejects by VPC ID
- Data Transfers by VPC ID
- Rejects by Source IP & Port
- Request by TCP Flags
- SSH Traffic by Action & Source IP
- Top Clients Establishing Connections
- Rejects SSH by Source IP
Additional rules to consider:
- Data Transfers via a NLB by Source/Destination Pair
- Data Transfers by AWS Service
- RST Packets by Source/Destination Pair
- HTTP Traffic by Instance ID & Destination IP
3. Sample Rules ▾
Custom Log Format:
TEXT
${account-id} ${vpc-id} ${subnet-id} ${interface-id} ${instance-id} ${srcaddr} ${srcport} ${dstaddr} ${dstport} ${protocol} ${packets} ${bytes} ${action} ${log-status} ${start} ${end} ${flow-direction} ${traffic-path} ${tcp-flags} ${pkt-srcaddr} ${pkt-src-aws-service} ${pkt-dstaddr} ${pkt-dst-aws-service} ${region} ${az-id} ${sublocation-type} ${sublocation-id}
3.1 Top Source/Destination
JSON
{
"Schema": {
"Name": "CloudWatchLogRule",
"Version": 1
},
"LogFormat": "CLF",
"AggregateOn": "Sum",
"Contribution": {
"Filters": [],
"Keys": [
"pkt-srcaddr",
"pkt-dstaddr"
],
"ValueOf": "bytes"
},
"Fields": {
"12": "bytes",
"20": "pkt-srcaddr",
"22": "pkt-dstaddr"
},
"LogGroupNames": [
"CloudWatch_Demo_VPC_Flow_Logs"
]
}
3.2 Top Source/Destination: REJECT
JSON
{
"Schema": {
"Name": "CloudWatchLogRule",
"Version": 1
},
"LogFormat": "CLF",
"AggregateOn": "Sum",
"Contribution": {
"Filters": [
{
"Match": "action",
"In": [
"REJECT"
]
}
],
"Keys": [
"pkt-srcaddr",
"pkt-dstaddr"
],
"ValueOf": "bytes"
},
"Fields": {
"12": "bytes",
"13": "action",
"20": "pkt-srcaddr",
"22": "pkt-dstaddr"
},
"LogGroupNames": [
"CloudWatch_Demo_VPC_Flow_Logs"
]
}
3.3 Top Source/Destination: Filter on NLB ENI's
JSON
{
"Schema": {
"Name": "CloudWatchLogRule",
"Version": 1
},
"LogFormat": "CLF",
"AggregateOn": "Sum",
"Contribution": {
"Filters": [
{
"Match": "eni",
"In": [
"eni-0cadc266d476ae847",
"eni-05c54c575a830536c"
]
}
],
"Keys": [
"pkt-srcaddr",
"pkt-dstaddr"
],
"ValueOf": "bytes"
},
"Fields": {
"5": "eni",
"12": "bytes",
"20": "pkt-srcaddr",
"22": "pkt-dstaddr"
},
"LogGroupNames": [
"CloudWatch_Demo_VPC_Flow_Logs"
]
}
3.4 Top Source/Destination: RST Packets
JSON
{
"Schema": {
"Name": "CloudWatchLogRule",
"Version": 1
},
"LogFormat": "CLF",
"AggregateOn": "Sum",
"Contribution": {
"Filters": [
{
"Match": "protocol",
"In": [
"4"
]
},
{
"Match": "tcp-flag",
"In": [
"18"
]
}
],
"Keys": [
"pkt-srcaddr",
"pkt-dstaddr"
],
"ValueOf": "bytes"
},
"Fields": {
"10": "protocol",
"12": "bytes",
"18": "tcp-flag",
"20": "pkt-srcaddr",
"22": "pkt-dstaddr"
},
"LogGroupNames": [
"CloudWatch_Demo_VPC_Flow_Logs"
]
}
3.5 Top Destination Port
JSON
{
"Schema": {
"Name": "CloudWatchLogRule",
"Version": 1
},
"LogFormat": "CLF",
"AggregateOn": "Sum",
"Contribution": {
"Filters": [],
"Keys": [
"dstport"
],
"ValueOf": "bytes"
},
"Fields": {
"9": "dstport",
"12": "bytes"
},
"LogGroupNames": [
"CloudWatch_Demo_VPC_Flow_Logs"
]
}
3.6 Top Destination Port & Subnet: ACCEPT
JSON
{
"Schema": {
"Name": "CloudWatchLogRule",
"Version": 1
},
"LogFormat": "CLF",
"AggregateOn": "Sum",
"Contribution": {
"Filters": [
{
"Match": "action",
"In": [
"ACCEPT"
]
}
],
"Keys": [
"dstport",
"subnet-id"
],
"ValueOf": "bytes"
},
"Fields": {
"3": "subnet-id",
"9": "dstport",
"12": "bytes",
"13": "action"
},
"LogGroupNames": [
"CloudWatch_Demo_VPC_Flow_Logs"
]
}
3.7 Top Destination AWS Service
JSON
{
"Schema": {
"Name": "CloudWatchLogRule",
"Version": 1
},
"LogFormat": "CLF",
"AggregateOn": "Sum",
"Contribution": {
"Filters": [],
"Keys": [
"pkt-dst-aws-service"
],
"ValueOf": "bytes"
},
"Fields": {
"12": "bytes",
"23": "pkt-dst-aws-service"
},
"LogGroupNames": [
"CloudWatch_Demo_VPC_Flow_Logs"
]
}
3.8 Top Destination IP: Filter on HTTPS/HTTP Traffic
JSON
{
"Schema": {
"Name": "CloudWatchLogRule",
"Version": 1
},
"LogFormat": "CLF",
"AggregateOn": "Sum",
"Contribution": {
"Filters": [
{
"Match": "dstport",
"In": [
"80,443"
]
}
],
"Keys": [
"dstaddr"
],
"ValueOf": "bytes"
},
"Fields": {
"8": "dstaddr",
"9": "dstport",
"12": "bytes"
},
"LogGroupNames": [
"CloudWatch_Demo_VPC_Flow_Logs"
]
}
3.9 Top Instance ID & Destination Address: Filter on HTTPS/HTTP Traffic
JSON
{
"Schema": {
"Name": "CloudWatchLogRule",
"Version": 1
},
"LogFormat": "CLF",
"AggregateOn": "Sum",
"Contribution": {
"Filters": [
{
"Match": "dstport",
"In": [
"80,443"
]
}
],
"Keys": [
"dstaddr"
],
"ValueOf": "bytes"
},
"Fields": {
"5": "instance-id",
"8": "dstaddr",
"9": "dstport",
"12": "bytes"
},
"LogGroupNames": [
"CloudWatch_Demo_VPC_Flow_Logs"
]
}