Analyzing Embedded Metric Format (EMF) Logs

1. Timeseries: EMF Log Events vs All Log Events (5-minute)

Generates a graph comparing EMF logs against the total log volume over time. A 5-minute period bin(5m) is applied. Use bin(1m), bin(1h), or bin(1d) for different granularities.

SQL
stats sum(@message like "_aws" and @message like "CloudWatchMetrics") as EMF_Log_Events, count(*) as Total_Log_Events by bin(5m)

2. Percentage of EMF Logs

Calculates the percentage of log events that are EMF-formatted relative to the total log volume. Useful for understanding how much of your log data is generating metrics.

SQL
stats 
    count(*) as Total_Log_Events,
    sum(@message like "_aws" and @message like "CloudWatchMetrics") as EMF_Log_Events,
    (sum(@message like "_aws" and @message like "CloudWatchMetrics") / count(*)) * 100 as EMF_Percentage

3. Summarize EMF Logs by Dimension

Breaks down EMF events by their dimension values to show how metrics are being derived. Returns the unique dimension combinations generating metrics.

SQL
filter @message like "_aws" and @message like "CloudWatchMetrics"
| stats 
    count(*) as eventCount
    by Environment, uri, method, status
| sort eventCount desc
Environment uri method status eventCount
prod /health GET 200 2
prod UNKNOWN GET 200 1
prod /store POST 200 1

4. Summarize Per-Dimension Cardinality

Returns the total unique values for each dimension independently. Illustrates the cardinality contribution of each dimension to help identify high-cardinality culprits.

SQL
filter @message like "_aws" and @message like "CloudWatchMetrics"
| stats 
    count_distinct(uri) as uniqueUris,
    count_distinct(method) as uniqueMethods,
    count_distinct(status) as uniqueStatuses,
    count_distinct(Environment) as uniqueEnvironments,
    count(*) as totalEMFEvents
uniqueUris uniqueMethods uniqueStatuses uniqueEnvironments totalEMFEvents
4 2 1 1 6

5. Summary by URI

Breaks down EMF events by uri dimension, highlighting which endpoints generate the most metric data.

SQL
filter @message like "_aws" and @message like "CloudWatchMetrics"
| stats count(*) as eventCount by uri
| sort eventCount desc

6. Summary by URI: /health

Calculates the percentage of EMF logs where uri = "/health" versus all other endpoints. Identifies whether health checks are a major contributor to metric volume.

SQL
# Health Check EMF Percentage
filter @message like "_aws" and @message like "CloudWatchMetrics"
| stats 
    count(*) as totalEMFEvents,
    sum(uri = "/health") as healthEvents,
    (sum(uri = "/health") / count(*)) * 100 as healthPercentage,
    sum(uri != "/health") as nonHealthEvents,
    (sum(uri != "/health") / count(*)) * 100 as nonHealthPercentage

7. Timeseries: EMF Health Check Events (5-minute)

Generates a graph of EMF logs with uri = "/health" received every 5 minutes. The granularity can be modified using bin(1m) or bin(1h).

SQL
filter @message like "_aws" and @message like "CloudWatchMetrics" and uri = "/health"
| stats count(*) as healthEMFEvents by bin(5m)
| sort @timestamp desc

8. Filter for EMF Logs

Returns EMF log events exclusively for export or further investigation.

SQL
filter @message like "_aws" and @message like "CloudWatchMetrics"

Usage Notes

  • These queries use the CloudWatch Logs Insights query language (not SQL mode).
  • Replace dimension names (Environment, uri, method, status) with the dimensions defined in your EMF log events.
  • The EMF filter pattern @message like "_aws" and @message like "CloudWatchMetrics" identifies EMF-formatted log events.
  • The bin() function in timeseries queries can be adjusted: bin(1m), bin(5m), bin(1h), or bin(1d).
  • Results can be visualized directly in a CloudWatch Dashboard using Log Insights widgets.