Elastic Beanstalk: CloudWatch Logs setup

Elastic Beanstalk configuration for CloudWatch Logs

CloudWatch Logs is an AWS service to collect and monitor system and application logs. At the top level, the setup is this:

install the CloudWatch agent to collect log data and send it to the CloudWatch Logs service
define log metric filters to extract useful data, like the number of all errors or information about some specific events
create alarms for metrics to get notifications about logs
make sure that the instance role has permissions to push logs to CloudWatch

All the configuration can be done using the Elastic Beanstalk config. In this case, when a new environment is launched or an existing environment is updated, the CloudWatch Logs setup is done automatically.

There is an example of the configuration in the Elastic Beanstalk docs: Using Elastic Beanstalk with Amazon CloudWatch Logs.

The Setting Up CloudWatch Logs Integration with Configuration Files section briefly describes how to use config examples for different environments, but there is no detailed information about the config files. And config files are complex. It is easy to use examples as is, but it is not that easy to do the setup for your own logs. I will start with the review of the Apache access log example and will show how to change it to collect data from the error log as well.

The example for PHP and Python is an archive containing:

cloudwatchlogs-apache/
  cwl-setup.config
  eb-logs.config
  cwl-webrequest-metrics.config

Files need to be placed under the .ebextensions folder. I recommend extracting the archive into .ebextensions/cloudwatchlogs-apache. You can also put the archive file itself, but then it will not be convenient to view/edit configs.

The first two files are generic and can be used as is:

cwl-setup.config (open)

########################################################################
##
########################################################################

Conditions:
  SNSTopicExists : { "Fn::Not" : [{ "Fn::Equals" : [ "", {"Fn::FindInMap" : ["AWSEBOptions", "options", "EBSNSTopicArn"]}]}]}    

Outputs:
  ElasticBeanstalkSNSTopicArn:
    Description: "The SNS Topic ARN for Elastic Beanstalk event notifications. Empty string if none specified."
    Value: { "Fn::FindInMap": ["AWSEBOptions", "options", "EBSNSTopicArn"] }


Resources:
  AWSEBAutoScalingGroup:
    Metadata:
      "AWS::CloudFormation::Init":
        configSets:
          "_OnInstanceBoot":
            "CmpFn::Insert":
              values:
                - CWLogsAgentClearConfig
                - CWLogsAgentConfigSetup
                - CWLogsAgentUpdaterInstallation
                - CWLogsAgentInstallation
          "CWLogsAgentInstallation":
            - CWLogsAgentClearConfig
            - CWLogsAgentConfigSetup
            - CWLogsAgentUpdaterInstallation
            - CWLogsAgentInstallation
          "CWLogsAgentConfigUpdate":
            - CWLogsAgentClearConfig
            - CWLogsAgentConfigSetup
            - CWLogsAgentUpdateConfig
        CWLogsAgentClearConfig:
          commands:
            01-clear-agent-config:
              command: rm -rf /tmp/cwlogs/conf.d
        CWLogsAgentConfigSetup:
          files:
            "/tmp/cwlogs/conf.d/general.conf":
              content : |
                [general]
                state_file = /var/awslogs/state/agent-state
              mode  : "000400"
              owner : root
              group : root
          commands:
            01-setup-agent-config:
              ## Every .conf file in /tmp/cwlogs/conf.d/ will be concatenated together to form the cwl agent config file
              command: |
                cat /tmp/cwlogs/conf.d/*.conf > /tmp/cwlogs/cwlogs-config.conf
        CWLogsAgentInstallation:
          files:
            "/tmp/cwlogs/awslogs-agent-setup.py" :
              source : "https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py"
              mode  : "000700"
              owner : root
              group : root
          commands:
            01-setup-cwlogs-agent:
              ## Install the cwlogs agent - this will also install the cli it needs (in a virtualenv)
              command: |
                setsid /tmp/cwlogs/awslogs-agent-setup.py -n -r `{"Ref" : "AWS::Region" }` -c /tmp/cwlogs/cwlogs-config.conf && exit 0;

        ## Configuration deployment command - update agent if needed
        CWLogsAgentUpdaterInstallation:
          files:
            "/opt/elasticbeanstalk/hooks/configdeploy/enact/00_update_cwlogs_agent.sh":
              content: |
                #!/bin/bash
                . /etc/elasticbeanstalk/.aws-eb-stack.properties
                /opt/aws/bin/cfn-init -v -s "$stack_name" -r "$resource" --region "$region" --configsets CWLogsAgentConfigUpdate > /var/log/eb-cfn-init-call.log 2>&1
              mode  : "000700"
              owner : root
              group : root
        CWLogsAgentUpdateConfig:
          files:
            "/tmp/cwlogs/bin/cwlogs-update.sh":
              content: |
                #!/bin/bash
                log() {
                   echo [$(date -u +"%Y-%m-%d %TZ")] $1
                }
                log "Executing cwlogs agent configuration update"

                if cmp /var/awslogs/etc/awslogs.conf /tmp/cwlogs/cwlogs-config.conf > /dev/null 2>&1; then
                   log "Configuration has not changed - not updating"
                   exit 0;
                fi
                
                log "Copying new cwlogs config to /var/awslogs/etc/awslogs.conf"
                cp /tmp/cwlogs/cwlogs-config.conf /var/awslogs/etc/awslogs.conf;
                log "Restarting awslogs service"
                service awslogs restart;
                log "awslogs service restart complete"
                log "Completed cwlogs agent configuration update"
                exit 0;
              mode  : "000700"
              owner : root
              group : root
          commands:
            01-update-agent-config:
              command: "/tmp/cwlogs/bin/cwlogs-update.sh >> /var/log/eb-cwlogs.log 2>&1"


## Version deployment command - ensure agent is installed
files:
  "/tmp/cwlogs/bin/cwlogs-install.sh":
    content: |
      #!/bin/bash
      log() {
         echo [$(date -u +"%Y-%m-%d %TZ")] $1
      }

      log "Initiating cwlogs agent installation/update"
      . /etc/elasticbeanstalk/.aws-eb-stack.properties
      CONFIG_SET=CWLogsAgentInstallation

      ## just do an update if we think it's already installed
      if [ -f "/var/awslogs/etc/awslogs.conf" ]; then   
        CONFIG_SET=CWLogsAgentConfigUpdate
        log "Running agent configuration update"
      else
        log "Running agent installation"
      fi

      /opt/aws/bin/cfn-init -v -s "$stack_name" -r "$resource" --region "$region" --configsets $CONFIG_SET;
      log "Completed cwlogs agent installation/update"
      exit 0;
    mode  : "000700"
    owner : root
    group : root
    
commands:
  01-ensure-agent-installation:
    command:  "/tmp/cwlogs/bin/cwlogs-install.sh >> /var/log/eb-cwlogs.log 2>&1"

eb-logs.config (open)

##################################################################
##  Sets up the elastic beanstalk log publication to include
##   the admin logs for cloudwatch logs
##################################################################

Resources:
  AWSEBAutoScalingGroup:
    Metadata:
      "AWS::CloudFormation::Init":
        configSets:
          "_OnInstanceBoot":
            "CmpFn::Insert":
              values:
                - EBCWLLogPublicationSetup
        EBCWLLogPublicationSetup:
          files:
            "/opt/elasticbeanstalk/tasks/taillogs.d/cwl-system.conf":
              content: |
                /var/log/awslogs-agent-setup.log
                /var/log/awslogs.log
                /var/log/eb-cwlogs.log
              mode : "000644"
            "/opt/elasticbeanstalk/tasks/bundlelogs.d/cwl-system.conf":
              content: |
                /var/log/awslogs-agent-setup.log
                /var/log/awslogs.log
                /var/log/eb-cwlogs.log
              mode : "000644"
            "/opt/elasticbeanstalk/tasks/systemtaillogs.d/cwl-system.conf":
              content: |
                /var/log/awslogs-agent-setup.log
                /var/log/awslogs.log
                /var/log/eb-cwlogs.log
              mode : "000644"
            "/opt/elasticbeanstalk/tasks/publishlogs.d/cwl-system.conf":
              content: |
                /var/log/awslogs-agent-setup.log    ## this isn't rotated
                /var/log/awslogs.log*.gz
              mode : "000644"

These files will set up the CloudWatch Logs agent on the instance and configure Elastic Beanstalk logs publication to S3.

Note: you also need to enable the access for Elastic Beanstalk instances to CloudWatch Logs: add the appropriate permission for the aws-elasticbeanstalk-ec2-role in IAM settings.

The last config is an example of CloudWatch Logs setup for Apache's access log; adjust it as necessary:

cwl-webrequest-metrics.config (open)

###############################################################################
## Option Settings
##    Namespace: "aws:elasticbeanstalk:application:environment"
##    OptionName: WebRequestCWLogGroup
##       Default:   <EnvironmentName>-webrequests
##    Description: This is the name of the cloudwatch log group for web requests (access log)
##
## To get an SNS alert, add a subscription to the Elastic Beanstalk Notification
##   topic.  (e.g. set your Notificiation Endpoint to your email address)
##
## Metrics
##  Namespace:  ElasticBeanstalk/<EnvironmentName>
##  Metrics:    CWLHttp4xx  CWLHttp5xx
##
## Cloudwatch Alarms
##   CWLHttp5xxCount   - this alarm fires if the number of calls returning
##                        5xx response codes > 10
##   CWLHttp4xxPercent - this alarm fires if the percentage of calls returning
##                        4xx response codes > 10%    
##
##  Note - this demo works with apache containers that are using the default common log format
##      (python and php ruby containers do this - tomcat does not)
###############################################################################


# Apache access log pattern:
## "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\""
## remote_host  remote_logname  remote_user  time_received "request" status response_size "referrer" "user-agent"
Mappings:
  CWLogs:
    WebRequestLogGroup:
      LogFile: "/var/log/httpd/access_log"
      TimestampFormat: "%d/%b/%Y:%H:%M:%S %z"
    FilterPatterns:
      Http4xxMetricFilter: "[..., status=4*, size, referer, agent]"  
      HttpNon4xxMetricFilter: "[..., status!=4*, size, referer, agent]"
      Http5xxMetricFilter: "[..., status=5*, size, referer, agent]"  
      HttpNon5xxMetricFilter: "[..., status!=5*, size, referer, agent]"  


Outputs:
  WebRequestCWLogGroup:
    Description: "The name of the Cloudwatch Logs Log Group created for this environments web server access logs. You can specify this by setting the value for the environment variable: WebRequestCWLogGroup. Please note: if you update this value, then you will need to go and clear out the old cloudwatch logs group and delete it through Cloudwatch Logs."
    Value: { "Ref" : "AWSEBCloudWatchLogs8832c8d3f1a54c238a40e36f31ef55a0WebRequestLogGroup"}


Resources :
  AWSEBCloudWatchLogs8832c8d3f1a54c238a40e36f31ef55a0WebRequestLogGroup:    ## Must have prefix:  AWSEBCloudWatchLogs8832c8d3f1a54c238a40e36f31ef55a0
    Type: "AWS::Logs::LogGroup"
    DependsOn: AWSEBBeanstalkMetadata
    DeletionPolicy: Retain     ## this is required
    Properties:
      LogGroupName: 
        "Fn::GetOptionSetting":
          Namespace: "aws:elasticbeanstalk:application:environment"
          OptionName: WebRequestCWLogGroup
          DefaultValue: {"Fn::Join":["-", [{ "Ref":"AWSEBEnvironmentName" }, "webrequests"]]}
      RetentionInDays: 14


  ## Register the files/log groups for monitoring
  AWSEBAutoScalingGroup:
    Metadata:
      "AWS::CloudFormation::Init":
        CWLogsAgentConfigSetup:
          files:
            ## any .conf file put into /tmp/cwlogs/conf.d will be added to the cwlogs config (see cwl-agent.config)
            "/tmp/cwlogs/conf.d/apache-access.conf":
              content : |
                [apache-access_log]
                file = `{"Fn::FindInMap":["CWLogs", "WebRequestLogGroup", "LogFile"]}`
                log_group_name = `{ "Ref" : "AWSEBCloudWatchLogs8832c8d3f1a54c238a40e36f31ef55a0WebRequestLogGroup" }`
                log_stream_name = {instance_id}
                datetime_format = `{"Fn::FindInMap":["CWLogs", "WebRequestLogGroup", "TimestampFormat"]}`
              mode  : "000400"
              owner : root
              group : root

      
  #######################################
  ## Cloudwatch Logs Metric Filters

  AWSEBCWLHttp4xxMetricFilter :
    Type : "AWS::Logs::MetricFilter"
    Properties :
      LogGroupName: { "Ref" : "AWSEBCloudWatchLogs8832c8d3f1a54c238a40e36f31ef55a0WebRequestLogGroup" }
      FilterPattern : {"Fn::FindInMap":["CWLogs", "FilterPatterns", "Http4xxMetricFilter"]}
      MetricTransformations :
        - MetricValue : 1
          MetricNamespace: {"Fn::Join":["/", ["ElasticBeanstalk", {"Ref":"AWSEBEnvironmentName"}]]}
          MetricName : CWLHttp4xx

  AWSEBCWLHttpNon4xxMetricFilter :
    Type : "AWS::Logs::MetricFilter"
    DependsOn : AWSEBCWLHttp4xxMetricFilter
    Properties :
      LogGroupName: { "Ref" : "AWSEBCloudWatchLogs8832c8d3f1a54c238a40e36f31ef55a0WebRequestLogGroup" }
      FilterPattern : {"Fn::FindInMap":["CWLogs", "FilterPatterns", "HttpNon4xxMetricFilter"]}
      MetricTransformations :
        - MetricValue : 0
          MetricNamespace: {"Fn::Join":["/", ["ElasticBeanstalk", {"Ref":"AWSEBEnvironmentName"}]]}
          MetricName : CWLHttp4xx

  AWSEBCWLHttp5xxMetricFilter :
    Type : "AWS::Logs::MetricFilter"
    Properties :
      LogGroupName: { "Ref" : "AWSEBCloudWatchLogs8832c8d3f1a54c238a40e36f31ef55a0WebRequestLogGroup" }
      FilterPattern : {"Fn::FindInMap":["CWLogs", "FilterPatterns", "Http5xxMetricFilter"]}
      MetricTransformations :
        - MetricValue : 1
          MetricNamespace: {"Fn::Join":["/", ["ElasticBeanstalk", {"Ref":"AWSEBEnvironmentName"}]]}
          MetricName : CWLHttp5xx

  AWSEBCWLHttpNon5xxMetricFilter :
    Type : "AWS::Logs::MetricFilter"
    DependsOn : AWSEBCWLHttp5xxMetricFilter
    Properties :
      LogGroupName: { "Ref" : "AWSEBCloudWatchLogs8832c8d3f1a54c238a40e36f31ef55a0WebRequestLogGroup" }
      FilterPattern : {"Fn::FindInMap":["CWLogs", "FilterPatterns", "HttpNon5xxMetricFilter"]}
      MetricTransformations :
        - MetricValue : 0
          MetricNamespace: {"Fn::Join":["/", ["ElasticBeanstalk", {"Ref":"AWSEBEnvironmentName"}]]}
          MetricName : CWLHttp5xx


  ######################################################
  ## Alarms

  AWSEBCWLHttp5xxCountAlarm :
    Type : "AWS::CloudWatch::Alarm"
    DependsOn : AWSEBCWLHttpNon5xxMetricFilter
    Properties :
      AlarmDescription: "Application is returning too many 5xx responses (count too high)."
      MetricName: CWLHttp5xx
      Namespace: {"Fn::Join":["/", ["ElasticBeanstalk", {"Ref":"AWSEBEnvironmentName"}]]}
      Statistic: Sum
      Period: 60
      EvaluationPeriods: 1
      Threshold: 10
      ComparisonOperator: GreaterThanThreshold
      AlarmActions:
        - "Fn::If":
            - SNSTopicExists
            - "Fn::FindInMap":
                - AWSEBOptions
                - options
                - EBSNSTopicArn
            - { "Ref" : "AWS::NoValue" }

  AWSEBCWLHttp4xxPercentAlarm :
    Type : "AWS::CloudWatch::Alarm"
    DependsOn : AWSEBCWLHttpNon4xxMetricFilter
    Properties :
      AlarmDescription: "Application is returning too many 4xx responses (percentage too high)."
      MetricName: CWLHttp4xx
      Namespace: {"Fn::Join":["/", ["ElasticBeanstalk", {"Ref":"AWSEBEnvironmentName"}]]}
      Statistic: Average
      Period: 60
      EvaluationPeriods: 1
      Threshold: 0.10    
      ComparisonOperator: GreaterThanThreshold
      AlarmActions:
        - "Fn::If":
            - SNSTopicExists
            - "Fn::FindInMap":
                - AWSEBOptions
                - options
                - EBSNSTopicArn
            - { "Ref" : "AWS::NoValue" }

The config file consists of several sections - 'Mappings', 'Outputs', 'Resources'. There are cross-references between these sections, for example, filters defined in the 'Mappings' section are used later in the 'Resources' section for metric filters configuration.

Unfortunately, there is no complete description of this config format in the Elastic Beanstalk documentation. As I understand, this is actually a CloudFormation template, but written in YAML (Elastic Beanstalk config format) instead of JSON (regular CloudFormation template format).

General template structure and information about its sections can be found here: CloudFormation - Template Anatomy.

Let's examine the cwl-webrequest-metrics.config file. The first section is 'Mappings'; here it is:

# Apache access log pattern:
## "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\""
## remote_host  remote_logname  remote_user  time_received "request" status response_size "referrer" "user-agent"
Mappings:
  CWLogs:
    WebRequestLogGroup:
      LogFile: "/var/log/httpd/access_log"
      TimestampFormat: "%d/%b/%Y:%H:%M:%S %z"
    FilterPatterns:
      Http4xxMetricFilter: "[..., status=4*, size, referer, agent]"
      HttpNon4xxMetricFilter: "[..., status!=4*, size, referer, agent]"
      Http5xxMetricFilter: "[..., status=5*, size, referer, agent]"
      HttpNon5xxMetricFilter: "[..., status!=5*, size, referer, agent]"

Here we see some definitions like the log file path (LogFile), log timestamp format (TimestampFormat) and filter patterns (Http4xxMetricFilter, HttpNon4xxMetricFilter, …). These definitions work as constants defined at the top of the template and are referred to from other sections of the file.

Filter patterns will be used to set up metric filters for the access log. Here we have patterns which will find all requests with 4XX response code, all requests with non-4XX code, all 5XX responses and all non-5XX responses.

The TimestampFormat setting is used by the CloudWatch Logs agent to get timestamps for log records, so it is important to verify that the format is set correctly. The timestamp format is the same as used by Python's strptime function.

Some more information about the format can be found in the CloudWatch Logs agent setup file: check around the middle of the file for the description of the datetime_format parameter - there is a table with placeholders and examples.

Since the timestamp format is the same as used by Python, it is easy to test it using the Python interpreter. Start Python in the command line and use code like this:

>>> import time
>>> time.strptime('30/03/09 16:31:32.123', '%d/%m/%y %H:%M:%S.%f')

In some cases, it can be complex to set the timestamp format because there can be just no placeholders to express the real format in the log. In this case, try to match at least part of the timestamp.

For example, the Apache error log has a timestamp like Sun May 17 21:59:15.837463 2015. The problem is that there is a fractional part of the second in the middle, before the year, and the official Python docs don't have a placeholder for this case (edit: there is an %f for microseconds, but let's pretend we don't know this).

The pattern I used for the Apache error log is %a %b %d %H:%M:%S (short weekday, short month name, day, hour:minute:second), and it matches only the start of the timestamp and does not include the year. But it works well, and I guess that the CloudWatch agent takes the current year as default.

The next section in the config file is Outputs:

Outputs:
  WebRequestCWLogGroup:
    Description: "The name of the Cloudwatch Logs Log Group created for this environments web server access logs. You can specify this by setting the value for the environment variable: WebRequestCWLogGroup. Please note: if you update this value, then you will need to go and clear out the old cloudwatch logs group and delete it through Cloudwatch Logs."
    Value: { "Ref" : "AWSEBCloudWatchLogs8832c8d3f1a54c238a40e36f31ef55a0WebRequestLogGroup"}

Here we describe the CloudWatch Logs group. A log group is a top-level entity, like "production-apache-errors", "staging-syslog", etc. Inside the group, we have streams, each stream contains log records from some EC2 instance.

The next section is Resources, where we define AWS resources used in our setup:

Resources :
  AWSEBCloudWatchLogs8832c8d3f1a54c238a40e36f31ef55a0WebRequestLogGroup:    ## Must have prefix:  AWSEBCloudWatchLogs8832c8d3f1a54c238a40e36f31ef55a0
    Type: "AWS::Logs::LogGroup"
    DependsOn: AWSEBBeanstalkMetadata
    DeletionPolicy: Retain     ## this is required
    Properties:
      LogGroupName:
        "Fn::GetOptionSetting":
          Namespace: "aws:elasticbeanstalk:application:environment"
          OptionName: WebRequestCWLogGroup
          DefaultValue: {"Fn::Join":["-", [{ "Ref":"AWSEBEnvironmentName" }, "webrequests"]]}
      RetentionInDays: 14

Above is the definition of the log group resource. Notice the usage of "Fn::FunctionName" constructs: the config file also has a mini-language to refer to other sections of the config or to join strings. For example, {"Fn::Join":["-", [{ "Ref":"AWSEBEnvironmentName" }, "webrequests"]]} will take the Elastic Beanstalk environment name and add -webrequests after it, so the log group name will be environment-webrequests.

Or a function call like this (it is used below) {"Fn::FindInMap":["CWLogs", "WebRequestLogGroup", "TimestampFormat"]} will look up a TimestampFormat value in the Mappings config section.

You can find more information on functions here:

The next resource in the Resources section is an autoscaling group:

## Register the files/log groups for monitoring
AWSEBAutoScalingGroup:
  Metadata:
    "AWS::CloudFormation::Init":
      CWLogsAgentConfigSetup:
        files:
          ## any .conf file put into /tmp/cwlogs/conf.d will be added to the cwlogs config (see cwl-agent.config)
          "/tmp/cwlogs/conf.d/apache-access.conf":
            content : |
              [apache-access_log]
              file = `{"Fn::FindInMap":["CWLogs", "WebRequestLogGroup", "LogFile"]}`
              log_group_name = `{ "Ref" : "AWSEBCloudWatchLogs8832c8d3f1a54c238a40e36f31ef55a0WebRequestLogGroup" }`
              log_stream_name = {instance_id}
              datetime_format = `{"Fn::FindInMap":["CWLogs", "WebRequestLogGroup", "TimestampFormat"]}`
            mode  : "000400"
            owner : root
            group : root

Here we describe the autoscaling group and say that during the new EC2 instance initialization (AWS::CloudFormation::Init) the /tmp/cwlogs/conf.d/apache-access.conf file should be created. This is a CloudWatch agent configuration which instructs an agent to collect data from the Apache access log. We also describe the content for this file:

content : |
  [apache-access_log]
  ## We take the file name from the Mappings - CWLogs - WebRequestLogGroup - LogFile
  file = `{"Fn::FindInMap":["CWLogs", "WebRequestLogGroup", "LogFile"]}`
  ## Log group is a reference to the log group resource we defined above
  log_group_name = `{ "Ref" : "AWSEBCloudWatchLogs8832c8d3f1a54c238a40e36f31ef55a0WebRequestLogGroup" }`
  ## Log stream name is an instance id
  log_stream_name = {instance_id}
  ## datetime_format for the CloudWatch agent is also defined in the Mappings section above
  datetime_format = `{"Fn::FindInMap":["CWLogs", "WebRequestLogGroup", "TimestampFormat"]}`

You can find more information about the CloudWatch agent configuration here.

Note: it is not always convenient to have one stream per instance; see the follow-up note on how to set up one stream for all instances.

The next four resources are metric filters. These filters will extract and count messages with specific status codes from the Apache access log.

#######################################
## Cloudwatch Logs Metric Filters

AWSEBCWLHttp4xxMetricFilter :
  Type : "AWS::Logs::MetricFilter"
  Properties :
    LogGroupName: { "Ref" : "AWSEBCloudWatchLogs8832c8d3f1a54c238a40e36f31ef55a0WebRequestLogGroup" }
    FilterPattern : {"Fn::FindInMap":["CWLogs", "FilterPatterns", "Http4xxMetricFilter"]}
    MetricTransformations :
      - MetricValue : 1
        MetricNamespace: {"Fn::Join":["/", ["ElasticBeanstalk", {"Ref":"AWSEBEnvironmentName"}]]}
        MetricName : CWLHttp4xx

AWSEBCWLHttpNon4xxMetricFilter :
  Type : "AWS::Logs::MetricFilter"
  DependsOn : AWSEBCWLHttp4xxMetricFilter
  Properties :
    LogGroupName: { "Ref" : "AWSEBCloudWatchLogs8832c8d3f1a54c238a40e36f31ef55a0WebRequestLogGroup" }
    FilterPattern : {"Fn::FindInMap":["CWLogs", "FilterPatterns", "HttpNon4xxMetricFilter"]}
    MetricTransformations :
      - MetricValue : 0
        MetricNamespace: {"Fn::Join":["/", ["ElasticBeanstalk", {"Ref":"AWSEBEnvironmentName"}]]}
        MetricName : CWLHttp4xx

AWSEBCWLHttp5xxMetricFilter :
  Type : "AWS::Logs::MetricFilter"
  Properties :
    LogGroupName: { "Ref" : "AWSEBCloudWatchLogs8832c8d3f1a54c238a40e36f31ef55a0WebRequestLogGroup" }
    FilterPattern : {"Fn::FindInMap":["CWLogs", "FilterPatterns", "Http5xxMetricFilter"]}
    MetricTransformations :
      - MetricValue : 1
        MetricNamespace: {"Fn::Join":["/", ["ElasticBeanstalk", {"Ref":"AWSEBEnvironmentName"}]]}
        MetricName : CWLHttp5xx

AWSEBCWLHttpNon5xxMetricFilter :
  Type : "AWS::Logs::MetricFilter"
  DependsOn : AWSEBCWLHttp5xxMetricFilter
  Properties :
    LogGroupName: { "Ref" : "AWSEBCloudWatchLogs8832c8d3f1a54c238a40e36f31ef55a0WebRequestLogGroup" }
    FilterPattern : {"Fn::FindInMap":["CWLogs", "FilterPatterns", "HttpNon5xxMetricFilter"]}
    MetricTransformations :
      - MetricValue : 0
        MetricNamespace: {"Fn::Join":["/", ["ElasticBeanstalk", {"Ref":"AWSEBEnvironmentName"}]]}
        MetricName : CWLHttp5xx

Metric filters use filter patterns defined in the Mappings section. Pattern syntax is described here, and it is convenient to test patterns in the AWS console:

Open the CloudWatch service in the AWS console
Click Logs on the left, select the log group on the right
Click the Create Metric Filter button

Now you can select existing log data if you already have it, or just paste some text from the local log file, enter a filter pattern and test it on the log data.

For example, by entering a filter like [timestamp, type = *, ...], you can see what is taken as a timestamp. If we have log data like this:

[Thu May 14 16:26:54.396347 2015] [suexec:notice] [pid 2631] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu May 14 16:26:54.405887 2015] [auth_digest:notice] [pid 2631] AH01757: generating secret for digest authentication ...
[Thu May 14 16:26:54.406397 2015] [lbmethod_heartbeat:notice] [pid 2631] AH02282: No slotmem from mod_heartmonitor
[Thu May 14 16:26:54.407930 2015] [mpm_prefork:notice] [pid 2631] AH00163: Apache/2.4.10 (Amazon) mod_wsgi/3.5 Python/2.7.5 configured -- resuming normal operations

Then the test result for the pattern [timestamp, type = *, ...] will be this:

Line Number $timestamp                       $type               $3       ...
1           Thu May 14 16:26:54.396347 2015  suexec:notice       pid 2631
2           Thu May 14 16:26:54.405887 2015  auth_digest:notice  pid 2631
3           Thu May 14 16:26:54.406397 2015  ...

Here the data is considered space-delimited, so each word becomes a data column. Spaces can be "escaped" with square brackets, so [Thu May 07 07:03:48.655204 2015] will be considered one field.

So for custom application logs, it is better to use square brackets to specify log record fields. For example, it is better to use [2015-05-14 19:00:03] [WARNING] -- the warning message instead of 2015-05-14 19:00:03 WARNING -- the warning message.

Note: in the Mappings section we also specify TimestampFormat, but it has no effect on filters and is only used by the CloudWatch agent.

And finally we define alarms which will watch the metrics above and generate an SNS notification if we have too many 5XX responses (here we count responses) or if the percentage of 4XX responses becomes high (here we calculate the percentage of 4XX responses).

Alarms configuration (click to expand)

######################################################
## Alarms

AWSEBCWLHttp5xxCountAlarm :
  Type : "AWS::CloudWatch::Alarm"
  DependsOn : AWSEBCWLHttpNon5xxMetricFilter
  Properties :
    AlarmDescription: "Application is returning too many 5xx responses (count too high)."
    MetricName: CWLHttp5xx
    Namespace: {"Fn::Join":["/", ["ElasticBeanstalk", {"Ref":"AWSEBEnvironmentName"}]]}
    Statistic: Sum
    Period: 60
    EvaluationPeriods: 1
    Threshold: 10
    ComparisonOperator: GreaterThanThreshold
    AlarmActions:
      - "Fn::If":
          - SNSTopicExists
          - "Fn::FindInMap":
              - AWSEBOptions
              - options
              - EBSNSTopicArn
          - { "Ref" : "AWS::NoValue" }

AWSEBCWLHttp4xxPercentAlarm :
  Type : "AWS::CloudWatch::Alarm"
  DependsOn : AWSEBCWLHttpNon4xxMetricFilter
  Properties :
    AlarmDescription: "Application is returning too many 4xx responses (percentage too high)."
    MetricName: CWLHttp4xx
    Namespace: {"Fn::Join":["/", ["ElasticBeanstalk", {"Ref":"AWSEBEnvironmentName"}]]}
    Statistic: Average
    Period: 60
    EvaluationPeriods: 1
    Threshold: 0.10
    ComparisonOperator: GreaterThanThreshold
    AlarmActions:
      - "Fn::If":
          - SNSTopicExists
          - "Fn::FindInMap":
              - AWSEBOptions
              - options
              - EBSNSTopicArn
          - { "Ref" : "AWS::NoValue" }

More information about the Resources section:

Apache error log setup

To create a CloudWatch Logs configuration for another log file, do the following:

Copy cwl-webrequest-metrics.config and save it under a new name into the same folder
In the Mappings section: change the log file path, timestamp format and filter patterns
In the AWSEBAutoScalingGroup resource: change the config file name: apache-access.conf to my-log-name.conf and change the [apache-access_log] section name in the content
Search for webrequest and replace all occurrences with an appropriate name; do a case-insensitive search to change webrequest, WebRequest, etc.
Review filter patterns, metrics and alarms; these will be different for each log file

For example, a config for the Apache error log can look like this:

Apache error log config in the cwl-weberror-metrics.config (click to expand)

Mappings:
  CWLogs:
    WebErrorLogGroup:
      LogFile: "/var/log/httpd/error_log"
      TimestampFormat: "%a %b %d %H:%M:%S"
    FilterPatterns:
      AllErrorsFilter: "[timestamp, type = *error*, ...]"


Outputs:
  WebErrorCWLogGroup:
    Description: "Apache error log - WebErrorCWLogGroup"
    Value: { "Ref" : "AWSEBCloudWatchLogs8832c8d3f1a54c238a40e36f31ef55a0WebErrorLogGroup"}


Resources :
  AWSEBCloudWatchLogs8832c8d3f1a54c238a40e36f31ef55a0WebErrorLogGroup:
    Type: "AWS::Logs::LogGroup"
    DependsOn: AWSEBBeanstalkMetadata
    DeletionPolicy: Retain     ## this is required
    Properties:
      LogGroupName:
        "Fn::GetOptionSetting":
          Namespace: "aws:elasticbeanstalk:application:environment"
          OptionName: WebErrorCWLogGroup
          DefaultValue: {"Fn::Join":["-", [{ "Ref":"AWSEBEnvironmentName" }, "weberrors"]]}
      RetentionInDays: 14


  AWSEBAutoScalingGroup:
    Metadata:
      "AWS::CloudFormation::Init":
        CWLogsAgentConfigSetup:
          files:
            "/tmp/cwlogs/conf.d/apache-error.conf":
              content : |
                [apache-error_log]
                file = `{"Fn::FindInMap":["CWLogs", "WebErrorLogGroup", "LogFile"]}`
                log_group_name = `{ "Ref" : "AWSEBCloudWatchLogs8832c8d3f1a54c238a40e36f31ef55a0WebErrorLogGroup" }`
                log_stream_name = {instance_id}
                datetime_format = `{"Fn::FindInMap":["CWLogs", "WebErrorLogGroup", "TimestampFormat"]}`
              mode  : "000400"
              owner : root
              group : root


  AWSEBCWLAllErrorsFilter :
    Type : "AWS::Logs::MetricFilter"
    Properties :
      LogGroupName: { "Ref" : "AWSEBCloudWatchLogs8832c8d3f1a54c238a40e36f31ef55a0WebErrorLogGroup" }
      FilterPattern : {"Fn::FindInMap":["CWLogs", "FilterPatterns", "AllErrorsFilter"]}
      MetricTransformations :
        - MetricValue : 1
          MetricNamespace: {"Fn::Join":["/", ["ElasticBeanstalk", {"Ref":"AWSEBEnvironmentName"}]]}
          MetricName : CWLAllErrorRecords


  AWSEBCWLAllErrorsCountAlarm :
    Type : "AWS::CloudWatch::Alarm"
    DependsOn : AWSEBCWLAllErrorsFilter
    Properties :
      AlarmDescription: "Application generated an error in error_log"
      MetricName: CWLAllErrorRecords
      Namespace: {"Fn::Join":["/", ["ElasticBeanstalk", {"Ref":"AWSEBEnvironmentName"}]]}
      Statistic: Sum
      Period: 60
      EvaluationPeriods: 1
      Threshold: 1
      ComparisonOperator: GreaterThanThreshold
      AlarmActions:
        - "Fn::If":
            - SNSTopicExists
            - "Fn::FindInMap":
                - AWSEBOptions
                - options
                - EBSNSTopicArn
            - { "Ref" : "AWS::NoValue" }

Note that metrics and alarms are optional, and log data will be sent to CloudWatch Logs even if there are no metrics/alarms. But in this case, it will be necessary to review logs manually and set up metrics and alarms later if needed.

Links

Using Elastic Beanstalk with Amazon CloudWatch Logs (+ config examples)

Amazon CloudWatch Monitoring Scripts for Linux (sample Perl scripts that demonstrate how to produce and consume Amazon CloudWatch custom metrics)

AWS Blog: Store and Monitor OS & Application Log Files with Amazon CloudWatch (manual setup, note about Elastic Beanstalk, information about CloudFormation and OpsWorks)

Logs Monitoring Using AWS CloudWatch (awslogs agent setup via script + metrics / alarms from the AWS console)

Stack Overflow: What's a good way to collect logs from Amazon EC2 instances?

How to use CloudWatch to generate alerts from logs?

Tags: aws