Send Application Logs To Cloudwatch

Setting up Log management system and alerting based on the erros is the crucial part in IT.

Amazon Web services has a centralized Log mangement systems known as Cloudwatch, Using which you can ship the server and application logs to cloudwatch and setup an alerting based on the error or log level.

This article guides you through the process of setting up CLOUDWATCH AGENT on the servers to collect and ship the logs to cloudwatch Log group.

Services Used:

EC2 INSTANCES : Instance With The Server And Application Logs.

IAM : Required Permission For EC2 Instance To Ship Logs To Cloudwatch.

Pre-Requisites:

• An IAM role with cloudwatch access policy should be created and attached to the EC2 Instances.

Configuring IAM Role:

Login to IAM console,

https://console.aws.amazon.com/iam/home?region=ap-southeast-1#/home

Before creating an IAM role , We should create a custom policy.

Choose Policies in the left pane , Click Create policy

You will see the following page , remove the existing code and add the below configuration.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:DescribeLogStreams"
    ],
      "Resource": [
        "arn:aws:logs:*:*:*"
    ]
  }
 ]
}

Click Review policy , Give a unique name to the policy

And then click Create policy.

Now We have to create an IAM role with the custom policy we have created.

To create Role , Click Roles in the left pane of the IAM home page,

Click Create role,

Choose EC2 and Next: permissions

Search for the policy name which you have created , Select it

Add a tag if required , Click Review

Give a name to the Role and Click Create role

Adding IAM Role To EC2 Instance:

Now We have to attach the IAM role with the EC2 Instance so that EC2 instance will be able to communicate with the Cloudwatch services , Create Log groups and Log streams.

To attach the Role , Go to EC2 Console,

https://ap-southeast-1.console.aws.amazon.com/ec2/v2/home

Select the EC2 instance , Under Actions , Select Instance Settings , Attach / Replace IAM Role,

Under IAM Role , Select the Role you have created and click Apply.

In the Description of the instance , You should see the IAM Role attached.

Setup Cloudwatch Agent:

SSH into the EC2 Instance , Let’s download the agent setup ,

curl https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py -O

When running the python file , You have to explicitly mention the region of your EC2 instances.

sudo python ./awslogs-agent-setup.py --region ap-southeast-1

In the next setup , It will ask you for Access Key Id and Secret Access key. Provide them and make sure that the Keys have required permissions to perform this operation.

Provide the log file location and the Log group name ,

You should also provide a Log stream name ,

And let the cloudwatch to store the logs in the following timestamps,

And finally , let the cloudwatch agent to send all the logs in the log file, As we might be having some data already present in the log file.

Now We have configured Cloudwatch agent successfully.

And you can manage the cloudwatch agent using the below commands,

service awslogs start

service awslogs status

Now the service is UP and running and You will find the cloudwatch agent file locations.

All the configuration files and start up scripts are stored under /var/awslogs/ folder.

You can edit the logs configuration in /var/awslogs/etc/awslogs.conf

[/var/log/nginx/access.log]
datetime_format = %Y-%m-%d %H:%M:%S
file = /var/log/nginx/access.log
buffer_duration = 5000
log_stream_name = {hostname}
initial_position = start_of_file
log_group_name = Nginx-logs

Once the setup is configured properly , You should be able to see the logs in Cloudwatch Management Console.

Under Logs , Select Log groups , search for the log group name you have provided while configuring cloudwatch agent.

Click the Log group and You should be able to see the log streams which holds all the logs of the server and the applications , Based on your configuration.

You can find all the log events here,

Hope this article helped you to manage the logs in the Centralized Log management console.

Please do check out my other publications.