Setting up Log management system and alerting based on the erros is the crucial part in IT.

Amazon Web services has a centralized Log mangement systems known as Cloudwatch, Using which you can ship the server and application logs to cloudwatch and setup an alerting based on the error or log level.

This article guides you through the process of setting up CLOUDWATCH AGENT on the servers to collect and ship the logs to cloudwatch Log group.

Services Used:

EC2 INSTANCES : Instance with the server and application logs.

IAM : Required permission for EC2 instance to ship logs to cloudwatch.

Pre-Requisites:

  • An IAM role with cloudwatch access policy should be created and attached to the EC2 Instances.

Configuring IAM Role:

Login to IAM console,

https://console.aws.amazon.com/iam/home?region=ap-southeast-1#/home

Before creating an IAM role , We should create a custom policy.

Choose Policies in the left pane , Click Create policy

You will see the following page , remove the existing code and add the below configuration.

Cloudwatch
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:DescribeLogStreams"
    ],
      "Resource": [
        "arn:aws:logs:*:*:*"
    ]
  }
 ]
}

Click Review policy , Give a unique name to the policy

And then click Create policy.

Now We have to create an IAM role with the custom policy we have created.

To create Role , Click Roles in the left pane of the IAM home page,

Click Create role,

Cloudwatch

Choose EC2 and Next: permissions

Search for the policy name which you have created , Select it

Cloudwatch

Add a tag if required , Click Review

Give a name to the Role and Click Create role

Adding IAM Role to EC2 Instance:

Now We have to attach the IAM role with the EC2 Instance so that EC2 instance will be able to communicate with the Cloudwatch services , Create Log groups and Log streams.

To attach the Role , Go to EC2 Console,

https://ap-southeast-1.console.aws.amazon.com/ec2/v2/home

Select the EC2 instance , Under Actions , Select Instance Settings , Attach / Replace IAM Role,

Cloudwatch
Cloudwatch

Under IAM Role , Select the Role you have created and click Apply.

In the Description of the instance , You should see the IAM Role attached.

Cloudwatch

Setup Cloudwatch agent:

SSH into the EC2 Instance , Let’s download the agent setup ,

curl https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py -O

When running the python file , You have to explicitly mention the region of your EC2 instances.

sudo python ./awslogs-agent-setup.py --region ap-southeast-1

In the next setup , It will ask you for Access Key Id and Secret Access key. Provide them and make sure that the Keys have required permissions to perform this operation.

Provide the log file location and the Log group name ,

Cloudwatch

You should also provide a Log stream name ,

Cloudwatch

And let the cloudwatch to store the logs in the following timestamps,

Cloudwatch

And finally , let the cloudwatch agent to send all the logs in the log file, As we might be having some data already present in the log file.

Cloudwatch

Now We have configured Cloudwatch agent successfully.

And you can manage the cloudwatch agent using the below commands,

service awslogs start
service awslogs status
Cloudwatch

Now the service is UP and running and You will find the cloudwatch agent file locations.

All the configuration files and start up scripts are stored under /var/awslogs/ folder.

You can edit the logs configuration in /var/awslogs/etc/awslogs.conf


[/var/log/nginx/access.log]
datetime_format = %Y-%m-%d %H:%M:%S
file = /var/log/nginx/access.log
buffer_duration = 5000
log_stream_name = {hostname}
initial_position = start_of_file
log_group_name = Nginx-logs

Once the setup is configured properly , You should be able to see the logs in Cloudwatch Management Console.

Under Logs , Select Log groups , search for the log group name you have provided while configuring cloudwatch agent.

Cloudwatch

Click the Log group and You should be able to see the log streams which holds all the logs of the server and the applications , Based on your configuration.

Cloudwatch

You can find all the log events here,

Cloudwatch

Hope this article helped you to manage the logs in the Centralized Log management console.

Please do check out my other publications.