Monitor & Auto-Recover Repaired EC2 Instances

In the real time We had faced an issue where the memory of the EC2 Instance got Exhausted.But Unfortunately We haven’t implemented any alerting and recovery mechanism for the same.

Due to which some of the critical operations which has been configured in the instance didn’t executed.

Later We came to know about the issue.To overcome such issues in future I have implemented an solution where the health of EC2 Instance will be monitored and the necessary actions will be taken automatically.

In this guide , We will learn how to setup a Health monitoring for the EC2 Instance and recovery of the Instance In case of any failure.

Services Used

• Cloudwatch Alarm
• Cloudwatch Event Rule
• SNS Topic with Subscriber

Configuring Cloudwatch Alarm

To know more about Cloudwatch and How to monitor EC2 Instances using Cloudwatch Alarms have been explained in my previous articles , Links below.

Monitor EC2 Instances Using Cloudwatch Alarms

How to Configure Cloudwatch Event Rule

Okay , Lets go ahead and configure Cloudwatch alarm for the existing EC2 Instance.

Login to EC2 Console , Select Instances , Choose the EC2 Instance

Under Actions –> Cloudwatch Monitoring –> Add/Edit Alarms

Click Create Alarm

First , We need to configure Alerting mechanism , So that We will be alerted with a message about an issue.

So We have to configure an SNS Topic with a Subscriber , If you want to know more about SNS ,

I have published a separate article on, How to Configure SNS Topic with a Subscriber

Else You can create a SNS Topic and add a subscriber on the go , To do so,

Make sure Send a notification to is checked , Click create topic , Give a topic name and

With these recipients , Enter an email address , You will get an confirmation email from Amazon SNS to that email address , Make sure you confirm the subscription.

Check Take the action , You can Let the Cloudwatch alarm to take any of the Below Actions if the Instance is repaired or Unreachable.

Note : Recover this Instance option is available only for the Below Instance types.

A1, C3, C4, C5, C5n, Inf1, M3, M4, M5, M5a, M5n, P3, R3, R4, R5, R5a, R5n, T2, T3, T3a, X1, or X1e

If your EC2 Instance doesn’t fall under one of the above instance type , The auto-recovery option won’t be available. In this case You can choose Restart the EC2 Instance

Choose Status check failed (Any) , For atleast 2 failures for the period of 1 minute.

And then Click Create Alarm

If your Instance is one of the above Instance type , Select Recover this Instance , Give a name for the alarm and click Create Alarm

Now that the Cloudwatch Alarm if Created Successfully.

Configuring Cloudwatch Event Rule

From the previous step , If the EC2 Instance is impaired , It will automatically be recovered using Cloudwatch Alarms and Alert will be sent by SNS.

But the alert sent by SNS won’t be having enough information of the automatic recovery action.

In this case , We need to monitor the AWS Health Events for the EC2 Instances.By doing this , You will get the exact results of the automatic recovery actions.

Login to Cloudwatch Console , In the left pane , Select Events , Click Rules

Choose Create Rule, Under Event Source ,

Select the Event Pattern ,

• Service name –> Health

• Event Type –> Specific health events

Select the Specific service as EC2 , Specific event type category as issue , Specific event type code as

AWS_EC2_INSTANCE_AUTO_RECOVERY_SUCCESS

AWS_EC2_INSTANCE_AUTO_RECOVERY_SUCCESS

On the Right hand side , Under Targets , Click Add Target

In the drop down menu , Select the SNS Topic , Under Topic , Choose the topic name you have created.

Click Configure details , Give a name for the Event Rule and Click Create rule

Once We have everything in Place.

Automatic recovery results will be mailed to the AWS’s Main account as well as the subscribed email address.

The results which we get will through the Mail will have one of the below as a Message Title.

[Auto recovery] Amazon EC2 instance recovery: Success

[Auto recovery] Amazon EC2 instance recovery: Failure

Conclusion

We have successfully configured Cloudwatch Alarm , SNS Topic and Cloudwatch Event Rule to automatically recover the repaired EC2 Instances and sent us an email about the results of the Recovery actions.

Hope It is really helpful for you , Thanks for reading this article.

Please do check out my other publications