As we all know, things go wrong. That’s why monitoring and alerting are essential topics. Wouldn’t it be nice, if problems in your AWS account would show up in Slack? So you can react quickly while using your favorite messaging tool. In this blog post, you will learn how you can turn CloudWatch Alarms into Slack messages like this:
- Aws Sns Slack Tutorial
- Aws Slack Group
- Aws Slack Channel
- Aws-sns-to-slack-publisher
- Aws Sns Slack Certification
How it works
- The slack-echo- blueprints will help you to write bots that respond to commands; the cloudwatch-alarm-to-slack- blueprints will help you to write bots that emit status reports and notifications. Because you have the ability to give the bots access to any desired AWS APIs, you can interact with your AWS resources in any desired way.
- Step 2: Integrate your Amazon SNS topic with AWS Chatbot. Firstly, Open you slack channel. Open AWS Chatbot console and choose configure client 3. Choose Configure new client, then Slack, then Configure. AWS Chatbot asks for permission to access your Slack workplace, as seen in the following screenshot.
On AWS, everything sends monitoring data (CPU utilization, estimated monthly charges, …) to CloudWatch. In CloudWatch, you define alarms to send a message to an SNS topic if the monitoring data gets out of normal bounds. Finally, you connect a Lambda function to the SNS topic to trigger a function execution. The Lambda function calls the Slack API to send a message. The following figure shows the data flow:
To deploy the components in the figure, you will use the Serverless Application Model (SAM). If you are not interested in implementing this on your own, give our Slack chatbot a try. Never miss an alert from your AWS infrastructure with marbot!
Implementing the Lambda function
You will use Node.js to implement the Lambda function. To send a request to the Slack API, you have to make an HTTPS request. The request module is easy to use, but I wanted a variant of the module that returns promises to avoid callback hell. That’s why I used request-promise-native. The Slack webhook URL is passed in as an environment variable that you define later in the CloudFormation template.
Slack It, SNS Codeship has a nice model for Slacking build status. If you haven’t seen it, here’s a glimpse. This is what we’ll implement: a Slack channel that displays the contents of “AWS Elastic. Chatbot also currently lets you receive messages from SNS in a Slack channel or in an Amazon Chime chatroom. The notification has to originate from a supported service though, which could be CloudWatch, Billing and cost management, CloudFormation infrastructure management, GuardDuty threat monitoring, AWS Health performance monitoring, or Security Hub, which combines GuardDuty and other. As part of the AWS free tier, SNS usage up to one million requests, 100 SMS messages, 1,000 emails, and 100,000 HTTP(S) calls is free for all AWS accounts. This applies even if your AWS account was created more than 12 months ago (some AWS services only offer a free tier for the first 12 months of a new account, but this is not the case for SNS).
Messages delivered from SNS to the Lambda function will look like this:
You need to convert the format into the Slack message format.
Finally, each Lambda function needs a handler function. The handler function takes 3 parameters:
The business context
At Impress, we use AWS SES (Simple Email Service) to send out all our transaction emails from our platform. We make use of the boto3 API for this which we’ve got working rather nicely.
One of the systems AWS has in place to keep bounces and complaints in check is that they monitor the reputation of our mailbox by seeing how many bounces and complaints are there. If there are too many bounces or complaints, they initially put the account under review. Subsequently, if fixes aren’t in place, then they pause sending. You can read more details here: https://docs.aws.amazon.com/ses/latest/DeveloperGuide/faqs-enforcement.html.
We had actually had our account put under review very early on in my journey at Impress when it was just me building things. Back then I’d read https://aws.amazon.com/blogs/messaging-and-targeting/handling-bounces-and-complaints/ and set up a system to post a CSV file to our team slack channel with the emails that are bouncing/complaining so that we could act on them.
However, the above setup broke a few months back when we switched to enabling server-side encryption on SQS/SNS. With other priorities, this got deprioritised to the extent that last week we got notified that our account was under review. This triggered the need to fix our whole monitoring system. In reviewing things online, as seems to be rather normal with AWS, we found a lot of “getting started” instructions and study materials but no single resource that told us “do this!”. So we decided that we should share what we set up in case the next poor soul trying to keep things together in a young startup runs into some trouble.
AWS services and other tools we’ll be using in this post:
We have a couple of things that we will be discussing in this post, so let me define them for those who don’t know.
- AWS Services:
- Simple Email Service (https://aws.amazon.com/ses/) – it’s a service that lets you send emails from your app
- Simple Queueing Service (https://aws.amazon.com/sqs/) – It’s a message queue system that sort of acts as a message broker between services. Keep hold of things until they can be handed over to someone else.
- Simple Notification Service (https://aws.amazon.com/sns/) – It’s a service to send notifications out based on triggers. It’s a sort of glue that links different services together. I wonder if Amazon has done some kind of study on whether putting “simple” in the name helps or gets people annoyed…
- Key Management Service (https://aws.amazon.com/kms/) – Create and manage cryptographic keys that help you ensure that your data is encrypted at rest on your server at all times. I’m not entirely sure how this could be exploited, but it seems like a no brainer today to encrypt in rest and transit whatever we can.
- AWS Lambda (https://aws.amazon.com/lambda/) – These are basically “serverless” compute instances. Think of it as AWS letting you run individual functions in the language that you choose and charging you only for the time it runs and the memory it uses.
- Other tools that we are using:
- Python 3.7: We are using python in our lambda functions but I don’t see any reason.
- Slack: We use slack for our internal communication, so we use Slack and it’s incoming webhooks to post messages as needed on our channel.
First steps in setting up monitoring on the AWS SES dashboard
We set up three pieces in our monitoring system. The bulk of the post will cover the most technically challenging and useful one, but I would recommend that you definitely have the other two sets up as well as they can prove to be pretty useful as well.
Aws Sns Slack Tutorial
Step 1: Enable email feedback forwarding. You can enable email feedback. Over multiple pages, AWS explains how you can do this by clicking on the “view details” on your domain/email address and under notification choose the email feedback forwarding option. Note that AWS sends the bounce/complaint notification to your from address or your reply-to address. This can be a complication. So make sure you can receive emails on this or enable step 2 and 3. More details here: https://docs.aws.amazon.com/ses/latest/DeveloperGuide/monitor-sending-activity-using-notifications-email.html
Step 2: Enable getting stats: You can always check your reputation and your current spam/complaint/bounce/reject count in your SES dashboard. But let’s face it it’s a pain to log in to and you won’t’ end up monitoring this on a daily basis. So we set up something using this to get daily notifications on our reputation over the last 15 days and the timestamps at which issues occurred. I’ve detailed how to send this in the section: Get regular reputation stats on slack.
Step 3: Enable notification on each bounce/complaint: For us what makes the most sense since bounces and complaints are few and far between is for us to have a system that notifies us of each bounce/complaint. I outline how we do this in the section: Getting bounce/complaint notifications on slack
Getting reputation stats on Slack:
Create a lambda function with the following code:
https://github.com/impressai/SESMonitoringTools/blob/master/basic-stats-lambda.py and use an event bridge to call it daily (or at whatever period you need)
The bulk of the code is just to prettify the message that we get when we use the code that is explained in this. The lambda handler is the function that is called when the lambda is called. This calls and gets the send statistics from boto3’s get_send_statistics API endpoint. The rest of the code then breaks this down into a nicer format, timezone and sorting and then posts it into a Slack incoming webhook. A webhook gives you a unique URL to which you can make an HTTP POST request and Slack will post it to the configured channel in your Company’s slack.
Enabling notifications for bounces through Amazon SNS
Aws Slack Group
This is the rather more difficult task and involves multiple pieces that required me to piece together information from different parts of the Internet to get done. So here’s a step by step process for those who are interested:
1. Create a KMS key. Go to key management service in AWS and create a new Key. You’ll have to follow a few steps in selecting who has access to this. Since this is not being used by anyone other than the services, I gave the minimum access necessary to remove and manage it. Beyond that, you have to add the following to the KMS key access policy for SES and SNS to use it in the below steps:
2. Next, go to AWS SNS and create two notification services, once for bounce and one for complaints. At this point, enable encryption and use the key you created above. The rest, use as you see fit. The default access policy is perfectly fine.
Aws Slack Channel
3. Next, go to AWS SQS and create a new queue. Again the defaults worked fine for me, I just enabled encryption using the above key.
4. Now go to your SES dashboard and choose your queues for bounces and complaints.
5. Now, if you send emails using AWS SES test email feature, to complaint@simulator.amazonses.com and bounce@simulator.amazonses.com you should be able to see the queue build-up
Aws-sns-to-slack-publisher
6. As a final step create an AWS lambda function, create a new role that has permission to read from SQS and also to decrypt using the kms key. Policy attached below for reference. You can get the first two automatically by choosing the SQS polling template policy as a basis when creating your lambda role.
7. Use the following python code in your lambda function: https://github.com/impressai/SESMonitoringTools/blob/master/notification-manager.py and configure two triggers. One from each of the SQS. The code is mostly self-explanatory. The Lmabda_handler is called as soon as an element enters the queue in SQS. The handler then reads through all the messages in the queue and processes them. What the code does is reads through each message, sees what kind of message it is, converts it to a decent text format and then forwards it to an incoming webhook on slack.
8. Please note that you have to set the SLACK_WEBHOOK environment variable to be your incoming webhook in the above code. Also for good housekeeping perhaps create a tag for all the resources you create for this project.
If the set up is all correct, you should already receive a couple of notifications on slack because of the bounces and complaints in step 5. Alternatively, send a few more tests to make sure things are working.
Side notes and gotchas:
Aws Sns Slack Certification
- If you want to see the full notification, add a print to see the raw notification when receiving from the queue itself. You can then modify the slack message format to show the information you consider to be important. For example, there is some cleanup still to be done in the “mail” part of the queue notification.
- Set up cloudwatch monitoring and budget alarms to throttle things in case things go wrong. For example, if the code is misconfigured and crashing, then SQS keeps sending the message to Lambda and lambda keeps restarting over and over again and it will probably be costly in the long term if this happens and you don’t notice.