UPDATES

Be taught serverless on AWS step-by-step – SQS




TL;DR

On this sequence, I attempt to clarify the fundamentals of serverless on AWS, to allow you to construct your individual serverless functions. With final article, we found methods to use EventBridge to construct event-driven functions. At present, we’ll dive deeper into occasions administration by having a look at SQS and its integration with lambda capabilities.



Introduction

SQS is Amazon’s Easy Queue Service. As its title suggests, it’s a totally managed queue service, that means that you can retailer messages whereas ready for them to be processed. It’s a very helpful service to decouple your functions, and to construct event-driven functions. It’s also an excellent method to deal with asynchronous duties, and to handle your utility’s load.

On this article, we’ll use SQS to discover a answer to an issue: think about you might have an exterior API that solely permits 1 connection at a time (for instance, to keep away from spamming). How do you stop it to be overwhelmed by your customers, however nonetheless be sure that each person request will likely be finally processed? That is the place SQS is useful!

One in every of SQS use instances is to retailer messages and restrict the throughput of your utility. If a lambda operate processes your messages, you possibly can restrict the variety of concurrent executions of this operate (right here we might set it to 1), and this Lambda operate will course of all of the messages saved within the queue 1 by 1.

Resumed in a small schema it will seem like this:

sqs explained



What are we going to construct?

Based mostly on this use case, let’s construct a faux ordering app, the place a constraint is that only one order will be processed at a time. We are going to use SQS to retailer the orders, and a lambda operate to course of them. Utilizing this technique, there’s a chance {that a} person has to attend a number of minutes earlier than his order is processed: to repair this, we’ll publish an occasion with EventBridge when the order is processed, and the person will likely be notified by an e mail (utilizing SES) when his order is prepared.

The app ought to seem like this as soon as we’re completed:

app architecture



Creating the SQS queue and its goal Lambda operate

As all the time, you’re going to use AWS CDK mixed with TypeScript to provision this utility. When you want a refresher, you possibly can verify the first article of this sequence, the place I’m going deeper into the setup of the venture.

Let’s begin by the core of our utility: the SQS queue and the lambda operate that may course of the orders.

import * as cdk from 'aws-cdk-lib';
import { Assemble } from 'constructs';
import path from 'path';

export class LearnServerlessStack extends cdk.Stack {
  constructor(scope: Assemble, id: string, props?: cdk.StackProps) {
    tremendous(scope, id, props);

    // create a FIFO SQS queue
    const ordersQueue = new cdk.aws_sqs.Queue(this, 'ordersQueue', {
      visibilityTimeout: cdk.Length.seconds(180),
      fifo: true,
    });

    // outlined an occasion supply for the queue, with a batch measurement of 1
    const eventSource = new cdk.aws_lambda_event_sources.SqsEventSource(ordersQueue, {
      batchSize: 1,
    });

    // create a Lambda operate that may course of the orders, bind it to the occasion supply
    const executeOrder = new cdk.aws_lambda_nodejs.NodejsFunction(this, 'executeOrder', {
      entry: path.be part of(__dirname, 'executeOrder', 'handler.ts'),
      handler: 'handler',
      reservedConcurrentExecutions: 1,
      timeout: cdk.Length.seconds(30),
    });
    executeOrder.addEventSource(eventSource);
  }
}
Enter fullscreen mode

Exit fullscreen mode

With this code snippet, you’ll provision a SQS Queue and a Lambda operate. Each message despatched to the queue will set off the lambda operate, and the lambda operate will course of the messages 1 by 1, due to the concurrency (1 execution at a time), and the batch measurement (every message consists of 1 order).

I set a timeout of 30 seconds for the lambda operate (for demo functions, I would like the faux processing to be very lengthy), and the visibility timeout to 150 seconds: AWS recommends to set the visibility timeout to six instances the timeout of your lambda operate, in order that the message is just not processed twice if the lambda operate fails. It is a difficult matter, be taught extra right here.



Provision the remainder of the infrastructure



Non-lambda assets

As seen on the introduction schema, we additionally have to provision an occasion bus, an API gateway and a SES Id. Let’s do it!

import { orderExecutedHtmlTemplate } from './orderExecutedHtmlTemplate';
// ...earlier code

// Provision a relaxation API
const restApi = new cdk.aws_apigateway.RestApi(this, 'restApi', {});

// Provision an occasion bus and a rule to set off the notification Lambda operate
const ordersEventBus = new cdk.aws_events.EventBus(this, 'ordersEventBus');
const notifyOrderExecutedRule = new cdk.aws_events.Rule(this, 'notifyOrderExecutedRule', {
  eventBus: ordersEventBus,
  eventPattern: {
    supply: ['notifyOrderExecuted'],
    detailType: ['orderExecuted'],
  },
});

// Provision a SES template to ship stunning emails
const orderExecutedTemplate = new cdk.aws_ses.CfnTemplate(this, 'orderExecutedTemplate', {
  template: {
    htmlPart: orderExecutedHtmlTemplate,
    subjectPart: 'Your order was handed to our supplier!',
    templateName: 'orderExecutedTemplate',
  },
});

// This half is frequent to my SES article. No have to observe it if you have already got a SES Id
const DOMAIN_NAME = 'pchol.fr';

const hostedZone = new cdk.aws_route53.HostedZone(this, 'hostedZone', {
  zoneName: DOMAIN_NAME,
});

const id = new cdk.aws_ses.EmailIdentity(this, 'sesIdentity', {
  id: cdk.aws_ses.Id.publicHostedZone(hostedZone),
});
Enter fullscreen mode

Exit fullscreen mode

On this snippet, I create all the required assets, that is primarily based on earlier articles, in case you want a refresher on API Gateway, EventBridge or SES, you possibly can verify them out!

I used a easy HTML template to ship the e-mail, exported from a .ts file, it incorporates the variables {{itemName}}, {{amount}} and {{username}}, that will likely be changed by the values of the order.

export const orderExecutedHtmlTemplate = `<html>
  <head>
    <fashion>
      * {
        font-family: sans-serif;
        text-align: heart;
        padding: 0;
        margin: 0;
      }
      .title {
        coloration: #fff;
        background: #17bb90;
        padding: 1em;
      }
      .container {
        border: 2px stable #17bb90;
        border-radius: 1em;
        margin: 1em auto;
        max-width: 500px;
        overflow: hidden;
      }
      .message {
        padding: 1em;
        line-height: 1.5em;
        coloration: #033c49;
      }
      .footer {
        font-size: .8em;
        coloration: #888;
      }
    </fashion>
  </head>
  <physique>
    <div class="container">
      <div class="title">
        <h1>Howdy {{username}}!</h1>
      </div>
      <div class="message">
        <p>Your order of {{amount}} {{itemName}} was handed to our supplier!</p>
      </div>
    </div>
    <p class="footer">That is an automatic message, please don't attempt to reply</p>
  </physique>
</html>`;
Enter fullscreen mode

Exit fullscreen mode



Lambda capabilities and interactions

To finish the provisioning a part of this text, let’s create the 2 lacking lambda capabilities, and the interfaces between theme and the opposite assets.

// ... earlier code

// Create the request order lambda operate
const requestOrder = new cdk.aws_lambda_nodejs.NodejsFunction(this, 'requestOrder', {
  entry: path.be part of(__dirname, 'requestOrder', 'handler.ts'),
  handler: 'handler',
  surroundings: {
    QUEUE_URL: ordersQueue.queueUrl,
  },
});

// Grant the lambda operate the fitting to ship messages to the SQS queue, add API Gateway as a set off
ordersQueue.grantSendMessages(requestOrder);
restApi.root.addResource('request-order').addMethod('POST', new cdk.aws_apigateway.LambdaIntegration(requestOrder));

const executeOrder = new cdk.aws_lambda_nodejs.NodejsFunction(this, 'executeOrder', {
  entry: path.be part of(__dirname, 'executeOrder', 'handler.ts'),
  handler: 'handler',
  surroundings: {
    EVENT_BUS_NAME: ordersEventBus.eventBusName, // NEW: Add EVENT_BUS_NAME to the surroundings variables of the executeOrder lambda operate
  },
  reservedConcurrentExecutions: 1,
  timeout: cdk.Length.seconds(30),
});

executeOrder.addEventSource(eventSource);
// NEW: grant the lambda operate the fitting to place occasions to the occasion bus
executeOrder.addToRolePolicy(
  new cdk.aws_iam.PolicyStatement({
    actions: ['events:PutEvents'],
    assets: [ordersEventBus.eventBusArn],
  }),
);

// Create the notifyOrderExecuted lambda operate
const notifyOrderExecuted = new cdk.aws_lambda_nodejs.NodejsFunction(this, 'notifyOrderExecuted', {
  entry: path.be part of(__dirname, 'notifyOrderExecuted', 'handler.ts'),
  handler: 'handler',
  surroundings: {
    SENDER_EMAIL: `contact@${id.emailIdentityName}`,
    TEMPLATE_NAME: orderExecutedTemplate.ref,
  },
});

// Grant the lambda operate the fitting to ship emails, add the lambda as a goal of the occasion rule
notifyOrderExecuted.addToRolePolicy(
  new cdk.aws_iam.PolicyStatement({
    actions: ['ses:SendTemplatedEmail'],
    assets: ['*'],
  }),
);
notifyOrderExecutedRule.addTarget(new cdk.aws_events_targets.LambdaFunction(notifyOrderExecuted));
Enter fullscreen mode

Exit fullscreen mode

We’re completed with the provisioning half! Let’s transfer on to essentially the most fascinating half: the code deployed contained in the lambda capabilities.



Lambda capabilities deployed code

Let’s begin with the requestOrder lambda operate. This operate is triggered by a POST HTTP request, and can ship a message to the SQS queue. It is going to additionally return a 200 HTTP standing code to the consumer in case of success.

import { SQSClient, SendMessageCommand } from '@aws-sdk/client-sqs';
import { v4 as uuidv4 } from 'uuid';

const consumer = new SQSClient({});

export const handler = async ({ physique }: { physique: string }): Promise<{ statusCode: quantity; physique: string }> => {
  const queueUrl = course of.env.QUEUE_URL;

  if (queueUrl === undefined) {
    throw new Error('Lacking surroundings variables');
  }

  const { itemName, amount, username, userEmail } = JSON.parse(physique) as {
    itemName?: string;
    amount?: quantity;
    username?: string;
    userEmail?: string;
  };

  if (itemName === undefined || amount === undefined || username === undefined || userEmail === undefined) {
    return Promise.resolve({
      statusCode: 400,
      physique: JSON.stringify({ message: 'Lacking required parameters' }),
    });
  }

  await consumer.ship(
    new SendMessageCommand({
      QueueUrl: queueUrl,
      MessageBody: JSON.stringify({ itemName, amount, username, userEmail }),
      MessageGroupId: 'ORDER_REQUESTED',
      MessageDeduplicationId: uuidv4(),
    }),
  );

  return Promise.resolve({
    statusCode: 200,
    physique: JSON.stringify({ message: 'Order requested' }),
  });
};
Enter fullscreen mode

Exit fullscreen mode

This snippet does the next issues:

  • Traditional: parse the physique of the POST request to get the 4 values we’d like
  • Ship a message to the SQS queue, with a novel ID to keep away from duplicates, and a continuing group ID to make sure the order of the messages inside this group
  • Return a 200 HTTP standing code to the consumer

Subsequent lambda operate: executeOrder. This operate is triggered by the SQS queue, so it is going to have a particular typing as enter. It is going to faux a 20 seconds reference to an exterior API, after which ship an occasion on the occasion bus.

import { EventBridgeClient, PutEventsCommand } from '@aws-sdk/client-eventbridge';

const consumer = new EventBridgeClient({});

export const handler = async (occasion: {
  Data: {
    physique: string;
  }[];
}): Promise<void> => {
  const eventBusName = course of.env.EVENT_BUS_NAME;

  if (eventBusName === undefined) {
    throw new Error('Lacking surroundings variables');
  }

  const { physique } = occasion.Data[0];

  console.log('Communication with exterior API began...');
  await new Promise(resolve => setTimeout(resolve, 20000));
  console.log('Communication with exterior API completed!');

  await consumer.ship(
    new PutEventsCommand({
      Entries: [
        {
          EventBusName: eventBusName,
          Source: 'notifyOrderExecuted',
          DetailType: 'orderExecuted',
          Detail: body,
        },
      ],
    }),
  );
};
Enter fullscreen mode

Exit fullscreen mode

This snippet does the next issues:

  • New: parse the SQS enter. The sort is an array of data. As a result of we set the batch measurement to 1, we will assume that the array will all the time have a size of 1
  • Look ahead to 20 seconds to faux a reference to an exterior API
  • Ship an occasion on the occasion bus, with the physique of the SQS message because the element. Discover I set for this name the identical supply and element sort because the occasion rule goal, in any other case the goal wouldn’t be triggered

Ultimate lambda operate: notifyOrderExecuted. This operate is triggered by the occasion bus, so it is going to have one other typing as enter (refresher right here). It is going to ship an e mail to the person, utilizing a template saved in SES.

import { SESv2Client, SendEmailCommand } from '@aws-sdk/client-sesv2';

const consumer = new SESv2Client({});

export const handler = async (occasion: {
  element: {
    itemName: string;
    amount: quantity;
    username: string;
    userEmail: string;
  };
}): Promise<void> => {
  const senderEmail = course of.env.SENDER_EMAIL;
  const templateName = course of.env.TEMPLATE_NAME;

  if (senderEmail === undefined || templateName === undefined) {
    throw new Error('Lacking surroundings variables');
  }

  const { itemName, amount, username, userEmail } = occasion.element;

  await consumer.ship(
    new SendEmailCommand({
      FromEmailAddress: senderEmail,
      Content material: {
        Template: {
          TemplateName: templateName,
          TemplateData: JSON.stringify({ itemName, amount, username }),
        },
      },
      Vacation spot: {
        ToAddresses: [userEmail],
      },
    }),
  );
};
Enter fullscreen mode

Exit fullscreen mode

This snippet does the next issues:

  • Parse the EventBridge enter. It was routinely parsed from string to object, we simply have to select properties we’d like.
  • Ship a templated e mail utilizing SES. Do not forget that the TemplateData should comprise precisely the identical keys because the template you created in SES, in any other case the ship will silently fail.

We’re completed with the code! Let’s end this text by testing our app!



Testing our utility

For this check, I’ll make 2 consecutive API calls to the /request-order endpoint. If all the pieces is alright, I ought to obtain an e mail after ~20 seconds, and a second e mail after ~40 seconds (as a result of the executeOrder Lambda solely processes one message at a time, and sleeps for 2O seconds).

Listed below are the two requests I made:

request-1

request-2

I ordered 4 bananas and 43 cookies! (I’m very hungry…)

Now let’s verify my emails:

email-1

email-2

I acquired the two emails, with the right portions! Belief me once I say that I acquired the primary e mail after ~20 seconds, and the second after ~40 seconds 😇.



Homework 🤓

We solely constructed a minimalistic utility, and there are quite a lot of issues we will enhance. Listed below are some concepts that you must positively be capable to strive in case you adopted this sequence:

  • Add a database to retailer the orders, and a GET endpoint to retrieve them
  • Solely permit authenticated customers to request orders
  • Work together with an actual API to record the objects and their costs

You possibly can additionally construct a small front-end interacting with this back-end, however I’ll cowl this in a future article 😉.



Conclusion

This tutorial was solely a small sensible instance of what you are able to do with occasions and SQS on AWS. SQS will be tailored to far more use instances, and I encourage you to verify the documentation to be taught extra about it!

I plan to proceed this sequence of articles on a bi-monthly foundation. I already coated the creation of straightforward lambda capabilities and REST APIs, in addition to interacting with DynamoDB databases and S3 buckets. You may observe this progress on my repository! I’ll cowl new matters like front-end deployment, sort security, extra superior patterns, and extra… When you have any solutions, don’t hesitate to contact me!

I’d actually admire in case you may react and share this text with your mates and colleagues. It is going to assist me lots to develop my viewers. Additionally, remember to subscribe to be up to date when the subsequent article comes out!

I you need to keep in contact right here is my twitter account. I typically publish or re-post fascinating stuff about AWS and serverless, be at liberty to observe me!



Leave a Reply

Your email address will not be published. Required fields are marked *