AWS Lambda Queue
AWS Lambda Queue
A Lambda function may be used for processing messages in a Simple Queue Service queue. The following queue types are supported by Lambda event source mappings: standard queues + first-in, first-out queues. SQS will allow you to start offloading tasks from one component of your app through sending them to a queue and then getting them processed asynchronously.
Lambda will poll the queue and later on invoke the function in a synchronous manner along with an event containing queue messages. Lambda will go ahead and read messages in batches then get your function invoked once for every batch. As your function gets a batch successfully processed, Lambda then tends to delete its messages from the queue. The below example displays an event for a batch having 2 messages.
Standard Queue
| { |
| “Records”: [ |
| { |
| “messageId”: “059f36b4-87a3-44ab-83d2-661975830a7d”, |
| “receiptHandle”: “AQEBwJnKyrHigUMZj6rYigCgxlaS3SLy0a…”, |
| “body”: “Test message.”, |
| “attributes”: { |
| “ApproximateReceiveCount”: “1”, |
| “SentTimestamp”: “1545082649183”, |
| “SenderId”: “AIDAIENQZJOLO23YVJ4VO”, |
| “ApproximateFirstReceiveTimestamp”: “1545082649185” |
| }, |
| “messageAttributes”: {}, |
| “md5OfBody”: “e4e68fb7bd0e697a0ae8f1bb342846b3”, |
| “eventSource”: “aws:sqs”, |
| “eventSourceARN”: “arn:aws:sqs:us-east-2:123456789012:my-queue”, |
| “awsRegion”: “us-east-2” |
| }, |
| { |
| “messageId”: “2e1424d4-f796-459a-8184-9c92662be6da”, |
| “receiptHandle”: “AQEBzWwaftRI0KuVm4tP+/7q1rGgNqicHq…”, |
| “body”: “Test message.”, |
| “attributes”: { |
| “ApproximateReceiveCount”: “1”, |
| “SentTimestamp”: “1545082650636”, |
| “SenderId”: “AIDAIENQZJOLO23YVJ4VO”, |
| “ApproximateFirstReceiveTimestamp”: “1545082650649” |
| }, |
| “messageAttributes”: {}, |
| “md5OfBody”: “e4e68fb7bd0e697a0ae8f1bb342846b3”, |
| “eventSource”: “aws:sqs”, |
| “eventSourceARN”: “arn:aws:sqs:us-east-2:123456789012:my-queue”, |
| “awsRegion”: “us-east-2” |
| } |
| ] |
| } |
FIFO queues have records that include extra attributes having to do with deduplication and sequencing.
FIFO Queue
AWS Lambda Queue – FIFO Queue
| { |
| “Records”: [ |
| { |
| “messageId”: “11d6ee51-4cc7-4302-9e22-7cd8afdaadf5”, |
| “receiptHandle”: “AQEBBX8nesZEXmkhsmZeyIE8iQAMig7qw…”, |
| “body”: “Test message.”, |
| “attributes”: { |
| “ApproximateReceiveCount”: “1”, |
| “SentTimestamp”: “1573251510774”, |
| “SequenceNumber”: “18849496460467696128”, |
| “MessageGroupId”: “1”, |
| “SenderId”: “AIDAIO23YVJENQZJOL4VO”, |
| “MessageDeduplicationId”: “1”, |
| “ApproximateFirstReceiveTimestamp”: “1573251510774” |
| }, |
| “messageAttributes”: {}, |
| “md5OfBody”: “e4e68fb7bd0e697a0ae8f1bb342846b3”, |
| “eventSource”: “aws:sqs”, |
| “eventSourceARN”: “arn:aws:sqs:us-east-2:123456789012:fifo.fifo”, |
| “awsRegion”: “us-east-2” |
| } |
| ] |
| } |
– As Lambda starts reading a batch, the messages will remain in the queue while turning hidden for as long as the queue’s visibility timeout.
– In case the function successfully gets the batch processed, Lambda will delete the messages from the queue.
– In case the function got throttled, returned a specific error, or simply didn’t respond, the message will go back to being visible once again.
– All messages located in a failed batch will go back to the queue, which means that the function code you rely on needs to be capable of processing the exact message many times without incurring any side effects.
Scaling and Processing for Queue types
-
Standard queues:
Utilizes long polling to get a queue polled till it turns active.
For available messages: up to 5 batches will be read and sent to your function.
In case messages remain available, the number of processes reading batches will be increased by up to sixty more instances for every minute.
Maximum number of batches processed simultaneously: 1000.
-
FIFO queues:
Messages are sent to your function in just the same order that they were received.
For sending a message to FIFO queue, you can have the chance to set a message group ID.
Amazon SQS will make sure that messages located in the same group will get delivered to Lambda in the same order.
Lambda sorts those messages into groups then merely sends one batch each at a time for every group.
In case an error gets returned by the function, every retry will be attempted on the messages that are affected prior to Lambda receiving any more messages from the similar group.
A function can scale in concurrency with how many active message groups are found.
How to Configure a Queue to get Used with Lambda?
AWS Lambda Queue – trigger configuration
Start by creating an SQS queue to be in the form of an event source for the chosen Lambda function.
Next go ahead and configure the queue for providing time for the Lambda function to finish processing every one of the batch of events and in order to retry in response to throttling errors as it is scaling up.
For giving your function time to continue processing each batch of records, specify the source queue’s visibility timeout to being at least six times the timeout configured on your function. The extra time you get allows for Lambda to begin with its retry in case your function execution gets throttled when your function processes a previous batch.
In the case that a message fails in getting processed various times, Amazon SQS is capable of sending it to a dead-letter queue.
AWS Lambda Queue – serverless service
When an error gets returned by your function, Lambda will leave it in the queue.
As soon as the visibility timeout happens, Lambda will receive the message once again.
For sending messages to a second queue when trying a number of receives, you need to start configuring a dead-letter queue on your source queue.
Permissions for the Execution Role
Lambda must have the below permissions to be able to start managing messages in your SQS queue. Get those permissions added to the execution role of your function.
- sqs:ReceiveMessage
- sqs:DeleteMessage
- sqs:GetQueueAttributes
How to Configure a Queue as an Event Source?
AWS Lambda Queue – Queue as an Event Source
An event source mapping gets created for the sake of telling Lambda to transfer items from the queue to a specific Lambda function. Multiple event source mappings may be created for processing items from various queues with just a single function. As soon as Lambda gets the target function invoked, the event will be able to include multiple items (up to a configurable max batch size).
For configuring your function to start reading from Amazon SQS in Lambda console, you will have to create an SQS trigger.
For creating a trigger
- Go to the Lambda console Functions page.
AWS Lambda Queue – functions page
- Select a function.
- From below Designer, select Add trigger.
- Pick a trigger type.
AWS Lambda EventSourceMapping – trigger type
- Get the required options configured then click on Add.
The below options are supported by Lambda for SQS event sources.
Options for Event Source
– SQS queue: The SQS queue where records can be read from.
– Batch size: Number of items for reading from the queue in every batch (up to 10). The event may even contain less items in case the batch which Lambda had read from the queue contained less items.
– Enabled: You may disable the event source in hopes of stopping the processing of items.
For managing the event source configuration another time, select the trigger found in the designer.
Get your function timeout configured for allowing enough time to get a whole batch of items processed. When items need a lot of time to get processed, you need to select a smaller batch size.
Large batch sizes are capable of improving efficiency for extremely fast workloads or those that include much overhead. Yet, in case your function gets an error returned, every single item found in the batch will get returned to the queue.
In case of configuring reserved concurrency on your function, you must set a min of five concurrent executions for reducing the probability of throttling errors as Lambda gets your function invoked.
APIs for Event Source Mapping
To be able to start managing event source mappings using the CLI or SDK, you need to utilize the below API actions:
- CreateEventSourceMapping
- ListEventSourceMappings
- GetEventSourceMapping
- UpdateEventSourceMapping
- DeleteEventSourceMapping
The below mentioned example relies on the CLI for mapping a function called my-function to an SQS queue which specified by its ARN, having a batch size of 5.
| $ aws lambda create-event-source-mapping –function-name my-function –batch-size 5 \ |
| –event-source-arn arn:aws:sqs:us-east-2:123456789012:my-queue |
| { |
| “UUID”: “2b733gdc-8ac3-cdf5-af3a-1827b3b11284”, |
| “BatchSize”: 5, |
| “EventSourceArn”: “arn:aws:sqs:us-east-2:123456789012:my-queue”, |
| “FunctionArn”: “arn:aws:lambda:us-east-2:123456789012:function:my-function”, |
| “LastModified”: 1541139209.351, |
| “State”: “Creating”, |
| “StateTransitionReason”: “USER_INITIATED” |
| } |