AWS Lambda Throttle – Limits of Concurrency
Every single region found in your account includes what is called a Lambda concurrency limit. This limit decides on the number of function invocations are capable of being run simultaneously in just a single region. It is applied to every single function located in the exact similar region and set to a 1000 [by default].
In case a concurrency limit gets exceeded, Lambda will begin throttling those offending functions through the act of rejecting requests. According to which type of invocation is made, the below situations will occur:
-
Synchronous
Sources of Invocation: On Demand, API Gateway, Cloudfront
429 status code will be returned. This request does not get retried.
-
Asynchronous
Sources of Invocation: SNS, S3, SES
The invocation shall be retried twice. In case both attempts fail, then the request will not get retried. I case of configuring a dead letter queue, this request will be sent there.
AWS Lambda Throttle – debugging and error handling
A dead letter queue may be configured for a function in your console.
- Polling [stream-based]
Sources of Invocation: Kinesis, DynamoDB
The invocation will be retried by AWS till reaching the retention period expiration.
- Polling [stream-based]
Sources of Invocation: SQS
The connection will be reattempted by Lambda till the queue’s conditions for message deletion get met.
No matter what the invocation source is, constantly occurring throttling made by Lambda will result in the failure of running your requests, while being displayed in production as an actual bug.
Lambda Limits Configuration:
The way of changing your default concurrency limit is dependent on the way Lambda functions get configured. Let’s check those configuration options: allocating the limit by function or jumping to troubleshooting unreserved concurrency limit.
What is a Function-Level Concurrency Limit Configuration?
The name Function-level concurrency limit tells all you need to know about it: a reserved amount of concurrent executions that get spared for a chosen function. [configuration option made available by Lambda through the console]
AWS Lambda Throttle – concurrency
Example of the way function-level allocations take place: In case you have an account limit of 1000 while you are reserving a number of 200 concurrent executions for one function as well as 100 concurrent executions for another function, the rest of the functions found in this region are going to share the 700 executions that remain.
Important to know: Lambda will limit the final amount of concurrency for you to reserve over every function found in a single region. Lambda needs a minimum of 100 unreserved concurrent executions for each account.
How to troubleshoot a Function with Function-Level Allocation?
In case of reserving concurrent executions for one chosen function, Lambda will assume that you are aware of the number for reserving to stay away from performance issues. Functions that have allocated concurrency are not capable of accessing unreserved concurrency. When your function gets throttled, you have the possibility to raise the concurrency which is allocated to the function.
In case of wanting to increase the concurrency limit for one of your functions, specify whether the functions that share unreserved concurrency allocation are closing near to this limit, because as reserved concurrency increases, it’s going to lower the available unreserved concurrency. Through going to CloudWatch’s UnreservedConcurrentExecutions metric in the Lambda namespace, you are capable of checking this.
In the case that there is no spare of unreserved concurrency, start comparing the reserved amount for every one of the other functions having values for the Invocations metric in CloudWatch. When this function’s invocations are not capable of approaching its reserved concurrency, this means it’s overallocated, and it’s possible to start lowering your reservation. Hence, you will get extra room for increasing the allocation of the function being troubleshooted.
How to Use Unreserved Concurrency for the sake of Troubleshooting your function?
In case your function starts to draw from the pool of unreserved concurrent executions, you are going to have to find out the cause which is using up your unreserved concurrency allocation.
In order for this to be done, go to the UnreservedConcurrentExecutions metric in CloudWatch to find out if you keep always (or regularly while at bursty workloads) bumping up at unreserved concurrency limit. In case you get near that limit, it’s possible for you to increase the unreserved concurrency through the act of reducing function-level limits, or simple requesting a greater account concurrency limit.
AWS Lambda Throttle – unreserved concurrency pool
AWS Lambda Throttle – invocation burst
There is constant exhaustion of the unreserved concurrency pool.
Occasional bursts in invocations = one of functions with a bursty workload.
Else, start searching for a bursty function which gets periodically spiked while it uses up the unreserved concurrency pool. Look at the Invocations metric while searching every function in Cloudwatch I order to discover your bursty function.
AWS Lambda Throttle – invocations
Function graphed in blue: bursting every now and then while using up the unreserved concurrency pool.
As soon as you get a bursty function identified, it’s possible to start through checking the upstream event source; in case your search ends up being unfruitful, you will the need to perform a bit of function-level allocation housekeeping. Here are your options:
- Find out which functions cannot be throttled whatsoever, and later on start reserving concurrency for them.
- Get a part of your concurrency limit reserved for the bursty function you chose so it will be guaranteed that the other functions may still be able to utilize the remaining unreserved capacity without getting throttled. Which will cause the bursty function to get throttled at the time when it should have otherwise spiked.
Increasing Lambda Concurrency Limits:
Start a support ticket with AWS so that you request to get an increase in the account’s level of concurrency limit.
- Start creating a new support case
- Specify Regarding value to be: Service Limit Increase.
- Select Lambda as being the Limit Type.
- Type the required info in the body of your form.
- Await the response of AWS to the request you made.
They are capable of increasing your limit so that you get the chance to run more Lambda functions in a concurrent manner.