I'm using AWS Control Tower and AWS Organizations to implement a multi-account setup for a client of mine. I needed to analyze the VPC Flow Logs, so I used the console to setup the VPC Flow Logs Athena integration. It uses a CloudFormation template that includes two Lambdas. I ran into the following error while deploying the stack:
Resource handler returned message: "Specified ReservedConcurrentExecutions for function decreases account's UnreservedConcurrentExecution below its minimum value of [100].
I investigated and the two Lambdas each had a ReservedConcurrentExecutions value of 1. I was stumped at first because the default Lambda concurrent executions quota for an account and region is 1000. Well, I was wrong—if you're using AWS Control Tower Account Factory to create the accounts. In my case, the applied limit was set at 10. 10!?!?
Since Lambda requires a minimum of 100 unreserved concurrency capacity for functions that don't use an explicitly reserved concurrency, my account was way underprovisioned. The 10 I had available had to be used for unreserved capacity only. There was no room for any reserved capacity, as evidenced by the CloudFormation error I received. I needed at least 102 in order to deploy this solution.
I checked another client account of mine that uses AWS Control Tower, and it allowed me a whopping 50 concurrency. I found documentation on the subject to explain why these accounts were not assigned 1000 quota to begin with. It's called out at the top of the page:
New AWS accounts have reduced concurrency and memory quotas. AWS raises these quotas automatically based on your usage.
That makes sense. New users can shoot themselves in the foot with a recursive Lambda bug that could cost them a LOT of $$$ if it runs unnoticed at a concurrency of 1000. Or maybe there are concerns of crypto miners hijacking AWS Orgs. But 10 for starters is a bit low. It's not clear how long it takes to see this increase from 10 to the default quota value of 1000.
The Solution
The immediate fix is straightforward. Go to the Quotas page in the AWS Console and request an increase. For each region. In each account. Kind of a pain. There is a feature that allows you to setup a template for your organization to automatically request quota increases any time a new account is provisioned via AWS Control Tower/AWS Organizations. Pretty handy. You can access it here.
NOTE: You must set your region to us-east-1 (N. Virginia) in the AWS Console.
It oddly assumes that you already have 1000 concurrency as your limit, so you have to ask for at least 1001. In my case, I would have been happy going from 10 to 200 for now.
Another odd layer of this experience is that my first quote increase request wasn't met with a quick, "of course we can" kind of response. It apparently had to be escalated, and it took well over a day to resolve. This was the response I received initially:
I'm following up to notify you that we've received your AWS Lambda Concurrent executions quota increase request for 200 in the Ohio region, and I’m happy to submit the request for you. For a quota increase of this size, I will need to collaborate with our Service Team to get approval. Please note that it can take some time for the Service Team to review your request. This is to ensure that we can meet your needs while keeping existing infrastructure safe.
I made a separate quota increase for a different region in the same account, requesting 500 concurrency this time, and it was quickly approved. I'm not sure what the difference was.
Resources
