Armand RegoLambda power tuning - what is it good for?
As AWS Lambda enthusiasts know, the primary lever for controlling function runtime performance is memory allocation, which ranges from 128MB to 10,240MB. Additionally, Lambda dynamically allocates virtual CPUs (vCPUs) in proportion to the configured memory, reaching a maximum of 1 vCPU at around 1,800MB.
However, finding the optimal memory configuration that balances performance (invocation time) and cost requirements can be a challenge. This is where AWS Lambda Power Tuning comes into play. It's a state machine that runs your function with a range of memory/power settings, collects runtime metrics, and presents them in a graphical format to empower data-driven decision making regarding the function configuration.
The issue
We recently developed a microservice for a client, deploying it as a single AWS Lambda function using Python CDK and fronting it with a Function URL. After verifying that the function was working as expected, we wanted to optimize its performance to strike an optimal balance between cost and latency.
Since our function was triggered via a Function URL, we initiated a request to the endpoint to capture the structure of the received event. We then configured this payload in Power Tuner, resulting in an input configuration similar to the example below:
1 | |
2 | |
3 | |
4 | |
5 | |
6 | |
7 | |
8 | |
9 | |
10 | |
11 | |
12 | |
13 | |
14 | |
15 | |
16 | |
17 | |
18 | |
19 | |
20 | |
21 | |
22 | |
23 | |
Computer says no
With everything configured correctly, we ran the Power Tuner and were immediately confronted with an invocation error!
A closer look at the logs showed that root cause was failure during the initialisation phase of the Lambda function due to the Lambda runtime not having sufficient permissions to read the files in the deployment package:
1 | |
2 | |
3 | |
4 | |
5 | |
6 | |
7 | |
8 | |
9 | |
10 | |
11 | |
12 | |
13 | |
14 | |
15 | |
16 | |
17 | |
A quick search for this error will yield many StackOverflow hits (see here, here and here) and it is explicitly mentioned in the AWS docs, but what made it strange in our case was the fact that the function ran without error when called via the function URL or invoked manually through the AWS Console!
The fix
After some RTFMing, we managed to get the Power Tuner to run by explicitly configuring the correct file permissions during the bundling process, as shown by the CDK code below:
1 | |
2 | |
3 | |
4 | |
5 | |
6 | |
7 | |
8 | |
9 | |
10 | |
11 | |
12 | |
13 | |
14 | |
15 | |
16 | |
17 | |
18 | |
19 | |
20 | |
21 | |
22 | |
23 | |
24 | |
25 | |
26 | |
Note the important line in the command for the bundling Docker image "chmod -R 755 /asset-output" which gives the runtime executable permissions to the files in the deployment package.
Conclusion
Although we successfully got Power Tuner running, we still don't have a complete understanding of why it didn't work initially, particularly given that the function executed correctly when triggered via the Function URL or through the AWS Console.
The current hypothesis is that the process of creating separate Lambda function versions during the execution of the Power Tuner step function somehow reconfigured the file permissions, making them non-executable. If anyone has encountered this issue before and can shed light on the underlying reason for the behaviour, please don't hesitate to reach out!