Armand RegoThe setup
The backend follows an event-driven, service-based architecture, with each service containing one or more Lambda functions. Each of those functions pulls its container image from an ECR repository at cold start. The stacks are synthesised with CDK's AppStagingSynthesizer, which creates its own staging stack to hold ECR repositories and other deployment assets.
We wanted scan-on-push enabled on every ECR repository, including the ones the synthesiser creates. The synthesiser doesn't expose a scanning option, and because it owns those repository definitions we can't set the property on them directly. So we wrote a CDK Aspect that operates on the L1 CfnRepository resource; that way the setting lands in CloudFormation regardless of how the repository was constructed.
1 | |
2 | |
3 | |
4 | |
5 | |
6 | |
7 | |
8 | |
9 | |
10 | |
11 | |
Add the Aspect at the app level and every repository gets ScanOnPush: true:
1 | |
About as innocuous as infrastructure changes get. One property on a resource with no other pending changes. It deployed without error, and with no evident regressions. Then, several hours later, CloudWatch alarms started hammering the Slack channel.
Two services controlling one property
When you create a Docker-image Lambda function, Lambda sets a resource-based policy on the backing ECR repository called LambdaECRImageRetrievalPolicy. It does this by calling ecr:SetRepositoryPolicy directly, outside CloudFormation. It's documented behaviour, but not something you might think to dig into under normal circumstances.
1 | |
2 | |
3 | |
4 | |
5 | |
6 | |
7 | |
8 | |
9 | |
10 | |
11 | |
This same policy can be set via the declarative CloudFormation property RepositoryPolicyText. If your template updates an AWS::ECR::Repository and doesn't declare a policy, CloudFormation takes that to mean the repository should have no policy. It deletes whatever is there, including the one set directly by the Lambda service's ecr:SetRepositoryPolicy API call.
The end result of all this is that any CloudFormation update to an ECR repository backing a Dockerised Lambda will delete the policy if the template doesn't declare it. The trigger is simply "a CloudFormation update to the repo."
In our case the Aspect was that trigger. It added ImageScanningConfiguration to repositories that had never been updated since creation, and that first update made CloudFormation reconcile the whole resource, dropping the undeclared RepositoryPolicyText on all of them at once. CloudTrail shows exactly what happened:
1 | |
2 | |
3 | |
4 | |
5 | |
6 | |
7 | |
8 | |
9 | |
10 | |
11 | |
12 | |
13 | |
14 | |
15 | |
16 | |
17 | |
18 | |
19 | |
20 | |
21 | |
22 | |
23 | |
24 | |
25 | |
26 | |
Every repository in the staging stack got the same DeleteRepositoryPolicy call at the same timestamp, all invoked by cloudformation.amazonaws.com.
Without the requisite permissions, our Lambda functions were no longer able to pull their images from ECR. The delay between deploy and the alarms going off was simply due to the fact that there were enough warm execution environments able to handle the incoming requests. It was only once these warm environments were eventually killed, and Lambda tried to spin up new ones, that the ImageAccessDenied errors started flowing in.
Getting back up
We were able to recover quickly, in a two-step manner.
First, we restored the policy on each affected repository with manual ecr:SetRepositoryPolicy API calls, allowing the Lambda functions to pull the Docker images once again.
But restoring the policy alone wasn't enough. Lambda caches the ImageAccessDenied result; a function that's already failed won't retry the ECR pull just because the policy is back. You have to force each function to re-pull. Any configuration change does this: an UpdateFunctionConfiguration call makes Lambda fetch the image again. We pushed a trivial environment-variable change across all functions, Lambda re-pulled from the now-readable repositories, and the functions came back to `Active`.
The fix
After bringing the system back up, we deployed the actual fix to prevent the issue reoccurring i.e stopping CloudFormation from treating the absence of a declared policy as an instruction to delete one, by explicitly defining the policy on the Aspect itself:
1 | |
2 | |
3 | |
4 | |
5 | |
6 | |
7 | |
8 | |
9 | |
10 | |
11 | |
12 | |
13 | |
14 | |
15 | |
16 | |
17 | |
18 | |
19 | |
20 | |
21 | |
22 | |
23 | |
24 | |
25 | |
26 | |
27 | |
28 | |
29 | |
30 | |
31 | |
32 | |
33 | |
34 | |
35 | |
36 | |
37 | |
38 | |
The Aspect now owns both properties: the scan-on-push setting we wanted and the policy Lambda would otherwise set directly. There's no undeclared property left for CloudFormation to delete.
We used an Aspect because the AppStagingSynthesizer owns our repository definitions. If you declare your repositories directly, just add the policy where you define them. During our investigation we found that CDK team learned this with the bootstrap stack years ago and fixed it by putting the policy in the template. AppStagingSynthesizer, still alpha, never got the same treatment:
- cloudformation-coverage-roadmap#2493 - the underlying bug: CloudFormation deletes undeclared
RepositoryPolicyTexton update, and drift detection ignores it. Open since April 2026. - aws-cdk#18473 - the same deletion hit CDK's bootstrap stack. Fixed in bootstrap v11 (March 2022) by declaring the policy in the template.
- aws-cdk#27007 -
AppStagingSynthesizercreates ECR repositories without that declared policy. Open since September 2023.
Takeaways
CloudFormation will delete a property it doesn't manage and never tell you. Lambda's ECR policy is set via direct service API call; it never appears in a template, isn't tracked in stack state, and isn't caught by drift detection. Any update to the repository can silently remove it.
Anything that touches CfnRepository: an Aspect, a direct property change etc. needs to account for LambdaECRImageRetrievalPolicy. If you're using AppStagingSynthesizer with Dockerised Lambdas, you're exposed until the CDK team fixes #27007.