Armand RegoThe setup
Our API runs on Docker-image Lambda functions, each pulling its container image from ECR at cold start. The stacks are synthesised with CDK's `AppStagingSynthesizer`, which creates its own staging stack to hold ECR repositories and other deployment assets. We don't manage those repositories directly.
We wanted scan-on-push enabled on every ECR repository, including the ones the synthesiser creates. The synthesiser doesn't expose a scanning option, and because it owns those repository definitions we can't set the property on them directly. So we wrote a CDK Aspect that operates on the L1 `CfnRepository` resource — that way the setting lands in CloudFormation regardless of how the repository was constructed.
1 | |
2 | |
3 | |
4 | |
5 | |
6 | |
7 | |
8 | |
9 | |
10 | |
11 | |
Attach it at the app level and every repository gets `ScanOnPush: true`:
1 | |
About as innocuous as infrastructure changes get. One property on a resource with no other pending changes. It deployed cleanly. Nothing broke. The API kept serving traffic. Then, hours later, it stopped.
Two services, one property
The root cause: two AWS services both believe they own `RepositoryPolicyText` on an ECR repository.
When you create a Docker-image Lambda function, Lambda sets a resource-based policy on the backing ECR repository called `LambdaECRImageRetrievalPolicy`. It does this by calling `ecr:SetRepositoryPolicy` directly — outside CloudFormation. It's documented behaviour, but nothing in your template ever mentions it.
1 | |
2 | |
3 | |
4 | |
5 | |
6 | |
7 | |
8 | |
9 | |
10 | |
11 | |
CloudFormation treats `RepositoryPolicyText` as a declarative property. If your template updates an `AWS::ECR::Repository` and doesn't declare a policy, CloudFormation takes that to mean the repository should have no policy. It deletes whatever is there — including the one Lambda set out of band.
This has nothing to do with how the repository ended up in your template. Any CloudFormation update to an ECR repository backing a Docker-image Lambda will delete the policy if the template doesn't declare it. The trigger is simply "a CloudFormation update to the repo."
In our case the Aspect was that trigger. It added `ImageScanningConfiguration` to repositories that had never been updated since creation, and that first update made CloudFormation reconcile the whole resource — dropping the undeclared `RepositoryPolicyText` on all of them at once. CloudTrail shows exactly what happened:
1 | |
2 | |
3 | |
4 | |
5 | |
6 | |
7 | |
8 | |
9 | |
10 | |
11 | |
12 | |
13 | |
14 | |
15 | |
16 | |
17 | |
18 | |
19 | |
20 | |
21 | |
22 | |
23 | |
24 | |
25 | |
26 | |
Every repository in the staging stack got the same `DeleteRepositoryPolicy` call at the same timestamp, all invoked by `cloudformation.amazonaws.com`.
The deletion was silent. No error, no alarm, nothing in CloudTrail marked as unusual. Drift detection doesn't even inspect `RepositoryPolicyText` when the template doesn't declare it. A property that one AWS service manages out of band gets deleted by another, and the tooling that exists to catch this kind of divergence looks straight past it.
Getting back up
Recovery is two steps, and the order matters.
First, restore the policy on each affected repository with `ecr:SetRepositoryPolicy`. This puts the repositories back to the state Lambda expects.
But restoring the policy alone isn't enough. Lambda caches the `ImageAccessDenied` result — a function that's already failed won't retry the ECR pull just because the policy is back. You have to force each function to re-pull. Any configuration change does this: an `UpdateFunctionConfiguration` call makes Lambda fetch the image again. We pushed a trivial environment-variable change across all functions, Lambda re-pulled from the now-readable repositories, and the functions came back to `Active`.
That's the clean path: restore the policy, then nudge each function into re-pulling. Two API calls per function, no need to delete or recreate anything.
The fix
Recovery gets you back online, but the bug will bite again on the next repository update unless you change something. The fix is to stop CloudFormation from treating the absence of a declared policy as an instruction to delete one. Declare the policy yourself, in the same Aspect that caused the problem.
1 | |
2 | |
3 | |
4 | |
5 | |
6 | |
7 | |
8 | |
9 | |
10 | |
11 | |
12 | |
13 | |
14 | |
15 | |
16 | |
17 | |
18 | |
19 | |
20 | |
21 | |
22 | |
23 | |
24 | |
25 | |
26 | |
27 | |
28 | |
29 | |
30 | |
31 | |
32 | |
33 | |
34 | |
35 | |
36 | |
37 | |
38 | |
The Aspect now owns both properties: the scan-on-push setting we wanted and the policy Lambda would otherwise set out of band. There's no undeclared property left for CloudFormation to delete.
We used an Aspect because the synthesiser owns our repository definitions. If you declare your repositories directly, just add the policy where you define them. Either way, deploy the fix before making any other change that could trigger a repository update — the declared policy needs to be in the template before CloudFormation has a reason to reconcile the resource.
This is a known issue with some history worth knowing:
- [**cloudformation-coverage-roadmap#2493**](https://github.com/aws-cloudformation/cloudformation-coverage-roadmap/issues/2493) — the underlying bug: CloudFormation deletes undeclared `RepositoryPolicyText` on update, and drift detection ignores it. Open since April 2026.
- [**aws-cdk#18473**](https://github.com/aws/aws-cdk/issues/18473) — the same deletion hit CDK's bootstrap stack. Fixed in bootstrap v11 (March 2022) by declaring the policy in the template.
- [**aws-cdk#27007**](https://github.com/aws/aws-cdk/issues/27007) — `AppStagingSynthesizer` creates ECR repositories without that declared policy. Open since September 2023.
The CDK team learned this with the bootstrap stack years ago and fixed it by putting the policy in the template. `AppStagingSynthesizer`, still alpha, never got the same treatment. The workaround above is really just applying the bootstrap-stack fix by hand.
Takeaways
CloudFormation will delete a property it doesn't manage and never tell you. Lambda's ECR policy is set out of band — it never appears in a template, isn't tracked in stack state, and isn't caught by drift detection. Any update to the repository can silently remove it.
Anything that touches `CfnRepository` — an Aspect, a direct property change, anything — needs to account for `LambdaECRImageRetrievalPolicy`. If you're using `AppStagingSynthesizer` with Docker-image Lambdas, you're exposed until the CDK team fixes [#27007](https://github.com/aws/aws-cdk/issues/27007).