The neighbour that starved checkout

Lab Overview

A shared office printer is fine until the person next to you sends a 400-page document. Nothing about your one-page memo changed. You just cannot get to the printer any more.

The problem. Lambda's concurrency limit is a property of the account and Region, not of a function. Every function without a reservation draws from the same pool. Teams discover this during an incident: a batch job scales up, and an unrelated customer-facing service starts returning errors that look like its own fault.

The objective. Build three services that share nothing but an account, starve the one that matters, and measure the damage. Then change exactly one attribute and measure again.

The outcome. Under identical load, checkout served 5.4% of its requests when it shared the pool and 99.5% when it held a reservation of 50. CloudWatch recorded 3,006 throttles against it in the first run and none in the second. The code, the region and the traffic were the same in both.

Cloud Services and Concepts

Concurrency is a multiplication, not a request rate

AWS states the formula plainly: concurrency is average requests per second multiplied by average request duration in seconds. A function answering in 50 ms at 20 requests a second occupies one slot. The same request rate against a 10-second function occupies 200. This single line decides everything below, including why my first two attempts failed.

The pool is never the number in the documentation

The default quota is 1,000 concurrent executions per Region. What a function can actually reach is 1,000 minus everyone else's reservations. AWS is explicit that reserving for one function reduces the pool available to the others even while the reserving function is idle, and that you may reserve only up to the unreserved total minus 100.

A reservation is a floor and a ceiling

Reserved concurrency "acts as both a lower and upper bound". It guarantees capacity and forbids borrowing beyond it. That makes it a protection and a cap in the same setting — and it costs nothing to configure.

Architecture and Workflow

1. A client posts to one of three routes on the same HTTP API.
2. Each route invokes its own function. The handlers are byte-identical apart from one environment variable, DURATION_MS, so no result can be argued away as a property of the code.
3. Every invocation writes to one shared DynamoDB table, which is what makes them a system rather than three unrelated demos.
4. exports holds a reservation of 900. That capacity leaves the pool every other function draws on, whether or not exports is busy.
5. Which leaves 100 unreserved for checkout and reports together. Each in-flight request holds a slot for as long as its handler runs — 10 seconds for reports, 50 ms for checkout — so the neighbour fills those 100 easily. Once they are gone Lambda throttles, API Gateway returns 503, and checkout fails for a reason that has nothing to do with checkout.

Design decisions

Identical handlers, one variable. The alternative was three realistic services, which would have been a better demo and a worse experiment: any difference in memory, runtime or work done becomes a competing explanation for the result.

An empty Region instead of a tagged sandbox. The test deliberately exhausts a quota that is scoped to the Region. Tags do not contain a quota. An empty Region does, and it also made the cost figure unambiguous: filtering the bill to us-east-2 isolates this lab completely.

Client-side measurement. The success rate is recorded by the thing making the requests, not inferred from CloudWatch. A service is up or down from its caller's position. CloudWatch then explains why.

Prerequisites

Implementation

week-25-serverless-microservices/
|
+-- terraform/
|   +-- environments/dev/     the three services, pinned to us-east-2
|   +-- modules/service/      one module, three instances, one differing variable
|   +-- modules/experiment/   throttle alarms and the concurrency dashboard
|
+-- src/                      the handler, identical for all three services
|
+-- scripts/                  load generation, recording, and the summary

The control variable lives on the function, because reserved concurrency is an attribute rather than a resource of its own:

reserved_concurrent_executions = (
  var.protect_checkout && each.key == "checkout"
  ? var.checkout_reserved_concurrency   # run 2
  : lookup(var.neighbour_reservation, each.key, -1)
)

-1 means "no reservation, share the pool". That single expression is the whole experiment: nothing else differs between the two runs.

Validation and Evidence

Both runs used the same load: checkout at 20 requests a second throughout, with 60 seconds of clean baseline before a neighbour ramped on and held 200 requests open continuously for three minutes.

Security and Cost

Security

HCP Terraform authenticates to AWS with dynamic credentials over OIDC, so no static access key exists for this lab in any file, variable or shell profile. The run role ARN is stored write-only in the workspace. Each function receives its own execution role and writes only to the one table it needs.

The honest caveat: this is a concurrency experiment, not a hardened service. The HTTP API is public and unauthenticated, acceptable only because it lives in an empty Region, writes a UUID to a scratch table, and was destroyed afterwards. A real deployment needs an authorizer and an API-level throttle — which matters more than it looks; see the cost below.

Cost

Measured from Cost Explorer for 30 October, filtered to us-east-2. The Region had no API Gateway usage at all the day before, so nothing else in the account can contaminate these figures.

The shape of that bill is the finding. Lambda cost nothing and API Gateway cost everything, because roughly 773,000 of those requests were throttled before they reached a function. At $1.00 per million requests you pay full price at the front door for work that never ran. Throttling protects the pool; it does not protect the bill. The control that would have is a throttle on the API itself.

Reservations themselves are free — AWS charges nothing for configuring reserved concurrency. The protection in run 2 cost $0.00.

Troubleshooting and Lessons

A convincing collapse that was entirely my own fault

Problem. An early run showed checkout falling to 2% success under load. It looked exactly like the result the lab was built to produce.
Root cause. It was the client dying, not the service. The failures were connection errors rather than HTTP 503s, they began ten seconds before the neighbours ramped on, and CloudWatch recorded zero throttles on all three functions with the pool peaking at 292 of 1,000.
Resolution. Change the success criterion from a success-rate curve to Throttles in CloudWatch, then re-run.
Lesson. A graph will show you a collapse whoever caused it. If the metric you celebrate can be produced by your own test harness failing, it is not evidence. Pick a metric only the system under test can move.

One machine cannot fill a 1,000-slot pool

Problem. Two attempts failed to exhaust the pool, peaking at 319 and 292 concurrent executions with no throttling at all.
Root cause. Concurrency is requests per second times duration, so occupying 1,000 slots means holding 1,000 requests open simultaneously. The original generator started a thread per request and delivered 13 requests a second when asked for 60. Rewriting it in asyncio — coroutines holding keep-alive connections, working in concurrency rather than request rate — raised the ceiling and still fell short.
Resolution. Stop trying to fill the pool and shrink it instead: reserve 900 to a neighbour, and AWS leaves exactly 100 unreserved for the others to contend over.
Lesson. This is closer to production than the original design was. Nobody's unreserved pool is 1,000 — it is 1,000 minus every reservation anyone has made, including idle ones.

A Terraform apply that was legal at the end and illegal in the middle

Problem. Setting checkout's reservation to 50 while lowering a neighbour's from 900 to 850 failed: InvalidParameterValueException: Specified ReservedConcurrentExecutions for function decreases account's UnreservedConcurrentExecution below its minimum value of [100].
Root cause. Terraform applied checkout's increase before the neighbour's decrease. The requested end state totals 900 reserved and is perfectly legal; the intermediate state asked for 950 and breached the account floor.
Resolution. Two applies — lower first, then raise.
Lesson. Reserved concurrency on two functions looks like two independent attributes and is really one shared account-level budget. Terraform orders by dependency, and these resources have none, so it cannot know. Any quota expressed as a sum across resources has this property.

Cleanup and Key Takeaways

Destroyed from the HCP workspace: 35 resources removed in one run. Removal was then verified by asking AWS rather than by reading the destroy output, because a destroy reporting success is a claim and an empty list is evidence — zero functions, zero APIs, zero tables, zero alarms, zero dashboards.

Nothing here billed while idle: the whole $0.81 came from the two four-minute runs.

Key takeaways

• Concurrency is an account-and-Region number, not a function property. A function with no reservation can be denied service by any other function in the account, including one another team deployed.
• Your unreserved pool is the quota minus everyone else's reservations. Idle reservations still take capacity, and AWS refuses to let you reserve the last 100.
• A reservation is a floor and a ceiling in one setting. It guarantees capacity to the function and forbids it from borrowing more. Protecting checkout with 50 also caps checkout at 50.
• Throttling protects the pool, not the bill. 773,000 refused requests still cost full price at API Gateway while Lambda charged nothing. Rate limiting belongs at the front door too.
• Choose a metric your own test harness cannot fake. The most convincing graph this lab produced was a measurement of my laptop running out of sockets.

References

• The full lab guide
• Configuring reserved concurrency
• Lambda quotas
• Understanding Lambda function scaling
• Amazon API Gateway pricing
• Terraform — aws_lambda_function