Give Your Coding Agent a Real Cloud to Break

Introduction

Coding agents write infrastructure code all day now. What they cannot easily do is find out whether it works.

The options are all bad. Point the agent at a real AWS account and every wrong guess is a real resource, a real bill, and a blast radius somebody has to clean up. Point it at mocked SDK calls and it will pass, because a mock returns whatever the agent expects it to return. The failures that matter in event-driven systems live between services: a poller redelivering a batch, a redrive policy firing, a type validation rejecting a write. None of that exists in a stubbed client.

So we gave an agent a cloud it could break. Claude Code (with Opus 4.8), the LocalStack MCP server, IAM enforcement switched on, and a production-style bug report about an inventory pipeline whose dead-letter queue was quarantining perfectly good records. No instructions on how to fix it, no hints about the cause, and no human in the loop until the diff was ready.

It took about 17 minutes and $4.60 in tokens. Here is what it did, and which parts of the run only worked because the cloud underneath was fully emulated instead of mocked.

How the MCP server works

The Model Context Protocol is a standard way to hand an agent a set of tools. The LocalStack MCP server exposes the emulator through it, so instead of you writing glue scripts, the agent gets tools for starting the container, deploying CDK or Terraform projects, running AWS CLI commands, reading logs, and inspecting traces.

Two of those tools carry most of the weight in this experiment:

1. IAM policy enforcement: This is off by default in LocalStack, which means every API call succeeds regardless of the policies attached to it. This is great for local development, as it allows you to develop and iterate without IAM issues getting in the way until you’re ready to deal with them, but not ideal for this use case. With ENFORCE_IAM=1, LocalStack evaluates identity policies, resource policies, and SCPs on every request, the way AWS does. An agent working in that environment cannot write a Lambda that quietly relies on permissions it was never granted.
2. App Inspector: This is the more interesting one. It records what happens inside the emulator as a tree: a trace is one flow through your system, a span is a single AWS API call inside that flow, and events are things that happened during a span. Those events include IAM policy evaluations, so a span tells you both what the call did and whether the policy engine allowed it. On AWS you would need X-Ray plus CloudTrail plus CloudWatch Logs to assemble a fraction of that, and you would still be correlating timestamps by hand.

For an agent, the combination of these two tools means the agent doesn’t have to guess what’s wrong, it knows.

Prerequisites

• Docker and a valid LOCALSTACK_AUTH_TOKEN
• Node.js for the MCP server
• Claude Code, or any other MCP client
• cdklocal for the CDK deploys

Step 1: Set up the MCP server

The server ships an interactive wizard that writes the client configuration for you:

npx -y @localstack/localstack-mcp-server init

It checks Docker, picks up LOCALSTACK_AUTH_TOKEN from your environment, detects which MCP clients you have installed (Cursor, Claude Code, Claude Desktop, VS Code, Codex, and others), and writes the config for the ones you pick. Re-running is safe, since it reports an existing entry rather than overwriting it:

◆  Docker daemon ✓
●  Using LOCALSTACK_AUTH_TOKEN from this shell environment.
◇  Checked selected clients: Claude Code

Nothing else to wire up. The next time Claude Code starts, the LocalStack tools are available to it.

Step 2: The pipeline and what is wrong with it

The application is sample-sqs-lambda-dynamodb, an inventory ingestion pipeline built with CDK. A CSV lands in S3, a Lambda parses it and fans the rows into an SQS queue, a second Lambda consumes the queue in batches of ten and writes each row to DynamoDB. The queue has a dead-letter queue attached with maxReceiveCount set to 5.

On paper the failure handling is already there. In practice it is broken in a way that is very common, and worth understanding before watching an agent walk into it.

When Lambda polls SQS, it hands the function a batch of messages. If the handler returns normally, all of those messages are deleted from the queue. If the handler raises an error, none of them are deleted, and the entire batch becomes visible again for redelivery. That is the default contract, and it means one bad message poisons its neighbours.

The consumer in this repo loops over the batch and writes each record straight to DynamoDB:

for message in messages:
    message_body = json.loads(message['body'])
    record_id = str(uuid.uuid4())
    item = {
        'id': {'S': record_id},
        'product_id': {'S': message_body['product_id']},
        'quantity': {'N': str(message_body['quantity'])},
        # ...
    }
    dynamodb_client.put_item(TableName=table_name, Item=item)

Two things go wrong when a feed contains one malformed row. DynamoDB rejects {'N': 'unknown'} with a ValidationException, the exception propagates, and the whole batch comes back. And because record_id is a fresh random UUID on every attempt, the nine healthy rows that did succeed get written again under new keys each time the batch is retried. After five attempts, all ten messages are dead-lettered together and the table holds five copies of everything.

Step 3: Hand the agent the bug report

The prompt matters here. The task was written the way a real ticket arrives: symptoms, acceptance criteria, and no diagnosis.

You are the on-call engineer for the inventory ingestion pipeline in this
repository. Operations filed this bug report from production:

> When a feed file contains even one malformed row, we end up with duplicated
> inventory records for the healthy rows of that file, and healthy updates keep
> appearing in the dead-letter queue next to the bad row. Example: yesterday's
> feed had one row with quantity "unknown"; nine correct rows were written
> several times each, and all ten messages ended up in the DLQ together.

Your job, start to finish:

1. Start LocalStack with IAM enforcement enabled (ENFORCE_IAM=1) and enable App
   Inspector for the instance.
2. Deploy this CDK app to LocalStack as-is.
3. Reproduce the reported behavior with a mixed CSV feed. Show concrete evidence
   of both symptoms.
4. Fix the pipeline so that a malformed row is quarantined in the DLQ after a
   bounded number of attempts, healthy rows from the same file are written
   exactly once, and no healthy message ends up in the DLQ. Do not hide
   failures: malformed rows must still reach the DLQ, not be silently dropped.
5. Prove the fix end-to-end with a fresh mixed feed.

It’s worth noting that the prompt did steer the agent on method: use App Inspector traces and spans as the primary evidence when investigating, and keep production values in whatever ships. Then run claude in the repo with Opus 4.8, paste, and let it run.

Step 4: The agent sets up its own environment

The first thing it did was refuse to trust its own configuration.

● localstack-management(action: "status")
● localstack-management(action: "restart", envVars: {"ENFORCE_IAM": "1"})
● localstack-iam-policy-analyzer(action: "get-status")
  ⎿  IAM enforcement mode: ENFORCED

It started the container with enforcement on, then called a second, separate tool to confirm the policy engine was actually in enforced mode before continuing. Then App Inspector, then the deploy:

● localstack-app-inspector(action: "set-status", status: "enabled")
● localstack-deployer(action: "deploy", projectType: "cdk")
  ⎿  Bootstrap + deploy completed

The deployer wraps cdklocal bootstrap and deploy, so the agent gets a working stack from a single call rather than reconstructing the CDK invocation itself.

Before reproducing anything it lowered the queue’s visibility timeout to 4 seconds, since the production value of 300 seconds would have meant waiting 25 minutes to watch five redelivery cycles. Then it did something we did not ask for and that turned out to matter:

● localstack-app-inspector(action: "delete-traces")

It cleared the trace buffer first, so every trace collected afterwards belonged to the reproduction and nothing else. Deploy noise didn’t end up in its evidence.

Step 5: Watch it reproduce the failure

It wrote a feed with nine good rows and one row with quantity: unknown, uploaded it, and then went looking at traces rather than logs.

● localstack-app-inspector(action: "list-traces")

The shape of that table illustrates the bug as demonstrated by how the agent read it:

Five identical error traces is maxReceiveCount counting down in front of you. It drilled into one of them, found nine successful PutItem spans, one failed PutItem span, and a failed Lambda Invoke span wrapping all of them, then pulled the failing span’s attributes:

"localstack.aws.service.request": "{\"Item\": {\"product_id\": {\"S\": \"P109\"},
  \"quantity\": {\"N\": \"unknown\"}, ...}}",
"localstack.aws.service.exception": "{\"code\": \"ValidationException\",
  \"message\": \"A value provided cannot be converted into a number\"}"

Then the step that is hard to get anywhere else. It listed the events attached to that failing span:

● localstack-app-inspector(action: "list-events", span_id: "…")
  ⎿  | iam.policy.allowed | iam.policy_evaluation | 2026-07-27T16:27:20.418Z |

A 400 error on a write under IAM enforcement has two plausible causes, and telling them apart is normally where you lose an afternoon. The span carried an iam.policy.allowed event, so the agent eliminated the permissions branch in one call and went straight at the data. The measured symptoms matched the report exactly: 45 items in DynamoDB for 9 healthy rows, and all 10 messages in the DLQ.

Step 6: The fix and the proof

The fix has two halves, and both are about the batch contract rather than the bad row itself.

Lambda supports a partial batch response. If you enable ReportBatchItemFailures on the event source mapping, the function can return a list of the message IDs that failed, and SQS deletes everything else in the batch. The agent enabled it on the mapping and rewrote the handler to catch per record:

batch_item_failures = []

for message in messages:
    message_id = message['messageId']
    try:
        record = json.loads(message['body'])
        product_id = str(record['product_id'])
        location = str(record['location'])
        update_date = str(record['update_date'])

        quantity = int(record['quantity'])
        record_id = f"{product_id}#{location}#{update_date}"

        dynamodb_client.put_item(TableName=table_name, Item={
            'id': {'S': record_id},
            'quantity': {'N': str(quantity)},
            # ...
        })
    except Exception as error:
        print(f"Failed to process message {message_id}: {error}")
        batch_item_failures.append({'itemIdentifier': message_id})

return {'batchItemFailures': batch_item_failures}

The second half is the key. Swapping the random UUID for product_id#location#update_date makes the write idempotent: a redelivered row overwrites its own item instead of creating a new one. Partial batch responses stop healthy rows from being retried, and the deterministic key means that even if they are, nothing duplicates. Validating quantity before the write is a small addition that shows up in the traces later.

The redeploy produced one surprise. After the in-place CloudFormation update, the event source mapping came back disabled, which the agent caught because it checked rather than assumed:

● localstack-aws-client(command: "lambda list-event-source-mappings …")
● localstack-aws-client(command: "lambda update-event-source-mapping --uuid … --enabled")

Then it cleared traces again, sent a fresh feed with the same one bad row, and collected the after picture. Every trace clean, no error spans anywhere. The interesting one is the final retry of the bad row on its own:

"Payload": "{\"Records\": [{\"body\": \"{\\\"product_id\\\": \\\"P209\\\",
  \\\"quantity\\\": \\\"unknown\\\"...}\", \"attributes\":
  {\"ApproximateReceiveCount\": \"5\"}}]}",
"localstack.aws.service.response": "{\"StatusCode\": 200,
  \"Payload\": \"{\\\"batchItemFailures\\\": [{\\\"itemIdentifier\\\": \\\"9c813f87-…\\\"}]}\"}"

The bad row arrives alone on its fifth receive, the invocation returns 200 with a single reported failure, and the trace contains no PutItem span at all, because the upfront validation rejected it before DynamoDB was ever called. It dead-letters on schedule. Nine items in the table, one message in the DLQ, main queue drained.

It then tested something nobody asked for: it re-sent an already-processed healthy row directly to the queue to confirm the deterministic key held. The table stayed at nine items. Finally it restored the production visibility timeout and maxReceiveCount before writing up its report.

We checked all of it by hand afterwards against the running emulator. The table held exactly nine items keyed P200#Warehouse A#2023-06-01 through P208#…, the DLQ held exactly one message, and the queue was back to 300 seconds with maxReceiveCount 5.

What a mock would have missed

Add up what the agent actually used to solve this, and almost none of it is an API response:

• The SQS to Lambda poller, whose batch delete-on-success behavior is the entire bug.
• The redrive policy and maxReceiveCount, which moved messages to the DLQ and later proved the fix bounded the retries.
• DynamoDB’s type validation rejecting a string in an N attribute.
• ApproximateReceiveCount on the redelivered message, showing the retry was the fifth.
• IAM policy evaluation events, which ruled out a permissions cause.

A stubbed SDK has none of that. There is no poller loop, so the poison-pill failure cannot occur; a mocked put_item returns success or whatever exception the test author imagined; and no mock emits an IAM evaluation event, because no mock evaluates policy. An agent iterating against mocks might have produced a plausible fix but, even then, would have no way to tell whether it actually worked.

The run cost $4.60 in tokens, took about 17 minutes, and cost nothing in cloud spend. It made 32 MCP tool calls, and App Inspector was the one it reached for most, at 16 of them. None of them touched an AWS account.

None of this is Claude-specific either. The server speaks MCP, so Cursor, VS Code, Codex, and the rest get the same tools from the same init command.

Summary

We handed a coding agent a broken event-driven pipeline and a bug report, with LocalStack running under IAM enforcement and App Inspector recording every call. It deployed the stack, reproduced both symptoms, read the failure out of the trace tree, and ruled out IAM from a policy evaluation event. Then it fixed the batch contract with partial batch responses and an idempotent key, caught a disabled event source mapping after the redeploy, and proved the result with a fresh feed before restoring production values.

The useful part is not that an agent can write a partial batch response. It is that it could tell whether the one it wrote actually worked, then keep going until it did, in an environment where being wrong twelve times in a row costs nothing. Give an agent a real cloud to break, and the diff that reaches you has already been tested against the thing that would have caught it in production.

Try it with the LocalStack MCP server, and see the App Inspector docs for what the traces expose.