Two ECS failures that produce no useful error

5 min read

A readiness probe blocked by the task's own IAM role, and readonlyRootFilesystem silently killing ECS Exec. Both look like broken infrastructure.

Both of these cost me hours, both are one-line fixes, and neither produces an error that points anywhere near the cause. They are also the sort of thing you only meet in production, because locally there is no IAM role and no read-only filesystem.

One: the readiness probe your IAM role forbids

Two services were permanently unhealthy. Tasks started, ran briefly, were marked unhealthy, were killed, and were replaced — forever. The application logs looked fine right up to the moment each task disappeared.

The health check was doing what health checks often do, which is check everything the service might need:

app.get('/ready', async (req, res) =&gt; {
  await db.query('SELECT 1')
  await s3.send(new HeadBucketCommand({ Bucket: UPLOADS }))   // ← the problem
  res.sendStatus(200)
})

That HeadBucket call needed s3:ListBucket on the uploads bucket. The task role did not have it, because this particular service never actually touched uploads — the check had been copied from a sibling service that did.

So: a permissions gap the service did not care about, turned into a crash loop by a probe that asserted more than the service needed. And because the task never went healthy, it was replaced before anyone could exec into it and look. The failure prevented the investigation.

What made it hard

The signal was "task never became healthy", which reads as infrastructure. I looked at the load balancer, the target group, the grace period, and the security groups before I read the probe. Nothing in ECS says your health check got a 403 — it reports only that the check did not pass.

The fix, and the rule

Assert what the service needs to serve traffic right now, and nothing else:

app.get('/ready', async (req, res) =&gt; {
  await db.query('SELECT 1')
  res.sendStatus(200)
})

A readiness probe that over-reaches converts an unrelated permissions problem into an outage. If a dependency is only needed by one endpoint, that endpoint can fail on its own — that is a degraded feature, not a dead service.

Log the failure inside the probe as well. A probe that returns a bare 500 with nothing written down is a puzzle you have set for yourself.

Two: readonlyRootFilesystem silently breaks ECS Exec

This one is worse, because the thing that breaks is the tool you would use to debug the first one.

readonlyRootFilesystem: true is good hardening and generally worth having. It also stops the SSM agent that backs ECS Exec, which needs to write to the filesystem to work.

What you get is execute-command failing, or a session that will not establish. Nothing is logged saying the agent could not write. enableExecuteCommand is true, the IAM permissions for SSM are correct, and it still does not work.

The fix

Give the agent somewhere writable:

{
  "readonlyRootFilesystem": true,
  "mountPoints": [
    { "sourceVolume": "ssm", "containerPath": "/var/lib/amazon" },
    { "sourceVolume": "tmp", "containerPath": "/tmp" }
  ]
}

with matching tmpfs or empty volumes in the task definition. You keep the hardening and you keep the ability to get inside a container.

Why it is worth knowing about specifically

Hardening and debuggability trade against each other, and that trade should be a decision you make on a quiet afternoon rather than a discovery you make during an incident. The failure mode here is precise and nasty: you harden the container, everything works, and weeks later — when something is wrong and you need to look inside — you find the door is bricked up.

Why do both of these waste so much time?

The error describes the observation, not the cause. "Task never became healthy." "Session could not be established." Both true, neither actionable, and both quite far from the thing that was actually wrong.

Both are IAM-adjacent without being IAM problems. One is a permission the service did not need and the probe demanded. The other is not a permission issue at all, but reads exactly like one, which sends you down the wrong path.

Both only exist in production. On your laptop there is no task role and the filesystem is writable. That is the category of bug worth writing down, because you cannot meet it early and you will meet it at the worst moment.

The system these came out of is a multi-tenant clinical platform I designed and built — 95 TypeScript source files, 29 test files, 36 migrations. It is built and owned by me and has no live client, so nothing here was a customer-facing incident. The architecture and the tenant-isolation tests are public:

github.com/jaklabs/telehealth-platform-reference

Before taking my word for any of it, the free website check is a public unauthenticated endpoint that drives headless Chromium at whatever URL a stranger types in — which is an SSRF liability unless the boundary is real. It is a fair sample of how I build the parts nobody sees.

If you have a system where the quiet failures are the expensive ones and you want someone to go looking, tell me what you are running.

5 min read