My AWS AI Agent Passed Its Tests. It Still Missed a Check.

I gave my AI agent a small deployment history and asked how often the checks were unhealthy. It wrote the Python, ran its own tests, and came back with the right answer.

Then I read the tests. They passed, but they did not cover everything I had asked for.

Some context: this was a small experiment in my personal AWS lab, with made up deployment data. There is no production incident behind it.

The input was a small CSV

I started with this CSV. It has twelve deployment checks, five minutes apart. Each row records whether the check was healthy, the expected and running application versions, the instance counts, and any active alarms.

timestamp,status,desired_version,running_version,healthy_instances,desired_instances,active_alarms
2026-09-20T10:00:00Z,healthy,2026.09.20.1,2026.09.20.1,3,3,0
2026-09-20T10:05:00Z,healthy,2026.09.20.1,2026.09.20.1,3,3,0
2026-09-20T10:10:00Z,unhealthy,2026.09.20.1,2026.09.19.4,2,3,1
2026-09-20T10:15:00Z,unhealthy,2026.09.20.1,2026.09.19.4,2,3,1
2026-09-20T10:20:00Z,unhealthy,2026.09.20.1,2026.09.19.4,2,3,1
2026-09-20T10:25:00Z,healthy,2026.09.20.1,2026.09.20.1,3,3,0
2026-09-20T10:30:00Z,healthy,2026.09.20.1,2026.09.20.1,3,3,0
2026-09-20T10:35:00Z,healthy,2026.09.20.1,2026.09.20.1,3,3,0
2026-09-20T10:40:00Z,unhealthy,2026.09.20.1,2026.09.20.1,2,3,0
2026-09-20T10:45:00Z,unhealthy,2026.09.20.1,2026.09.20.1,2,3,0
2026-09-20T10:50:00Z,healthy,2026.09.20.1,2026.09.20.1,3,3,0
2026-09-20T10:55:00Z,healthy,2026.09.20.1,2026.09.20.1,3,3,0

This is the exact file I gave the agent, so you can run the same experiment yourself.

I wanted two numbers: how many checks were unhealthy, and how long each completed unhealthy period lasted.

The second one needed a rule, so I wrote it down. A period starts at the first unhealthy check and ends at the first healthy check after it. If the file ends while checks are still unhealthy, that period stays incomplete.

For example, the first unhealthy check is at 10:10 and the next healthy one is at 10:25. That makes the period fifteen minutes. This only tells me what the supplied checks show. It does not tell me when a real application failed or recovered, and I wanted that clear before handing the calculation to an agent.

The model wrote the Python, a tool ran it

I used Amazon Nova 2 Lite as the model, running through my managed agent setup. I gave it the calculation rules, three small test cases and the CSV. I did not write any of the Python for it.

It generated the program and sent it to Amazon Bedrock AgentCore Code Interpreter, which runs code in a separate execution environment.

The whole flow looks like this:

Supplied CSV → model generates Python → Code Interpreter runs it → result comes back

On my side, I did two things. I compared the returned numbers with my expected result, and I read the generated tests.

I could see the exact Python the agent ran and what came back. The output included the test pass marker and the results, so I did not have to take the model's word for it.

The numbers matched, the tests had a gap

Before the agent run, I had worked out the expected results myself. The output matched:

The 41.7% is just five unhealthy checks out of twelve. It does not mean the application was down for 41.7% of the time.

The program also printed MODEL_GENERATED_TESTS=PASS. That is the program telling me its own tests passed.

But there was a gap. I had asked the agent to write a test for every expectation I listed, including the exact start and end time of each unhealthy period. Its tests only checked how many periods there were and how long each one lasted. Nothing checked the start and end times.

The answer was still right, and the start and end times in the output were correct. But if they had been wrong, those tests would not have caught it.

So "the tests passed" only means the tests the agent wrote passed. It does not mean it wrote every test I asked for. That is why I did two separate things: compare the result with my own numbers, and read the tests.

What this run does and does not show

It showed an agent writing and running analysis code, and the output matched a result I had calculated myself. It also showed why I needed to read the generated tests.

It does not show that every future run would work. Malformed inputs, access restrictions, session behaviour and repeated runs still need to be tested.

For this kind of analysis, I would keep three things together: the input, the code that actually ran, and its output. Then I would check the generated tests against my original requirements before trusting the pass marker.