Independent AI intelligence Two editions daily · ET
Fervor AI

Analysis · November 19, 2026 · concept

Microsoft FoundryFoundry Agent Serviceagent-infrastructureagent-securitymulti-agent

Microsoft Foundry's Crash Recovery Reruns Your AI Agent. Make Sure It Doesn't Email Twice.

Long-running agents can now survive a crash. The catch is one sentence in the docs, and it decides whether your customer gets one refund or two.

Long-running AI agents have a dull, expensive failure mode. The job runs for twenty minutes, the container restarts halfway through, and the work vanishes. So the platforms are doing the obvious thing and adding crash recovery.

Microsoft did it on November 18. Hosted agents in Foundry Agent Service, Microsoft's managed service for running agents in Azure, gained "resilient execution": opt-in recovery that picks a job back up after a crash, restart or redeploy. The announcement calls it generally available.

Here is the sentence that matters, straight from that post: "Recovery reruns your handler, so save progress as you go and make sure actions like sending an email aren't repeated."

Read it twice. Recovery doesn't resume your agent where it stopped. It starts your code again from the top. Anything the agent did before the crash, it may do again.

That changes how you should write an agent. A platform that recovers crashed work can invoke the same logical work more than once (engineers call this at-least-once execution), and it hands the "not more than once" part back to you. The rest of this piece explains what Foundry actually saves, what it doesn't, and the small pattern that keeps a recovered agent from sending the same email, payment or ticket twice.

What "recovery" actually means here

Think of a cook who faints halfway through a recipe. A colleague takes over. If that colleague can see the half-chopped onions on the board, they carry on. If the board was wiped, they start the recipe from step one, and the soup gets salted twice.

Foundry's recovery is the second kind of colleague, unless you leave notes.

The Microsoft Learn page on resilience for long-running hosted agents spells out the mechanics. Before your handler (the function that does the agent's work) starts, the runtime saves the input and takes a lease, a claim that says "this worker owns this job." If the process disappears, a later process reclaims the abandoned lease and calls your handler again with the same work identity and the same input.

Then comes the line that should shape your design: "Recovery reenters the handler from its beginning." The page says plainly that this isn't deterministic replay. Local variables and the in-memory call stack are gone.

So what survives? Per the same page:

  • The work identity and the saved input.
  • Small durable metadata values you write yourself.
  • Replay events for streamed output, so a client that reconnects can pick up the stream from a cursor.

And what doesn't: your progress. The doc puts the burden squarely on you: "Your agent is still responsible for preserving meaningful progress and preventing duplicate side effects."

Two more details trip people up.

First, recovery covers only part of the traffic. The Learn page says full crash recovery applies to stored background responses (jobs your agent runs in the background and saves, rather than answers streamed live to a user) when the server opts in, and foreground responses "aren't reinvoked after a process interruption." A chat turn the user is watching won't come back on its own.

Second, recovery isn't retry. Retry handles a failure your handler reports, like a 500 from an API, and it spends your retry budget. Recovery continues the same durable attempt after the process itself vanished. You need to handle both, and they fail differently.

Why this is the new normal

None of this is a Microsoft quirk. Any system that restarts crashed work faces the same choice: run things possibly twice, or risk running them zero times. Losing work silently is worse, so platforms pick "possibly twice" and document it.

What's new is who now meets the problem. Durable-workflow engines have handled this for years, and Microsoft says a Temporal integration is coming, with public preview planned for early 2027. But a lot of agent code gets written by people who've never had to think about a handler running twice. The agent framework hides the loop, the model decides which tools to call, and the side effects hide inside tool functions with friendly names like send_followup.

Agents also make the duplicate worse than in ordinary code. A rerun handler doesn't just repeat the same lines. It calls the model again, and the model may take a different path. On the first run it wrote a polite email. On the rerun it writes a slightly different polite email and sends that too. Now your customer has two versions of the same promise.

That's the shift worth naming. Crash recovery used to be an infrastructure feature you could ignore. Now it's a setting on the agent platform, and turning it on changes the correctness of every tool your agent can call.

The fix: idempotency keys plus a watermark

The Learn page gives the recipe in two sentences: "Persist the watermark before an operation that can't be safely repeated, and clear it after the operation commits." A recovered handler checks the watermark before it issues the operation again.

In plain words, two tools do the work.

An idempotency key is a unique ID you attach to an action so that sending it twice has the same effect as sending it once. Many payment APIs support this directly. Stripe, for example, accepts an Idempotency-Key header and returns the original result if it sees the same key again. The Learn page tells you to use downstream idempotency support "whenever it's available."

A watermark is a note you write to durable storage that says "I'm about to do step X" and, after it succeeds, "step X is done." On a rerun, your handler reads the notes and skips finished steps.

Build the key from things that stay the same across a rerun: the work identity Foundry gives you plus the step name. Never from a timestamp or a random UUID generated inside the handler, because the rerun would mint a new one and defeat the point.

Here is the shape of it, as Python-style pseudocode. Swap in your own storage and providers.

def handle(work_id, payload, store):
    # Step 1: draft the reply (safe to repeat, it only costs tokens)
    draft = store.get(work_id, "draft")
    if draft is None:
        draft = model_write_reply(payload)
        store.put(work_id, "draft", draft)   # save progress as you go

    # Step 2: send the email (not safe to repeat)
    key = f"{work_id}:send-email"
    status = store.get(work_id, "send-email")
    if status == "done":
        return
    if status == "started":
        # we crashed mid-send last time: ask the provider, don't guess
        if provider_already_sent(key):
            store.put(work_id, "send-email", "done")
            return
    store.put(work_id, "send-email", "started")   # watermark BEFORE the call
    send_email(draft, idempotency_key=key)
    store.put(work_id, "send-email", "done")      # clear AFTER it commits

Notice the trick in step 1. Saving the draft means a rerun reuses the exact email the model already wrote instead of asking for a new one. That closes the "two different polite emails" problem along with the duplicate.

Notice the honest gap in step 2, too. If the process dies after the send but before the "done" write, the watermark says "started" and you don't know whether the email went out. That's why the key matters. With a provider that honors idempotency keys, sending again is harmless. Without one, you need a way to check (provider_already_sent), or you accept a rare duplicate and say so.

Keep the stored notes small. The Learn page recommends treating durable metadata as a checkpoint index: a checkpoint ID, the last completed phase, an idempotency key, and a pointer to bigger state in a database or blob store. Don't stuff whole transcripts in there.

Put this into practice this week

You don't need to redesign anything. Do these in order.

1. List every side effect. Open your agent's tool definitions and mark each tool that changes something outside the agent: sends a message, moves money, creates or closes a ticket, writes to a customer-facing database, calls a webhook. Read-only tools (search, fetch, query) are safe to rerun. Everything you marked needs a key or a watermark.

2. Give each marked tool a stable key. Build it from the work identity plus the step name. If the downstream API accepts an idempotency key, pass it. If it doesn't, put a small "outbox" table in front of it, a list of actions waiting to go out: write the intended action with its key first, and have a sender skip any key it has already delivered.

3. Save model output before acting on it. Anything the model writes that a later step depends on (a drafted email, a chosen refund amount, a plan) goes to storage before the action. On a rerun, read it back instead of regenerating it.

4. Test it by breaking it. Turn on resilient execution in a test agent, start a background job that sends something to a sink you control (a test inbox, a request logger), and redeploy the agent mid-run. Then count. You want exactly one message per job. Two means a tool is missing its key.

5. Do the same discipline for the network. The same release shipped network egress controls: an ordered allow and deny list for outbound traffic, enforced outside the agent's sandbox. The egress guide says rules match top to bottom, first match wins, and evaluation is fail-closed (when in doubt, block): "if the policy can't be evaluated, the request is denied." Start in Audit mode with a default of Deny, which logs what would be blocked without blocking it. Then switch to Enforced and deploy a new agent version, because running sandboxes don't reload policy changes. A blocked host then gets an HTTP 403.

Egress and idempotency solve different halves of one problem. Egress decides where the agent can reach. Idempotency decides how many times it reaches there.

Honest limitations

Not everything around them is GA. The November 18 post calls resilient execution and core egress controls generally available, but related pieces are still preview, including Azure Policy governance for egress and Transform rules (which add or rewrite outbound headers). Earlier versions of the docs carried preview labels too, so check what your subscription and region actually get before you rely on it in production.

Recovery won't save foreground work. Only stored background responses on an opted-in server get full recovery. Interactive chat turns that die mid-answer stay dead.

Some sessions forget more than others. Foundry also added memory snapshots in preview, which start new sessions from a saved ready-to-run state to cut cold starts. The blog warns that in snapshot sessions, "Memory and local files, including $HOME, don't survive an idle period." If your watermarks live on local disk, snapshots will erase them. Put them in external storage.

Idle and retention limits apply. Idle timeout is configurable from 2 minutes to 4 hours, with a 15-minute default, and the blog says a session is deleted after 30 days of inactivity when snapshots aren't on. A job paused for human approval longer than that won't be there to recover.

Not every action can be made safe. Some downstream systems offer no idempotency and no way to check what already happened. For those, the watermark narrows the duplicate window to a crash between the call and the "done" write. It doesn't close it. Decide whether a rare duplicate or a rare miss is worse for that action, and write that choice down.

The part only you can do

Crash recovery is good news. Jobs that used to die at minute nineteen will now finish. But the platform made a trade on your behalf: it would rather run your code twice than lose it.

So walk your tool list today. Every tool that sends, pays, posts or closes something gets a stable key and a watermark, and every model output a later step depends on gets saved before you act on it. Then redeploy a test agent mid-run and count the emails.

If the count is one, your agent is ready for recovery. If it's two, you found the bug before your customer did.

Sources: Hosted agents in Foundry Agent Service at Ignite 2026 (Microsoft, Nov 18, 2026); Resilience for long-running Microsoft Foundry hosted agents (Microsoft Learn); Add guardrails to hosted agents: network egress controls (Microsoft Learn); Stripe idempotent requests. Pages read November 19, 2026.