# We killed our event relay 25 times on purpose

"Exactly once" is a claim you should test by breaking things. So while building the event log for Hafen, we ran kill -9 on the relay 25 times during a 5,000-event write.

The failure we cared about

A job finishes, writes to Postgres, and needs to tell the rest of the system. If you write to the database and then publish, a crash between the two loses the event. If you publish first, a crash leaves an event for work that never happened.

For an AI platform this is expensive. Agents retry tool calls, and each page fetched or credit used has to count exactly once.

The pattern

There are two halves, and neither works alone:

[ OUTBOX TO CONSUMER ]
transaction
outbox
relay
relay
event log
consumer
  • Outbox. The event is written in the same transaction as the work. A relay publishes the rows and deletes them only after the log acknowledges them.
  • Deduplication. The consumer records each event in processed_events in the same transaction as its handler, so a redelivered event is skipped.

The outbox gives you at-least-once delivery. If the relay dies after publishing but before deleting, it publishes those rows again. Deduplication turns that into exactly-once effects. Nothing here makes the broker exactly-once, and you don't need it to.

An event that fails five times goes to a dead topic with the reason and the original event ID, so one bad event can't block the rest.

What the crash tests showed

[ CRASH TESTS ]
  • kill -9 the relay 25 times during 5,000 events4,846 duplicate copies in the topic. 0 missing, 0 applied twice
  • kill -9 the log server mid-write3,000 messages for 3,000 events. A new group applied all 8,104 events once
  • Two consumer groups, then a restartEach resumed from its own offset, re-reading 0 events
  • Third member joins a groupPartitions rebalanced from 8/8 to 6/5/5
  • An event that always failsEnds up in dead, once per group
5 of 5 done
[ 25 KILLS, 5,000 EVENTS ]

  • 4,846

    duplicate copies

  • 0

    events missing

  • 0

    applied twice

The first row is the one to remember. The system produced almost one duplicate for every event, and the result was still correct. If you only test the happy path, you never see those 4,846 copies, and you'd never know whether your deduplication works.

We also kept one automated test that recreates what a crash leaves behind: rows already published and still in the outbox. With deduplication turned off, it fails. That's what makes it a real test.

What we couldn't verify

The PR listed what the local stack couldn't prove:

[ NOT VERIFIED ]
  • Docker-wide seccomp default on a real host
  • io_uring on CI runners
  • Service discovery in production
0 of 3 done

A crash test on one machine says nothing about a network partition. Write that list down, because otherwise "verified" quietly starts to mean more than it should.

Then we replaced the broker

This ran on Apache Iggy. We've since decided to move to NATS JetStream. Iggy's Node SDK calls itself a work in progress and had about 120 downloads a week. It also needed an unreleased edge build to recover consumer groups after a restart, and io_uring, which Docker's default seccomp profile blocks.

The broker changed, but the pattern and the checks didn't. The port has to pass the same crash tests. That's the real value of this work: the tests belong to the behaviour you need, not to the tool you happen to be using.