Why repeats are normal
In outbox mode, an entry in the delivering state whose reservation lasts longer than 300 seconds returns to the pool and can be fetched again. A failed acknowledgement returns the entry to the pending state and increases the attempt counter. With this design, the final receiver will now and then see the same payload a second time, and none of these mechanisms is a bug.
The guarantee is therefore at-least-once, not exactly-once. For an alarm this is the right choice: it is better to receive an alarm twice than not at all. The price of this choice falls on the recipient, which has to be resistant to repeats.
Practices 1 to 3: the event key and the marker
Practice one: define the key that identifies an event. The best option is a combination of the event type, the identifier of the object and the time of the event. In reception events every payload carries the time in ISO 8601 format in the Europe/Warsaw time zone, which makes it possible to build such a key without looking anything up in the source system.
Practice two: record the processing marker atomically, before you perform the side effect. The pattern used in the system itself is a SET NX marker in Redis, which blocks duplicates of the alert for a failed training. Setting the marker is an atomic operation: only the first recipient takes it, and the next one sees that the entry is taken.
Practice three: release the marker when execution fails. The system does exactly that: a failed send releases the lock so the alert can go out again. A receiver that marks an event as handled before the action is performed, and does not roll the marker back after an error, turns a temporary failure into a permanent loss of the alarm.
Parameters that drive repeats
Practices 4 and 5: acknowledgements and ordering
Practice four: acknowledge delivery at the end, and rely on the idempotence of the acknowledgement. The acknowledgement accepts the status delivered or failed with an optional error description, and acknowledging an entry that is already delivered changes nothing. The receiver can therefore safely repeat an acknowledgement after losing the network response.
Practice five: do not assume an order. Fetching a batch reserves up to 89 entries atomically, but the order in which they reach the final recipients depends on the poller and the network. For events that change state (a group update, then an interruption), the recipient should compare them by the time of the event, not by the order of arrival.
Practices 6 and 7: resilience to failure and observability
Practice six: an error in the receiver must not stop the source. The system sends reception events asynchronously with a 5-second timeout, so a webhook failure does not interrupt a guest's registration. On its side, the receiver should respond quickly and process in the background, instead of holding the connection open until a complicated action is finished.
Practice seven: log every event with its key and its result (processed, skipped as a duplicate, error). Without this it is impossible to tell a lost message from a correct rejection of a repeat. In integrations with the exchange log, such a record should be separate from the application log.
Typical mistakes we see in receivers
The most common mistake is deduplicating by the text of the message instead of by the event key: two different alarms with the same text are then merged into one. The second is a marker with zero lifetime, which allows reprocessing after a restart. The third is acknowledging delivery before the action is performed, so a failed action is never retried.
Before you run the receiver in production, send the same event twice and check that the channel shows one entry, and that the log has one entry for processing and one for skipping.
Checklist
Video
Frequently asked questions
Does the system guarantee delivery exactly once?
No. Retry mechanisms give at-least-once delivery, so the receiver should be idempotent.
What is the best way to deduplicate events?
With a composite key of the event type, the object and the time of the event. The message text is not a good key.
What if the receiver acknowledges delivery twice?
Nothing bad happens. An acknowledgement of an entry that is already delivered changes nothing.
A receiver that tolerates repeats
We will go through a double-event test and the processing log with your integrator.
Let's talk