A warehouse ships duplicate packages after a consumer deployment. Refund and recall costs are rising.
Messaging incident · Senior
Orders are fulfilled twice after a consumer restart
At-least-once delivery meets a non-idempotent warehouse side effect.
Establish the operating contract
Incident brief
Order events are read from a partitioned log. The consumer calls a warehouse API, writes a local processed-event row, then commits its offset.
Each order may create at most one shipment, while no paid order may be silently skipped.
Constraints
- The log and local database support transactions independently, not together.
- The warehouse API accepts an optional client reference but does not document retention.
- Losing a paid order is worse than delayed fulfillment.
Build a causal model
Evidence timeline
Consumers restart after processing events but before committing offsets.
The same event IDs are delivered again, as designed.
The local deduplication row exists for some duplicates, but warehouse shipment IDs differ.
Operators propose committing offsets before calling the warehouse.
Reason before revealing
Your response
1Diagnose
Where is the unsafe gap in the current operation order?
Need a nudge?+
List every crash point between external effect, local record, and offset commit.
2Stabilize
How do you stop new duplicates without losing events?
Need a nudge?+
Keep at-least-once input and make the side effect safely repeatable.
3Recover
How do you identify and reconcile already duplicated shipments?
Need a nudge?+
Event IDs alone are not enough if the external provider created new identities.
4Prevent
Which patterns work when the transactional boundary ends at an external API?
Need a nudge?+
Think idempotency contracts, inbox/outbox records, and explicit state machines.
Compare reasoning, not wording
Model response
Reveal structured response
The warehouse call occurs outside the atomic boundary. A crash after shipment creation but before the dedup row or offset commit causes redelivery and a second external side effect.
- Pause the affected consumer partitions while retaining the log backlog.
- Require a stable shipment idempotency reference derived from order ID for every warehouse call.
- Query the warehouse by that reference before retrying ambiguous requests.
- Do not commit offsets early; that trades duplicates for silent loss.
- Join event IDs, orders, client references, and warehouse shipment records to locate duplicates.
- Define compensating actions—cancel, intercept, or refund—based on shipment state.
- Resume from known offsets with idempotent calls and bounded concurrency.
- Record every ambiguous external outcome for operator reconciliation.
- Persist an inbox/state-machine transition before dispatch and an outbox command for shipment work.
- Contractually define idempotency-key scope and retention with the warehouse.
- Make consumers safe under crash injection at every boundary.
- Monitor duplicate deliveries separately from duplicate business outcomes.
Use evidence-based self-assessment
10-point rubric
Your score0 / 10
Check only the signals your answer demonstrated.
Common traps
- Committing the offset before the side effect
- Assuming the broker can deduplicate warehouse calls
- Keeping deduplication records for less time than redelivery is possible
- Calling compensation without checking shipment state
Strengthen the underlying concepts