Fraud decisions depend on events arriving within 30 seconds. A warehouse sink slows during maintenance.
Messaging incident · Senior
A slow sink stalls the event pipeline
Consumer lag, oversized batches, and unbounded buffering turn degradation into data loss risk.
Establish the operating contract
Incident brief
Producers → partitioned log → enrichment workers → warehouse and fraud sinks. Workers poll 5,000 events and buffer outputs in memory before acknowledging input.
Fraud events must not be lost or reordered per account; warehouse freshness may degrade for two hours.
Constraints
- Fraud processing has priority over analytics.
- Events can be replayed for 24 hours if retention remains healthy.
- Dropping warehouse events is not allowed, but delaying them is.
Build a causal model
Evidence timeline
Warehouse latency rises 20×; fraud sink remains healthy.
Worker memory climbs and garbage-collection pauses exceed poll heartbeat intervals.
Partitions rebalance repeatedly and replay large unacknowledged batches.
Producer latency increases because broker retention disks approach their alert threshold.
Reason before revealing
Your response
1Diagnose
Trace how one slow sink causes rebalances and upstream pressure.
Need a nudge?+
Follow buffer growth, pauses, heartbeats, replay, and retention.
2Stabilize
How do you isolate the critical fraud path?
Need a nudge?+
Separate queues, concurrency, acknowledgement, and service classes.
3Recover
How do you drain backlog without causing another collapse?
Need a nudge?+
Control batch size and rate by observed downstream capacity.
4Prevent
Where should backpressure be explicit in this topology?
Need a nudge?+
Every bounded buffer needs an owner, limit, and overload policy.
Compare reasoning, not wording
Model response
Reveal structured response
Warehouse output blocks shared workers. Large in-memory batches cause pauses, missed heartbeats, rebalances, and replay amplification. Growing retention then threatens producers and the critical fraud stream.
- Isolate or pause the warehouse sink while keeping its offsets and backlog durable in the log.
- Reserve workers, partitions, and broker capacity for fraud traffic.
- Reduce batch and concurrency sizes to bound memory and heartbeat delay.
- Extend retention capacity or reduce noncritical ingress before disks reach emergency limits.
- Resume warehouse consumption with adaptive rate limits below measured sink capacity.
- Drain oldest backlog first where ordering permits and monitor queue age, not just count.
- Avoid mass consumer restarts; ramp partitions gradually and verify heartbeat stability.
- Reprocess from committed offsets and validate idempotent warehouse writes.
- Use independent consumer groups and resource pools for fraud and warehouse sinks.
- Bound every queue and define pause, spill, shed, or reject behavior.
- Alert on lag age, retention headroom, replay ratio, processing time, and useful throughput.
- Load-test slow-sink and recovery conditions with production-like backlogs.
Use evidence-based self-assessment
10-point rubric
Your score0 / 10
Check only the signals your answer demonstrated.
Common traps
- Adding larger in-memory buffers
- Restarting all consumers simultaneously
- Watching message count but not oldest-event age
- Allowing analytics and fraud to share every resource pool
Strengthen the underlying concepts