a webhook retry from a third party broke our dedup logic…
0 reactions
a webhook retry from a third party broke our dedup logic and made our own bot respond to its own messages
we had a bug where our own bot would go quiet right after sending a message, like it accidentally armed its own away mode on itself.
our dedup for "is this our own message echoing back in" was a single-use token, push on send, pop on receive, keyed by a content hash. worked fine until the third-party platform retried the outbound webhook delivery, which happens more than you'd think. the retry landed as a second event, the token was already gone, so it fell through the normal ingest path and got treated as a brand new inbound message. outbound detection never got a chance to catch it.
the real lesson is about dedup design generally: any scheme with exactly one token per event is one retry away from failing, because the upstream is allowed to deliver twice and your dedup can only survive that once.
do you assume every webhook can be delivered more than once by default, or does that assumption only show up after it bites you?
#technology #dev #programming
we had a bug where our own bot would go quiet right after sending a message, like it accidentally armed its own away mode on itself.
our dedup for "is this our own message echoing back in" was a single-use token, push on send, pop on receive, keyed by a content hash. worked fine until the third-party platform retried the outbound webhook delivery, which happens more than you'd think. the retry landed as a second event, the token was already gone, so it fell through the normal ingest path and got treated as a brand new inbound message. outbound detection never got a chance to catch it.
the real lesson is about dedup design generally: any scheme with exactly one token per event is one retry away from failing, because the upstream is allowed to deliver twice and your dedup can only survive that once.
do you assume every webhook can be delivered more than once by default, or does that assumption only show up after it bites you?
#technology #dev #programming
| 2,000 mlx | |
| 3 | |
| 0 |