We are providing a detailed post-mortem report regarding a production change that caused continuous emails to be sent again (duplicates) to some attendees.
The impacted scope was limited to events created before February 4, 2025 that still had continuous emails enabled at the time of the incident.
The goal of this post-mortem is to share what happened, why it happened, how it was resolved, and what we are doing to prevent a recurrence.
Summary
A background task runs every 30 minutes to process continuous email campaigns:
- Retrieve all enabled continuous email campaigns
- For each campaign, verify whether each targeted attendee already received the corresponding email
- Send the email only to attendees who have not received it yet
The incident was caused by the "already received" check relying on a database that only started being populated after February 4, 2025. For events created before that date, the delivery history existed in a different (legacy) datastore. As a result, the system incorrectly concluded that certain attendees had not received the email yet, and re-sent it.
Timeline
- 2026-02-04 07:56 UTC | A change was released that caused continuous emails to be incorrectly sent again (duplicates) to some attendees.
- 2026-02-04 11:10 UTC | The change was reverted, stopping further duplicate sends.
Resolution
We resolved the incident by reverting the change introduced at 2026-02-04 07:56 UTC, restoring the previous behavior and preventing additional duplicate emails from being sent.
Root Cause Analysis
The root cause was an incorrect assumption in the delivery-history lookup logic:
- The deduplication check queried a datastore that only contained delivery records populated starting 2025-02-04.
- For events created before 2025-02-04, delivery records were stored in a legacy datastore.
- Because the check did not account for the legacy source, it returned false negatives ("not sent yet"), resulting in duplicate emails being sent on subsequent background-task executions.
Immediate Actions Completed
- Reverted the problematic change to stop duplicate email sends.
- Identified and confirmed the gap between legacy and current delivery-history datastores for pre-2025-02-04 events.
Short-Term Improvements
- Update the "already received" check to consult the correct source for both legacy and current events.
- Add safeguards to prevent re-sending when delivery history is missing/ambiguous (fail-safe behavior).
- Add targeted tests covering pre-2025-02-04 events with continuous emails enabled.
Long-Term Improvements
- Consolidate delivery-history storage to a single source of truth (including backfill/migration strategy for legacy data).
- Introduce idempotency guarantees at send-time (e.g., campaign+attendee idempotency keys) to prevent duplicates even if checks regress.
- Add monitoring and alerting on abnormal duplicate-send rates for continuous email campaigns.
Conclusion
We recognize the impact that duplicate emails can have on attendee experience and event organizer trust. We take full responsibility for this regression and are implementing the preventative measures above to ensure legacy data paths are consistently handled and duplicates are prevented by design.
If you have any questions or concerns regarding this incident, please reach out to our support team.