The incident was caused by an interaction between a recurring data integrity job and a recent improvement to badge scanning. Each behaved correctly on its own, and the fault condition existed only while both were active at the same time.

Every three days at 12:00 UTC, an automated job rebuilds the search index behind leads and connections. Its purpose is to guarantee completeness: it republishes every record from our primary database into the search index, so that a lead can never be missing from a search result or a filtered list because of a synchronization gap.

The job builds a fresh copy of the index alongside the live one and switches over only when the copy is complete. While that copy is being filled it is not yet read by anyone, so the step that makes newly written records searchable is deliberately switched off to keep the rebuild efficient; live changes continue to be written into the copy so that it stays current, and the setting is restored when the rebuild completes.

On 28 August we deployed an improvement to badge scanning so that a scanned lead is written to the search index immediately, rather than relying only on background synchronization. This makes a new lead appear in lead lists as soon as it is scanned. To guarantee that, each scan waited for confirmation that the lead had become searchable before returning a response to the app.

During a rebuild, a scan writes to both the live index and the copy being rebuilt, and this confirmation applies to every index a write touches. The scan therefore waited on the copy where that step had intentionally been switched off, so the confirmation could never arrive and the request waited until it reached its internal 30 second limit. That exceeded the time the scanning app waits for a reply, so the app treated the scan as having no connectivity and correctly fell back to its offline queue, showing the error and offline indicator that users reported. The offline safety net worked as designed; what was misleading was that it engaged on devices that were in fact online.

Because the underlying requests still completed successfully after that wait, they were recorded as slow rather than failed, which is why automated monitoring did not raise an alert and why detection depended on onsite reports and a manual review of platform logs.

The condition can only arise while a rebuild of this particular index is in progress, and it was introduced by the scanning improvement deployed three days earlier. In hindsight the overlap between the two should have been covered before that improvement was released: our testing exercised each behaviour individually, but not the two together. On this occasion the condition also did not clear on its own, because the rebuild stopped before completing and therefore never restored the setting it had changed, which is why manual intervention was required.

To restore service, the index being rebuilt was reconfigured so that it no longer held scan requests waiting for a confirmation it was unable to provide. Scans returned to normal within one minute of that change, and onsite teams confirmed that scanning worked immediately afterwards. Scans that had been stored on devices in offline mode synced normally once scanning recovered.

A permanent code fix was written, reviewed and deployed to production the same day. Writes sent to an index that is being rebuilt no longer wait for that index to make the record searchable, while writes to the live index still do. The two are now sent as separate requests, so a rebuild in progress can no longer delay a write. The fix applies to every part of the platform that writes to a search index, not only to badge scanning, and it addresses the cause rather than the symptom.

We are strengthening our processes to prevent recurrence. Beyond the code fix already deployed, we are also adding automated alerts on scan processing times and on this class of internal timeout, so that a degradation which completes successfully but slowly is detected within minutes rather than through onsite reports. Our automated test suite now covers a live write occurring while an index rebuild is in progress, which is the exact overlap that was missing.

Reliable scanning is a priority for us and we have treated this incident accordingly: the cause has been removed from the platform, and the gap that delayed detection is being closed.