Session pages not loading in Backstage format

Incident Report for Swapcard

Resolved

The incident commenced around 6:30 PM UTC. We are now initiating an in-depth post-mortem analysis to unveil the root cause.

In our preliminary examination, we identified that the root cause was associated with the build and linking of OpenSSL in one of our dependencies, which was constructed using cross-compiled images. This resulted in glibc being inadvertently statically linked, leading to complications. The problem appeared sporadically and went unnoticed for a few hours. No issues have been detected since the deployment of this version in our production, staging, and testing clusters.

The problem has been rectified, and an additional layer of security has been implemented to prevent the recurrence of a similar issue.

We will soon furnish a more detailed post-mortem analysis.

Posted Nov 09, 2023 - 18:30 UTC