What Happened

On Tuesday 8 June at 09:58 UTC, we experienced an increase in error rates for ~1h in the Exhibitor center and with customers using the vanity url (*.app.swapcard.com & app.swapcard.com remained functional during the entire incident) due to the major failure of our CDN provider Fastly (https://status.fastly.com/incidents/vpk0ssybt3bj), which is used to provide vanity urls and worldwide access across vanity urls.

The major issue was a global outage from Fastly provider, which also impacted most of the biggest tech companies and media websites including: Reddit, CNN, Paypal, Twitter, Amazon, Twitch, and so on.

We received an automatic alert that the service started facing issues ~09:59 UTC.

Resolution

We immediately created an incident room with a dedicated engineering team and support team to mitigate the impact on the domain we can control (meaning team.swapcard.com). In Swapcard we have the Multi CDN architecture for Swapcard domains. After ensuring our backup CDN was ready and configuration was okay, we immediately triggered the switch to use our emergency CDN (Cloudfront) for team.swapcard.com. In addition, we had provided transparent communications about the incident on our Twitter account and status page (https://www.swapcardstatus.com/).

Following this incident, and in favour of providing the best service to our customers, we will evaluate a Multi CDN architecture for the vanity urls. Although this extremely rare issue was not under our control, it is important to ensure that in the future we can address this issue faster and hot switch from one CDN to another for vanity url.

Impact

A degraded experience in the exhibitor center (team.swapcard.com). For customers using vanity url, error messages were shown when accessing the event app (vanity url only) or exhibitor center. *.app.swapcard.com & app.swapcard.com remained functional during the entire incident.