We would like to present you with a retrospective report regarding a service disruption that affected Swapcard customers on the 18th of September, 2023, from 4:55 AM UTC to approximately 8:00 AM UTC.
The purpose of this retrospective is to provide insights into our initial assessment of the incident, as communicated on the Swapcard status page, and to outline the corrective actions we've taken to restore service. This incident pertains to issues related to logged-in and unusual logged-out scenarios, which resulted in several internal server errors.
On Monday, September 18th, at 04:55 AM UTC, we encountered an issue with user logins and logouts, along with various internal errors on the delivery of the service. The nature of the problem was inconsistent, which led to a slightly longer resolution time than usual. This was primarily because not all requests were failing, and users could still access the product.
The root cause of this problem was the automatic activation of our security protection system when it detected a potentially malicious or system-flooding. Specifically, one particular request triggered our Web Application Firewall (WAF) to flag certain IPs as a security precaution. Subsequently, our corporate rules blocked actions originating from these flagged IPs. Unfortunately, this also affected one of Swapcard's own IP addresses being incorrectly blocked, resulting in a range of issues.
Immediately following the incident, the Infrastructure Team at Swapcard, in collaboration with our Site Reliability Engineers (SREs) & Security Team, identified the root cause and implemented a mitigation strategy to prevent incidents related to the affected components, specifically knowing the criticality of this components inside our architecture.
Our systems & team detected a disruption in traffic, and as a result, the Swapcard Incident Response team was promptly activated. Our team worked diligently to prioritise and restore the quality of services to minimize the impact on our customers. Concurrently, we conducted a thorough investigation into the cause of the issue and implemented mitigations with the assistance of dedicated members from the Infrastructure and SRE teams.
As part of our mitigation plan, we are enhancing our Web Application Firewall (WAF) system for flagged IPs by implementing a range of rules. This is aimed at preventing recurring issues where certain IPs are mistakenly flagged due to the actions of an individual who has been flooding our system.
Please note that the events are presented in chronological order with their respective timestamps for clarity and coherence.
In accordance with our high standard in terms of deliverability, Swapcard has conduct and planned several improvement on the relative components to prevent further incident of same type. We also improve our capacity to detect earlier this issues, for preventing impact on our customer. Procedures and controls are already in place but this incident highlights the need for improvement.