Christopher Bouzy
A
thread 1/8
At approximately 12:30 PM EST, we observed a disruption in the availability of several of our services. Upon conducting a thorough investigation, it became apparent that several of our servers had become unresponsive.
04:35 PM - Nov 13, 2023 (Edited)
Avatar Avatar Avatar
0
55
247
Christopher Bouzy
A
thread 2/8
Subsequently, we promptly contacted our data center provider and were informed that they were experiencing technical difficulties.
04:35 PM - Nov 13, 2023
1
60
Christopher Bouzy
A
thread 3/8
Roughly 30 minutes later, our entire platform experienced an outage due to the failure of the redundant systems we had in place, resulting from the simultaneous shutdown of all our servers.
04:35 PM - Nov 13, 2023
0
65
Christopher Bouzy
A
thread 4/8
At approximately 2:54 PM EST, we received notification that the issues had been resolved, albeit requiring a complete restart of all our servers. This endeavor is significant in scale as it encompasses several dozen servers, a substantial portion of which are part of clustered configurations.
04:35 PM - Nov 13, 2023
0
72
Christopher Bouzy
A
thread 5/8
While network-related challenges are not uncommon, the necessity of restarting all our server instances is an unusual occurrence. We are currently in the process of resynchronizing services such as search, which remains outstanding.
04:35 PM - Nov 13, 2023
0
72
Christopher Bouzy
A
thread 6/8
We sincerely apologize for the extended downtime and any inconvenience this may have caused. We are actively exploring options to mitigate such incidents in the future, including the establishment of a secondary data center.
04:35 PM - Nov 13, 2023
1
106
Christopher Bouzy
A
thread 7/8
Presently, all our servers are housed within a single data center, leaving our services vulnerable to widespread disruption in the event of an issue.
04:35 PM - Nov 13, 2023
0
74
Christopher Bouzy
A
thread 8/8
By implementing a secondary data center, we aim to enhance redundancy and enable traffic rerouting, thereby minimizing the likelihood of similar outages.
04:35 PM - Nov 13, 2023
17
140
Richard Head
A
Not that any of us users could have done a thing about it, but it's really nice to have a breakdown of what happened.

Thank you.
In response to Christopher Bouzy.
09:38 AM - Nov 14, 2023
0
8

 

{{ notificationModalContent }} {{ promptModalMessage }}