October 14th availability incident analysis

shayfrendt
Mon, 14 Oct 2024 15:01:55 +0000
General

Good morning Glide Community. For many of you, I know that your business or client businesses you serve are off to a chaotic start due to the availability incident that occurred overnight (Pacific Time).

Our oncall engineers responded to the incident and were able to help our systems recover after an extended period of time lasting more than 4 hours.

We are currently aware that the following functionality experienced interruptions:

Glide Big Tables, SQL Data sources, Big Query

Syncing with external data sources (Google Sheets, Airtable, etc.)

Integrations (Glide AI, PDFMonkey, DocsAutomator, etc.)

Call API

This extended outage that affected so much of the platform is very unfortunate, and we always seek to avoid this type of negative impact.

Please accept our apologies for these interruptions! We know that you count on Glide to be able to run your business, and we will do everything we can to ensure strong platform stability and reliability.

Today I am working with our platform engineering team to conduct an incident postmortem review and root cause analysis in order to document the facts around what led to this incident.

Our goal is to learn why the changes that we made to our production infrastructure resulted in negative impact, while testing those same changes on our (pre-production) staging environment did not encounter the same failures.

Once we are able to fully understand what unexpected condition was encountered in our production systems, we will update this post with additional context. I expect that to be today or tomorrow.

11 posts - 5 participants