Zendesk Chat Outage
Incident Report for Ultimate AI's System Uptime
Postmortem

Zendesk bots connect to Zendesk Chat using websockets, which are a very unstable way of communicating between backend services. An unknown event caused the websockets to end up in a recovery loop where they could not recover from.

We deployed a fix to try and prevent the recovery from going in these loops, and after a while the websocket connections were able to initialise properly.

During the starting time of our incident, there was a Zendesk incident which could be the cause.

Posted Jun 07, 2023 - 09:31 CEST

Resolved
We have resolved the incident and all services should have returned to normal operations.
Posted May 26, 2023 - 13:50 CEST
Monitoring
The system has recovered and we are monitoring the situation.
Posted May 26, 2023 - 13:39 CEST
Update
We are continuing to work on a fix for the issue.
Posted May 26, 2023 - 13:22 CEST
Identified
We are currently investigating an issue, which impacts the following features & services:
- zendesk integration

We will provide an update as soon as possible or at least once every 30 minutes.
Posted May 26, 2023 - 12:47 CEST
This incident affected: Chat integrations (Zendesk Chat).