Some Zendesk chat bots are not operational
Incident Report for Ultimate AI's System Uptime
Postmortem

During a deployment to our Zendesk Chat Integration we introduced an issue to the bot initialization, causing chat bots to not start immediately after deployment. A few minutes after the deployment our bot monitoring service automatically restarted the bots and the majority of bots recovered.

Some bots did not recover as they were not authorized even though they were active at the time. This resulted in a locked state that could only be resolved by manual intervention on the database, which resolved the issue.

We have since pushed a fix for the initialization issue, we had introduced during the deployment. Additionally we have implemented validation to prevent active bots from being unauthorized, which was what prevented some bots from being able to recover.

Posted Mar 04, 2024 - 16:03 CET

Resolved
This incident has been resolved.
Posted Feb 28, 2024 - 14:38 CET
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Feb 28, 2024 - 12:13 CET
Identified
The issue has been identified and a fix is being implemented.
Posted Feb 28, 2024 - 11:35 CET
Update
We have identified a small number of bots that are potentially affected and will continue to investigate.
Posted Feb 28, 2024 - 10:59 CET
Update
We are continuing to investigate this issue.
Posted Feb 28, 2024 - 10:35 CET
Investigating
Some bots using Zendesk chat integration are not receiving messages.
We are investigating.
Posted Feb 28, 2024 - 10:34 CET
This incident affected: Chat integrations (Zendesk Chat).