Messaging service delays
Incident Report for Ultimate AI's System Uptime
Postmortem

One of our customers experienced a malicious conversation. This exposed an unforeseen inefficiency in our database layer. A temporary fix was deployed within 90 minutes of incident discovery. The fix limited the speed of consuming events, in effect limiting the load on the database it caused. This stabilized the production systems. A long-term solution introduced a hard limit on conversation length. According to our analytics, no production bots should be affected by this limitation.

Posted Feb 07, 2024 - 14:50 CET

Resolved
The issue related to delayed bot messages has been resolved. If you happen to encounter this behavior again, please let us know at support@ultimate.ai.
Posted Jan 31, 2024 - 17:16 CET
Monitoring
A fix has been implemented and we are monitoring the system.
Posted Jan 31, 2024 - 17:11 CET
Identified
The issue has been identified and we are working on a resolution.
Posted Jan 31, 2024 - 17:08 CET
Update
We are continuing to investigate this issue.
Posted Jan 31, 2024 - 16:56 CET
Update
We are continuing to investigate the issue and will provide further updates once available.
Posted Jan 31, 2024 - 16:45 CET
Investigating
We are currently investigating an issue with our messaging service where we have seen delays in bot responses.
Posted Jan 31, 2024 - 16:42 CET
This incident affected: Chat integrations (Sunshine, LiveChat.com CRM Integration, Zendesk Chat, Salesforce, Freshchat, Zendesk Support Automation).