Queued events and triggers are suffering a delay

Incident Report for Ultimate AI's System Uptime

Resolved

Due to a high and unexpected load paired with a recent change that made our queued events be less performant, since last night at around 00:30 CEST some asynchronous operations and triggers are suffering a delay to be consumed by our workers that could cause them to take up to 5 hours to execute, the mentioned events and triggers are the following ones:

- Inactivity timers and their related actions
- Chat ended events and their related actions
- Feedback reply triggers
- Delayed ticket messages

We rolled back the changes and the queue is getting back to a healthy state, our estimation is that it will be fully consumed by 21:00 CEST today, from that point onwards the delays should disappear.

Posted Sep 06, 2024 - 00:30 CEST