Following a global incident in OpenAI Azure manifesting as timeouts for all calls requiring generative capabilities, all of our bots were timing out intermittently. This resulted in degraded performance across all of our customers, including bots that do not rely on such capabilities. As soon as OpenAI had fixed the incident on their end, our bots resumed to full operation. A small subset of bots participating in the zero shot experiment continued to experience issues thereafter, and so the feature had to be disabled for them.

Through this incident we have discovered an unwanted resource dependency between bots using LLM capabilities and bots that do not use it, a dependency which should not have been there. As per further investigation, this dependency was introduced recently by zero shot functionality. We are exploring a range of short to long term solutions to reach full isolation for similar future cases.