Around 14:23 UTC on 2024/09/02, our system began predicting non-existent intents for Zero Shot (ZS) bots, which inadvertently triggered actions like tags appearing on tickets. The issue arose after we released a new inference
version to production. Upon investigation, we identified that the problem was caused by an error in our cache logic within the IntentsFetcher class. Specifically, there was a mistake in how the cache key was generated to determine if the same bot was requesting the data. This error only impacted Zero Shot bots due to a recent modification in the ZS pipeline. This has not impacted the end-users directly.Because this bug did not generate any service errors, it was challenging to detect immediately. However, once identified, we quickly mitigated the impact by rolling back the affected service to a previous version, which has now resolved the issue.