Backend Integrations not working correctly

Incident Report for Ultimate AI's System Uptime

Postmortem

Around 19:32:13 UTC on 09.11.2023, an integration-middleware service started giving the following error on the impacted clients.

``` :\"Error: Request failed with status code 403\n


    This incident started post-upgrade of one of the K8s cluster's nodePools where the impacted service was operating, We checked the release notes of k8s to see if anything was introduced which might have caused the service to throw these exceptions.


* After a thorough investigation we concluded that 3 newly added NAT IP were not added to the allowed list of impacted clients which led to this incident. After an upgrade of our K8s cluster, the impacted service was migrated to a node, which was using the new NAT IPs for the egress traffic. Adding the new IPs to affected clients allow-list mitigated the issue.

Posted Nov 16, 2023 - 16:05 CET

Resolved

This incident has been resolved.

Posted Nov 10, 2023 - 12:23 CET

Update

We are continuing to monitor the services for any further issues.

Posted Nov 10, 2023 - 11:48 CET

Monitoring

A fix has been implemented. We are monitoring the services.

Posted Nov 10, 2023 - 10:56 CET

Update

We are still investigating the issue. We will keep you posted as soon as we have an update.

Posted Nov 10, 2023 - 10:22 CET

Update

We are still investigating the issue. We will keep you posted as soon as we have an update.

Posted Nov 10, 2023 - 09:46 CET

Investigating

We are currently investigating an issue with our Backend Integrations. We will keep you updated.

Posted Nov 10, 2023 - 09:15 CET

This incident affected: Backend Integrations.