Minor Issue - Functionality - Edgraph Looker - US Central

Incident Report for EdCast by Cornerstone

Postmortem

On September 27th, the Cornerstone team found Security patch auto update applied to GCP Cloud Run Functions without compatibility testing with EdGraph service. The CloudOps team addressed the issue and taken necessary actions.

The security patch update did not include sufficient advance communication from GCP, leaving teams unprepared to test and mitigate any breaking changes and no backward compatibility or mitigation measures were provided.

Input Validation: Future patches include compatibility testing and should be added under planned maintenance.

Enhanced Monitoring & Maintenance: Disable auto updates & proactive testing to mitigate risks from such Security/third-party updates.

Posted Oct 14, 2024 - 16:50 PDT

Resolved

The CSOD Technology Team observed a performance degradation with Edgraph Looker Functionality affecting US Central Production. The problem began at 4:28 am Pacific Time and was resolved at 7:28 am Pacific Time. During this time, clients with portals on US Central Production may have experienced slow performance or intermittent errors while accessing Edgraph Looker Functionality.
Posted Sep 26, 2024 - 04:30 PDT