System Latency on GCP Asia region

Issue Summary:

On February 17, users in the GCP Asia region experienced increased latency due to elevated database CPU utilization triggered by background processing activity. During the affected period, some users observed slower response times while accessing the platform.

Root Cause:

The incident was caused by a product defect in a background processing queue that generated excessive database activity. This resulted in high CPU utilization on the database servers, which temporarily degraded system responsiveness and increased service latency in the affected region.

Corrective Action:

As an immediate mitigation, the number of connections used by the background processing queue was reduced to lower the load on the database. This action stabilized database utilization and restored normal platform performance.

Preventive Measures:

To reduce the likelihood of recurrence, the following actions are being implemented:

• Query optimization: Improving the database queries executed by the background processing queue to reduce unnecessary load.
• Queue processing improvements: Enhancing the queue processing mechanism to better control resource usage and prevent similar spikes.