Loss of Content Logs in APAC Region

We experienced a Content logging service interruption in our APAC region beginning at 02:31 UTC on 16.10.2024. The issue was caused by full disk space on one of our databases, which led to delays and potential data loss in data ingestion processes.

What Happened:

• Disk Space Limit Reached: Content logging database on our APAC-01 server reached full disk capacity.
• Data Ingestion Impacted: This caused delays in data processing and potential loss of data for some customers.
• Services Affected: Data ingestion and logs processing services were temporarily disrupted.

Actions Taken:

• Increased Disk Capacity: We added additional disk space to the affected server to accommodate data growth.
• Service Restart: Our team restarted the data processing services to resume normal operations.
• Monitoring: We are enhancing our monitoring systems to proactively detect and prevent similar issues.

Current Status:

• The issue has been resolved as of 02:50 AM UTC on October 17, 2024.
• Logs processing services have returned to normal operation.
• We are closely monitoring the system to ensure ongoing stability.

We apologize for any inconvenience this incident may have caused.