Issue currently under investigation

Major incident Content Platform - CH01
2021-03-17 12:20 CEST · 1 week, 1 day, 3 hours, 27 minutes, 13 seconds

Updates

Resolved

Root Cause Analysis:

The storage server of CP CH01 was repeatedly shutdown due to a erroneously applied space left halt condition on a log partition which filled up quickly due to massively increased customer activities. A relabeling process on all file systems later blocked a timely reboot of the server. Identifying this process and finally circumventing it via alternate boot options took a long time.

During the incident, upload/downloads and displaying of content item previews and thumbnails was not possible, and access for all Picturepark customers was intentionally blocked in order to prevent potential follow-up issues that can occur when storage servers are unavailable. The incident has been debriefed internally and actions taken in order to mitigate similar issues in the future. These actions include, among others, changing the default configuration parameters and alert levels as well as improving emergency procedures for similar incidents.

March 25, 2021 · 15:47 CEST
Resolved

We were able to resolve the issues with the Picturepark file storage at 3:04 pm CET and Picturepark has been running stably since then. We’re still executing extended tests on various services and are investigating the root cause which will take some time. We sincerely apologize for the inconvenience. Please contact your account manager or support for more information on this issue.

March 17, 2021 · 15:37 CEST
Update

We are still experiencing issues related to file storage resulting in no previews and no downloadable file content. We are resolving this as quickly as we can and will post updates here as they become available. We sincerely apologize for the inconvenience.

March 17, 2021 · 14:48 CEST
Issue

The platform is currently experiencing issues. We are currently investigating this and will update you here. Apologies for any inconvenience.

March 17, 2021 · 12:21 CEST

← Back