Live Status\Incident History\Image Transform Queue Degraded
Degraded
Image Transform Queue Degraded
- Resolved
The backlog was worked through and the queue length has been consistently at zero for about 40 minutes now. Marking as resolved & will dig deeper into the underlying cause to put in place more robust handling for this type of error in the future.
- Monitoring
The cluster is healthy and processing messages from the queue again. There is a backlog of several million images; it will take a few minutes to churn through the transformations. We will update when the queue has cleared and in-flight versions' images have finished generating.
- Identified
We've identified an error that caused a cascade of crashes in the cluster. We're rebooting the instances & working on a fix to the underlying issue.
- Investigating
The queue that handles image transformation jobs is currently backed up causing delays in generating new dataset versions. We're looking into the issue & will provide updates when we know more.