Resolved: Issues with Printix Cloud 2022-09-01

We are currently experiencing issues with Printix Cloud. We are working to resolve the issues. We will post updates to this announcement when we have news. Select Follow if you wish to be notified about these updates.

During this period you may experience slow performance, inability to sign in and release documents via Printix App. Direct print to printers on the local network is still possible.

• 2022-09-01 10:20 (CET)
  This announcement is published, as we can see that Printix Cloud is becoming unresponsive.
• 2022-09-01 11:40 (CET)
  System is starting to accept connections again. However, due to the massive number of queued requests, it will be approximately one more hour before operation is back to normal..
• 2022-09-01 12:17 (CET)
  Status changed to resolved.
• 2022-09-01 16:50 (CET)
  Incident report published below.

Incident report

The root cause of this incident was a massive (unintended) update of computers to Printix Client version 1.3.1190.0.

On 2022-09-01 (Thursday, September 1st. 2022) we were again (after Azure DNS issues) in a position where we could start to deploy to production in preparation for Changelog 2022-06. Production was configured to update computers to the new Printix Client version 1.3.1190 at a reduced pace (1%). The deployment was also correctly configured for this reduced update pace. Unfortunately, the deployment exceed the timeout (20 minutes), and this caused Kubernetes to revert to the previous deployment, which had full update pace (100%). This resulted in the system becoming slow/unresponsive shortly after 10:00 (CET).

To get back to the reduced pace (1%) we had to re-deploy additional services. They were successfully deployed one by one and at 12:17 (CET) the system was back to normal. At 14:00 (CET) we extended the deployment timeout (40 minutes) and were successful at deploying the planed changes as part of Changelog 2022-06.

Friendly regards
The Printix Team