We are currently experiencing issues with Printix Cloud. We are working to resolve the issues. We will post updates to this announcement when we have news. Select Follow if you wish to be notified about these updates.

During this period you may experience slow performance, inability to sign in and release documents via Printix App. Direct print to printers on the local network is still possible.

• 2023-02-28 10:15 (CET)
  A Printix Client update was unintentionally started across all customers rather than the intended limited number. Printix Client connections were then dropped by systems, causing disruption to the Printix Cloud.
• 2023-02-28 10:30 (CET)
  Printix Cloud Services team was alerted by service monitoring and immediate escalation was made.
• 2023-02-28 10:33 (CET)
  This announcement is published.
• 2023-02-28 11:00 (CET)
  The software update rate was adjusted down to stop the problem from escalating further.
• 2023-02-28 11:33 (CET)
  This announcement is marked as resolved.
• 2022-02-28 11:54 (CET)
  A second announcement is made, as it is discovered the issue is not completely resolved. The demand on connections was passing through the Printix Cloud causing further disruption to customers.
• 2022-02-28 11.15 to 14:30 (CET)
  Additional capacity was added throughout the Printix Cloud to relieve demand bottlenecks and to return the service to normal operation.
• 2022-02-28 15:56 CET
  All connections were back online and normal operating state confirmed.
• 2022-02-28 15:57 (CET)
  The second announcement is marked as resolved.
• 2023-03-03 15:48 (CET)
  Incident report published. The second announcement has been deleted, as it was in reality one incident.

Incident report

The root cause was an error in deployment configuration resulting in the update rate for Printix Client not being adjusted down to the correct level.

The update rate was subsequently reduced to 1% to stop Printix Clients requesting updated software. This addressed the cause of the incident but not the demand that had already been triggered. Additional storage, server and CPU capacity was added to the Printix Cloud to allow all outstanding connection requests to be appropriately handled and for the service to be stabilized. Changes were also made to throttling within the Printix Cloud.

Actions

• Updating release management process.
• Default values will be re-evaluated to reduce the chance of a similar error in the future.
• Additional monitoring and alerting will be added to provide greater insight to service quality.
• Automatic scaling will be developed to speed adjustment to future increases in demand.
• Additional throttling logic will be added to better manage high demand.

Friendly regards
The Printix Team