We are currently experiencing issues with Printix Cloud. A restart of selected services has been initiated, but there may be a period thereafter (expected 20 minutes) during which the release of documents via Printix App, printing that goes via the cloud, mobile printing, and Chromebook printing will not be stable. Direct print to printers on the local network is still possible. We will post updates to this announcement when we have news. Select Follow if you wish to be notified about these updates.

• 2021-02-18 11:35 (CET)
  This announcement is published.
  Connections are being re-established. Expect stability issues to be resolved at around 12:25 CET

• 2021-02-18 12:50 (CET)
  Issue marked as resolved.
• 2021-02-22 14:05 (CET)
  Incident report published below.

Incident report

On 2021-02-18 (Thursday, February 18th. 2021) at 10:50 CET one of our larger customers modified a very large number of print queues. This resulted in a massive number of messages being sent out to the customer's Printix Clients over a very short period. This triggered subsequent PagerDuty alerts to our backend team which restarts the selected services. 

Actions

• Same afternoon on 2021-02-18 at 15:00 development task (PP-6418) is created to implement a throttling mechanism per tenant for outgoing messages to the Printix Clients. 
• Same evening on 2021-02-18 at 19:00 CET an update to Printix Administrator is released to production to prevent this and show the message: Restriction for modifying multiple items exceeded.
• During the now completed scheduled maintenance on 2021-02-20 the number of proxy2 services was increased.
• As of 2021-02-22 it is estimated that 1-2 days of the development task (PP-6418) is remaining. This will be followed by setup and testing in our test environment. After that the new throttling mechanism will be deployed to production as part of scheduled maintenance on 2021-02-27.

Friendly regards
The Printix Team