Resolved: Issues with Printix Cloud 2022-08-30

Printix Cloud is impacted by DNS errors at Microsoft Azure. As a result, Printix Cloud is not taking in any new connections.

• Azure customers running Canonical Ubuntu 18.04 experiencing DNS errors

As recommended by Microsoft we are working to restart Printix impacted services to resolve the issue. We will post updates to this announcement when we have news. Select Follow if you wish to be notified about these updates.

• 2022-08-30 13:00 (CET)
  This announcement is published, as we can see that Printix Cloud is not taking in any new connections and we restart the affected instance of the proxy2 service. System is back to normal at 13:50 CET.
• 2022-08-31 09:44 (CET)
  Restarted another service, as it was updated to the poisoned Ubunto version.
• 2022-09-01 09:59 (CET)
  Status changed to resolved.
• 2022-09-01 15:00 (CET)
  Incident report published below.

Incident report

The root cause of this incident was Azure deploying a poison Ubunto version.

On 2022-08-30 (Tuesday, August 30th. 2022) in the morning (CET) our development team was experiencing issues deploying updated services in both our Azure based development and test environments. After extensive research we found that it was not an error with our updated services but was caused by DNS errors introduced with the Azure update of selected virtual machines to Ubuntu version 18.04. We studied the recommendations to mitigate the issue as described by Azure. We then turned to our production system in Azure and could see, that Printix Cloud had not taken in new connections since 9:30 (CET). At 12:30 (CET) the system started to drop connections also, so at 13:00 (CET) we went ahead and restarted one of the instances of the proxy2 services (running on one of the updated Ubunto virtual machines). At 13:50 (CET) system is back to normal.

We continued to monitor the system and were waiting for Azure to mitigate the issue completely. On 2022-08-31 (Wednesday, August 31st. 2022) we saw another instances of the proxy2 service starting to drop connections at 09:05 (CET). We restart it at 09:10 (CET) and at 09:35 (CET) system is back to normal.

Azure mitigates the issue completely and on 2022-09-01 (Thursday, September 1st. 2022) we changed the status to resolved.

Friendly regards
The Printix Team