The system is now running with normal performance again. The incident report can be seen below.

We sincerely apologize.

Friendly regards
The Printix Team

Incident Report

This outage and slow performance started at 16:20 PM GMT on Monday March 2nd, 2020 and lasted for approximately 2 hours. There was no disruption of local printing during this period and the services including the Printix Administrator Dashboard was available intermittently during this period.

A lot of log reading and analysis has been performed. The root cause of the issue points to a bottleneck in the Printix Cloud, mainly caused by an increased demand in I/O resources on one of the database clusters. This is due to an overall increased load on the Printix Cloud due to an increasing amount of customers. The issue occurred during the peak load period. 

Findings, learning and actions

• Low priority back-end jobs and maintenance services are suspended for the rest of the week to ensure stability while performing a number actions increasing the capacity on one of the database clusters.

• Moving the History log to a new Cloud Database Service starting 8:00 AM GMT on March 4th, 2020. The old history log will not be available from the Printix Administrator, but available through a support request. The new history will build it self automatically. This new service is more scalable.

• Move more databases to a new and more scalable Cloud Database Service during the weekend of March 7th and 8th, 2020. This will cause some unavailability.

• Upgrade the existing Database cluster with more RAM and CPU power for the transition period to the Cloud Database Service.

• A number of new monitoring points are identified and are analyzed on an ongoing basis going forward.