Networks are missing. We are working to resolve the issue. We will post updates to this announcement when we have news. Select Follow if you wish to be notified about these updates.
- 2021-02-01 14:10 (CET)
The issue is now resolved. - 2021-02-02 09:38 (CET)
Added below instructions (Workaround to get computers back on known network). - 2021-02-03 13:03 (CET)
Incident report published below.
Workaround to get computers back on known network
Some customers has reported, that they have computers, which when they look at the Computer properties page appears to be on Unknown network even though their gateway IP and MAC address is that of a gateway that belongs to a known named network.
- Open the Networks page. Select (+) to Create a new network. Name the network (Example 00test), so it will appear at the top of the list. The network does not get any gateway.
- Leave the created network (00test) for a few minutes and check that the computer(s) that appeared as being on an Unknown network is now showing the correct network name matching the gateway IP and MAC address.
- On the Networks page select the vertical three dots for the network you created (00test) and select Delete.
- Uncheck Move printers to another network.
- Select OK.
Incident report
On 2021-02-01 (Monday, February 1st. 2021) at 13:05 PM CET new software was deployed to the Printix Cloud. Before we deploy to production, software is tested in our test environment and before that it is tested in our development environment. At 13:12 PM CET we had the first support request indicating that networks were missing. At 13:38 PM CET the issue was escalated to our backend team. At 13:54 PM CET the first version of this announcement was published.
At 14:02 PM CET the backend team had established the root cause as a human error in the production deployment configuration for the network service. Configuration was corrected and networks reappeared in the Printix Administrator. At 14:12 PM CET this announcement was updated to reflect that the issue was resolved. Feedback via support requests confirmed that networks were now back.
In the subsequent hours we received support requests from customers indicating they had computers that appeared to be on Unknown network even though their gateway IP and MAC address was that of a gateway belonging to a known named network.
At 16:32 PM CET the backend team had developed a tool that should update the computers network settings. The tool ran for 1.5 hour.
On 2021-02-02 (Tuesday, February 2nd. 2021) at 9:38 AM CET we updated this announcement with 'Workaround to get computers back on known network'. We published the workaround, as a minor set of customers were still experiencing that some computers appeared to be on Unknown network even though their gateway IP and MAC address was that of a gateway belonging to a known named network. Through additional analysis of the Printix Client log files retrieved from the customer's computers an update was made to the tool, so it would also ensure that computers that came online after the tool has run would also have their network settings updated. During the day, we used the mentioned workaround to resolve customer support requests.
At 18:01 PM CET the revised tool was started, and it completed at 19:27 PM CET. Since then we have not had any further support requests regarding this.
Actions
- It was the first time we experienced this particular issue (caused by a configuration error) and due to its nature, it was not caught by our PagerDuty alert system. We will add triggers in PagerDuty to also catch issue if number of networks suddenly drops.
- Develop additional recovery tools that can be used in the unlikely event that a misconfiguration should happen again.
- To reduce risk of misconfigurations in the future the QA process for configuration changes will be updated and sharpened.
- Seen in the hindsight, we should have published a new announcement, once we had changed the status to resolved of the first announcement.
Friendly regards
The Printix Team
Comments
0 comments
Article is closed for comments.