Tuesday 8th September 2020

Prod 08 [resolved] Servers not responding. (Prod06 and Prod08)

We have identified that a server is currently not responding, we're looking into the source of the issue.

[Update 14:15] It appears that our service provider is having some network connectivity issues, we're looking into possible solutions for the sites hosted on Prod06 and Prod08

[Update 14:35] An update from our service provider says that 80% of the services have been restored, the other 20% is currently being migrated and should be coming back soon.

[Update 16:15] Our service provider has updated us: around 12200 + 3200 IPs were initially not reachable and by 16:00 only 457 + 554 IPs remain inaccessible. Sadly both Prod06 and Prod08 servers are in that ~1000 IPs. We expect the servers to come back online at any time, but for the whole situation to be solved we might need to wait up to 4~6 hours,. another update when we get some extra info.

[Update 18:30] Still over 100 IP not reachable, but the situation should be solved within 1-2 hours.

[Update 19:20] Both servers are back up and operational, performance might still be affected, but overall everything should be back to normal. We will keep an eye on the situation and post an update if necessary.