UPDATE: the issues should be fixed and our services are recovering. We are monitoring closely.
]]>We have identified that a server is currently not responding, we're looking into the source of the issue.
]]>Your website/server itself is not experiencing any internal problems. The outage is external and widespread.
Cloudflare is actively working on resolving the incident, and you can follow their updates here: cloudflarestatus.com.
We are monitoring the situation closely and will provide updates as soon as more information becomes available.
Update from Cloudflare 15:42: A fix has been implemented and we believe the incident is now resolved. We are continuing to monitor for errors to ensure all services are back to normal.
]]>Update 12:40: Our service provider is already working on fixing the mail delivery.
Update 13:30: The email delivery works as expected again. All queued emails should have been transferred now.
]]>We apologize for the inconvinence.
]]>[30.10.2024 14:30] We identified a major incident, that the majority of our servers have reported a short downtime.
]]>Electrical issues have been resolved and normal operations resumed.
]]>Update 11:35: The sites should be coming back available step by step.
Initial report 11:20: We have identified that a server is currently not responding, we're looking into the source of the issue.
]]>Update 02.10.2023 15:30: The project management tool at https://teamwork.ramsalt.com is accessible again.
]]>Update 15:02: One of the server's system disk reported to be broken. Booting up the services even though in RAID-mode (mirroring) is failing. We are working on a replacement.
Initial report 13:25 We have identified that a server is currently not responding, we're looking into the source of the issue.
]]>Update 12:09: We identified the issue and will restore the searches for all affected sites step by step.
Update 15:12: All searches should be working as expected again.
]]>We are very sorry for the inconvenience and the down time caused by this.
]]>See: http://travaux.ovh.net/?do=details&id=53798
[Update 10:10 GMT+2] In a previous update we said that "servers are coming back", this was not working, but we sent this to early. right now only a small fractiuon of the servers are back. We are still waiting on the other servers to come back online.
[Update 10:20 GMT+2] The network appears to be still quite unstable, some servers came back online for ~10 minutes, but now are not reachable again. We will keep you updated.
[Update 10:25 GMT+2] All servers appear to be back online and reachable.
[Update 10:50 GMT+2] All services are back online, however we still observe some anomalies. We will keep an eye on the situation to see if things get completely resolved.
[Update 11:15 GMT+2] Issue is fully resolved.
]]>The situation is under control and the sites should be marginally affected with only a few minutes of downtime. We apologize for the inconvenience.
ETA 01:15 (CEST/UTC+2)
[Update 01:25 CEST/UTC+2] We are validating the changes and will restart all services within 10 minutes.
[Update 01:45 CEST/UTC+2] The last checks took more than expected, but all services are fully functional now.
]]>[Update 12:10 GMT+1] All issues have been resolved.
]]>[Update 01:30 GMT+1] The issue is affecting 2 DC rooms where most of our servers are located. We are keeping the situation monitored.
[Update 02:00 GMT+1] The technicians are working to fix the issue, we will check in the status and post an update later tonight.
[Update 05:30 GMT+1] It seems that the problem is worse then anticipated. We were notified that a fire did spread through the dataceter. We are proceeding with a disaster recovery. We will keep you informed.
[Update 10:00 GMT+1] We are getting closer to recover the sites in a new data center with the backups that was stored outside of the data center that burnt down. We still don't have an estimate for when we can be finished.
Some more information about the incident:
[Update 11:25 GMT+1] We are getting nearer but there is constantly obstacles popping up. But soon we should see some sites coming up.
[Update 13:45 GMT+1] Due to the high amount of people migrating their plans the console for our service provider is also having some trouble, which is slowing down our restore process since every now and then we are unable to access it.
[Update 14:47 GMT+1] Most of our servers are not in the data center that burned down (SBG2), but in one of the 3 nearby data centers are closed down and partly damaged due to the fire in SBG2. The plan for recovery of the 3 other data centers are still 1-2 weeks so we can't wait for that and are still working fully to recover all sites from cold backups.
[Update 15:20 GMT+1] We are starting to get more sites online, we will contact site owner so they can verify that the recovery has gone well.
[Update 16:47 GMT+1] We are progressing and are getting even more sites online.
[Update 18:00 GMT+1] Most of our clients should see their sites coming up in the new data center during this evening. Many of our sites are already running well. Our clients might need to update their DNS so we will be in touch with you if that is the case, during the evening and tomorrow morning.
[Update 19:45 GMT+1] Our backups comes out in randomized order so it has been hard to estimate when sites are going to come up. We are still progressing fine with more sites coming up during the evening.
[Update 20:45 GMT+1] All server infrastructure is fully recovered, we are now working to restore the remaining sites. Some sites require a DNS change, if this is the case for your site you should have received a notification email and/or in TeamWork. If your site is still down it's CRITICAL that you check your email and update the DNS if required.
[Update March 11th 06:45 GMT+1] We continue to resolve issues. Get in touch with support@ramsalt.com or your Project Manager if any issues on the newly created servers pops up.
[Update March 15th 10:00 GMT+1] All sites are restored from backups. Get in touch with support@ramsalt.com for any issue remaining.
]]>Affected servers:
Problem is found and will be available in ETA ~15 minutes.
[Update 14:30] Issues to Prod10 are resolved.
[Update 14:45] We are still working on a fix. New update in ~15-20 minutes
[Update 15:15] We applied a fix to Prod09 and the sites hosted there should be back in the next few minutes.
[Update 15:25] Ths issue is fully solved for Prod09, we're still working on Prod11.
[Update 15:45] Ths issue is solved also on the last remaining Prod11 server
]]>[Update 14:15] It appears that our service provider is having some network connectivity issues, we're looking into possible solutions for the sites hosted on Prod06 and Prod08
[Update 14:35] An update from our service provider says that 80% of the services have been restored, the other 20% is currently being migrated and should be coming back soon.
[Update 16:15] Our service provider has updated us: around 12200 + 3200 IPs were initially not reachable and by 16:00 only 457 + 554 IPs remain inaccessible. Sadly both Prod06 and Prod08 servers are in that ~1000 IPs. We expect the servers to come back online at any time, but for the whole situation to be solved we might need to wait up to 4~6 hours,. another update when we get some extra info.
[Update 18:30] Still over 100 IP not reachable, but the situation should be solved within 1-2 hours.
[Update 19:20] Both servers are back up and operational, performance might still be affected, but overall everything should be back to normal. We will keep an eye on the situation and post an update if necessary.
]]>We're monitoring the situation and preparing to take actions if this persists.
[Update 15:40 UTC+2] The issue seems to being handled on our provider side, we still see some performance degradation however all our sites are back up and running.
[Update 15:55 UTC+2] All performances are back to the initial values, the issue is resolved.
]]>We expect this maintenance to be very quick and the downtime for the hosted sites to last only a couple of minutes.
]]>This is a temporary issue and our service provider will be fixing as soon as possible.
We will update this as we have other news.
[18:10 CEST] Our service provider has identified the issue and have been restoring some services, we expect the issues to be soon fixed.
[18:20 CEST] All our sites are currently up and our provider stated that problems should be solved. We'll monitor the situation.
]]>[Update 12:50] Load on the server worsened, but now it seems to be stabilizing, more updates shortly.
[Update 13:45] We have identified a wrong configuration which might have caused a chain of events leading to the current issue. We're now updating the config and restarting the server. this should fix the performances issues and bring all sites back up.
[Update 13:55] The configuration update solved the core issue, server is back at full capacity and sites should be coming up any second now. We're monitoring the situation however core problems are solved.
]]>Currently we had to clear the backlog of emails due to the huge number of unsent emails, from our analysis the majority of emails did not contain any crucial information and were caused by a software bug which caused a incredibly high amount of mails to be sent.
If you experienced issues with email delivery in the past few days please get in touch with us via support and we will look into it, currently we're monitoring the situation and looking into solutions to prevent this from happening in the future.
]]>The situation is under control and the sites should be marginally affected with only a few minutes of downtime. We apologize for the inconvenience.
]]>The situation is under control and the sites should be marginally affected with only a few minutes of downtime. We apologize for the inconvenience.
[Update 10:25] We found some issues following the maintenance and wee had to perform a server reboot, sites should be coming up shortly.
[Update 10:45] All issues should be resolved, we'll be monitoring the situation for a while more.
]]>Server is now starting up and sites should be back up shortly.
[Update 10:50 UTC+2] Server health is restored, some performance issues while everything stabilises, other than that issues should be solved.
[Update 10:58 UTC+2] All issues should be resolved.
]]>A short downtime is expected of around 10-15 minutes.
]]>You can still contact us via support@ramsalt.com for critical issuers.
Update: Problems are resolved.
]]>Update 16.42: This is a major issue at our hosting vendor. We are following the issue and will give you more information as soon as we can.
Update 17.12: The issues still persists, but the hosting partner identified the issue and it's working on a fix.
Update 17.19: The fix is gradually being applied on the host servers and most sites are coming back up. Should all be solved shortly.
Update 17.29: All sites are back up, we are monitoring the status.
]]>08:58 Update: Reinstalling the software is causing us some further issues, we're working to get it back working as expected. ETA: 15-20 minutes
09:14 Update: We were able to setup the correct version of the software, but it's still behaving not as expected, we're now cleaning up some configurations and restarting the services one by one.
09:35 Update: Software is up and running and within the next few minutes all sites should be coming back online.
09:45 Resolved: All sites are back up and running.
]]>The upgrade will be applied sequentially on all servers between 05:00 AM and 06:00 AM (Oslo timezone). It should cause only an imperceptible or minimal impact , a short downtime might occur on some sites, however should be limited to 1-2 minutes.
This upgrade will provide:
Faster websites with the added support for HTTP/2. Speed comparison here; Technical info here.
Safer HTTPs requests with improved DH Params file. tech info: here and here
Important Note: Both improvements will only be visible on your site only if you have HTTPS enabled on your site, if you don't have it already get in touch with us and we will set it up on your site!
]]>In addition we patched 5 of our Drupal 7 sites that have been affected by the same security issue.
]]>Our technical team is aware and prepared for the security release and patch the corresponding sites automatically as soon as possible after the release. If we identify, that also Drupal 7 sites are affected, we will perform the required updates, too.
You can find more information on drupal.org.
]]>Due to this maintenance sites hosted on this server will suffer short downtime, service should be fully restored in about 30 minutes. If your site is still experiencing issues after the maintenance window contact our support.
Do you need more info? Curious to know which server is your site currently hosted on? Chat with us and we'll be happy to answer you!
]]>[Update 11:07] Seems like the issues are getting fixed, we'll keep an eye on it.
It seems that TeamWork is experiencing some trouble, we're sorry about this, in the meanwhile you can still contact us for critical support requests at support@ramsalt.com
]]>Update 11:33: We identified the issue with the service provider and have fixed it. We're now monitoring that everything works as expected. Technical info here.
Update 11:40: Everything should be back to normal.
]]>Update 11:33: We identified the issue with the service provider and have fixed it. We're now monitoring that everything works as expected. Technical info here.
Update 11:40: Everything should be back to normal.
]]>Update (12:15): We identified the issue and working with out service provider to fix it.
Update (12:25): The host server encountered issue and required an emergency reboot, all sites should be working now.
Update (12:30): Some performances issues still occurring, we'll be keeping an eye on it.
Update (12:40): All seems good now.
]]>Update: all systems are functional.
]]>[Update 04/05/2018 07:40]
All apps are running fine, however the server has some issues and some services needs to be restarted which will cause a short downtime. We will monitor the situation trough the day and restart such services tonight, unless the issue becomes critical before that.
[Update 05/05/2018 09:20]
We will perform now the reboot required.
[Update 05/05/2018 09:35]
Reboot is completed and all Apps are coming back online.
[Update 05/05/2018 09:40]
All systems are functional.
]]>The Ramsalt Lab team is are preparing for this and will start automated deployment of those security patches as soon as they are released. We will make sure to monitor the progress and make sure that this vulnerability will not be exploited on the sites managed by us.
If you want more info get in touch with us via Teamwork and we'll answer your questions.
]]>Added notes:
Update 07.12.2017 10:25: We have identified the issue as being a OCH router in Amsterdam. Our service provider is working to get this fixed.
Update 07.12.2017 14:00: From what we can see the issue is fully resolved.
]]>Update 19:50: We have identified the problem, now working on a fix.
Update 20:15: Problems are solved, all sites should be up and running in moments. We will keep an eye on the situation.
]]>We are now keeping an eye on the situation to prevent further issues.
Update 19:50: All seems good, closing this report.
]]>Update 10:20: We are working on this issue closely with pout service provider, right now it seems it could be still a fallout of the issues following the datacenter blackout.
Update 11:15: The problem with Prod 01 OVH still persists, but it has bee identified to be an issue is with a network port on the server which is currently being replaced.
Update 12:20: Our service provider's specialists are still at work to bring Prod 01 back online, the intervention is taking more than expected, we will update you as soon as we have other news.
Update 14:20: The hardware issue on the network interface seems to be worse than expected. Technicians are still at work, we were also able to get in touch with the CEO of our service provider which is trying to help speed up the process.
]]>Vi har fulgt opp saken tett med OVH, og de har fått fikset maskinen som serveren deres er på, og dere nettside er oppe og går. Opplever dere problemer med deres nettside etter nedetiden, ta kontakt med support@ramsalt.com. Supportteamet er parat til å hjelpe.
Det er fortsatt veldig mange nettsider rundtom i verden som er nede pga av problemene med datasenteret til OVH så akkurat nå er vi bare glade for at vi ble prioritert, omsider. Vi beklager nok en gang på det sterkeste denne ekstremt lange nedetiden vi har opplevd. Vi har aldri opplevd noe lignende. Vi vil sette oss ned og se på eventuelle tiltak som kan gjøres for å unngå en slik nedetid i fremtiden, selv om sjansen for at et helt datasenter med flere hundre tusen servere går ned på samme tid alltid kan skje.
For de av dere som har spørsmål ikke nøl med å ringe oss eller send oss en e-post på support@ramsalt.com.
]]>OVH har publisert en lengre forklaring på hva som feilet i går fra OVH, engelsk versjon helt nederst
]]>Oppdatering 10.11.17 07:00: Nettverket er nede og serveren svarer ikke. Vi har gjennom hele natten prøvd å få tak på OVH, men de svarer ikke. Det er en vanskelig situasjon, men vi håper å få løst den nå på morgenen.
Oppdatering 10.11.17 08:30: Fortsatt ingen løsning fra OVH. Vi følger opp.
]]>Fortsatt ingen løsning. Vi følger med, men gleden av å være oppe ble kortvarig for noen sider.
]]>Vi vil sende ut info om problemet på mail til teknisk kundekontakt.
OPPDATERING: Vår leverandør OVH oppdaterer status fortløpende: https://twitter.com/ovh_support_en
Underliggende forklaring fra leverandør:
This morning we had 2 separate incidents that have nothing to do with each other. The first incident impacted our Strasbourg site (SBG) and the 2nd Roubaix (RBX). In SBG we have 3 datacentres in operation and 1 under construction. In RBX, we have 7 datacentres in operation.
SBG: In SBG we had an electrical problem. Power has been restored and services are being restarted. Some customers are UP and others not yet. If your service is not UP yet, the recovery time is between 5 minutes and 3-4 hours. Our monitoring system allows us to know which customers are still impacted and we are working to fix it.
RBX: We had a problem on the optical network that allows RBX to be connected with the interconnection points we have in Paris, Frankfurt, Amsterdam, London, Brussels. The origin of the problem is a software bug on the optical equipment, which caused the configuration to be lost of and the connection to be cut from our site in RBX. We handed over the backup of the software configuration as soon as we diagnosed the source of the problem and the DC can be reached again. The incident on RBX is fixed. With the manufacturer, we are looking for the origin of the software bug and also looking to avoid this kind of critical incident.
We are in the process of retrieving the details to provide you with information on the SBG recovery time for all services / customers. Also, we will give all the technical details on the origin of these 2 incidents.
We are sincerely sorry. We have just experienced 2 simultaneous and independent events that impacted all RBX customers between 8:15 am and 10:37 am and all SBG customers between 7:15 am and 11:15 am. We are still working on customers who are not UP yet in SBG.
Best, Octave
]]>