Page 1 of 3

Server's down

Posted: Mon Apr 24, 2017 2:32 am
by Poisonous
See title!

Re: Server's down

Posted: Mon Apr 24, 2017 5:45 am
by Kilaana
Woke up to find out the server went down. Sorry to those who were expecting otherwise.

It seems to be a problem with the server host. They had a scheduled maintenance some hours ago to replace a failed drive in their systems, but that window has passed. I'm not sure why TER's server is still down, if it is part of the cluster that was affected.

I've submitted a ticket for it, fingers crossed we'll hear back from them soon.

Re: Server's down

Posted: Mon Apr 24, 2017 6:12 am
by Kilaana
Update #1 - From server host: "Unfortunately it appears there was a further problem which caused further degrading of the array on this node. We now need to fully reinstall this node and rebuild the array entirely from a new state."

Update #2 - From server host: "Technicians will be reinstalling the affected box - we expect this process to be complete at 4PM GMT. We sincerely apologize for the inconvenience this may cause. If you are on this box, please contact support after 4PM GMT and we will assist you with re-deploying a new instance and crediting your account. "

Re: Server's down

Posted: Mon Apr 24, 2017 7:42 am
by Khaela Mensha Khaine
Thanks for keeping everyone updated and keeping an eye on things, Kilaana.

Re: Server's down

Posted: Mon Apr 24, 2017 2:01 pm
by Poisonous
Thank you much kilaana!

Re: Server's down

Posted: Mon Apr 24, 2017 2:56 pm
by Kilaana
Thank you, Khaine & Poisonous.

Update #3 - From server host @ 2:05pm GMT: "We are currently attempting a final RAID rebuild to recover data which is currently at 5% and has just under 2 hours left to complete."

Update #4 -Ticket submitted @ 4:15pm GMT to re-deploy the server at 'draugr'. Pending response.

Re: Server's down

Posted: Mon Apr 24, 2017 4:33 pm
by Kilaana
Update #5 - From server host @ 5:16pm GMT: "The rebuild failed again so we are now forced to reinstall the node."

Re: Server's down

Posted: Mon Apr 24, 2017 7:47 pm
by Kilaana
Update #6 - The reinstall on the array node seems to be complete, but our server needs to be rebuilt which means copying all files and installing NWNX all over again. Our last backup was dated 22 April, so we will restart the server from there.

The entire process is rather tedious and could take anywhere from a couple hours to 24.

Re: Server's down

Posted: Mon Apr 24, 2017 7:52 pm
by Copper Dragon
Thank you so much for the updates, and bless you that you are up at this unseemly hour.

The server will be up when it will be; we will wait. If there is something we can do on our end - alas, I'm afraid to think it won't be much - we'll leap at the chance.

~ Copper

Re: Server's down

Posted: Mon Apr 24, 2017 9:40 pm
by Ostheim
Bless your heart, Kilaana.