The issues with our internal databases have been corrected and we are back online without any issues to report.
We are currently working on a MySQL issue that is effecting all of the TCH sql driven pages. The RAID array that powers our SQL pages has failed and we are in the process of restoring service.
This does not effect any servers across the TCH network. This only effects the main totalchoicehosting.com pages.
This issue cropped up at 2:38PM EST.
We are working as quickly as possible to restore services in full.
Update 3:56PM EST: All databases have been restored from backups. The website and back end services (billing, forums, helpdesk) are all back online.
As some of you know last week TCH was the victim of a ddos attack on two different days. We have learned quite a bit about the attacks and how to fight them in the future. I think our biggest lessons came in the way of customer support during any sort of outage. What I learned from this past attack is that customers start calling us about 10 minutes into the outage. As everyone knows, we simply have had no major outages at TCH in several years. The last network outage that lasted more than 15 minutes was at least 3 years ago.
Enough of my rambling, everyone that hosts at TCH knows about our budget hosting with excellent up time and services. Even with the two outages of last week our network up time is still 99.982%. In fact, TotalChoice is ranked #1 by Hyperspin for hosts with more than 50 servers monitored. Number #1 uptime of all monitored hosting companies. This is quite a feat for a budget hosting company. I am really proud of TCH and its staff.
Whilst our up time is excellent , we have learned something from our short periods of down time. We need to move our network status page and blog to another Data Center. Currently if the TCH Data Center goes off line, so do our status pages. This is going to change. We are in the process of preparing to do just that. I have a friend that owns and operates another data center on the West Coast. He was looking to do the same and wanted to host his status pages off network. So we have struck a deal. A server for server type of barter.
We expect to roll out these changes in the next 48-72 hours. I will be sending out a mass email to our client base informing them of this after we have done a bit of testing on it to make sure everything is working as it should.