Just a brief notice about continued infrastructure upgrades here at TCH:

We will be upgrading our Intrusion Detection System (IDS) hardware on Friday 1/27/2012. This hardware is dedicated to behind the scenes work that reviews all inbound traffic to our entire server farm and helps us sanitize bad or unwanted traffic to our servers. As our network traffic increases, so has the requirements of the IDS system, this is why we are upgrading the hardware. We do not anticipate any network downtime during this upgradation to our IDS hardware. This is simply a notice of our continued improvements to the services we provide to our clients.

In addition, on 1/27/2012 we will be performing upgrades to our private network switching hardware. This will be a multi-step process and will take us several maintenance windows to complete. If your servers are relying on our private network, you should receive an email from us with the maintenance window for your servers. In this process we will be replacing each distribution switch with new units. These new switches will feature the same 1Gbps server to switch speed but will feature a 10Gbps switch to switch speed. We will also be upgrading our backup server’s network interfaces to 10gbps.

There are many more improvements coming, but the above are the issues we have on the planning board for this week.

Today is a great day to be a TCH GURU!

Over the past 6 months TCH has experienced some really amazing growth. In the last two months alone we have shattered month to month sales records! November 2011 marked our best all time month in the company’s history. We ended 2011 with our best December in over ten years. Even more amazing is the fact that we just broke our best 15 day period for new sales records. Thousands of new TCH family members have joined us in the past few months and we are very proud to have them in our family.

We owe this all to our clients and staff. I wanted to personally thank everyone that has made TCH such an amazing company.
With this explosion in growth we are making some changes to our internal infrastructure. All of these changes are happening behind the scenes and should be totally transparent to our clients.

This morning around 6am EST, we brought on a new cluster of servers to handle our support desk. We have never had the need for such dedicated hardware to handle our support system, however with all the growth we started to notice a decline in the performance of our help desk and decided to take action. This new boost in speed also translates into allowing our desk to handle a larger volume of clients help requests per day.

Over the past few days we have also brought on new cluster of DNS servers to handle our new clients Domain Name Resolution. This allows us to not overload our existing DNS Servers and most importantly allows us to continue to grow and not affect the performance of our current hosted domains.

We are also bringing on new servers at an amazing rate. Just this past week we added several more servers to our growing server farm. Unlike other small hosts, we actually own and operate all our own hardware. We have our own IP allocations and maintain our very own Data Center Presence. We do not rent servers from other providers and resell other providers servers. We are truly independent and rely on only ourselves to reboot your servers or upgrade your servers ram. If our tech team sees a server that needs more ram, we just take action and do not have to sit around and wait for our vendor to send a technician to work on our rented server.

We have much more to announce and will be doing so right here in our family forums. Many of you have been asking for a new web site with improved services and we are very close to doing so. Please stay tuned for more announcements and in the meantime enjoy the improved infrastructure here at TCH.

Thank you again for all your support.

It’s a great day to be a TCH GURU!

As many of you are aware, we started enforcing a new policy for password rotation.  In theory this was a splendid idea from both a security and administrative view point.  Over the past 24 hours we have had a wide range of customer comments regarding this new policy.  These have ranged from this is the best new policy ever all the way to TotalChoice has gone to the dogs.

The largest concern about this new policy came from our reseller client base.  This is due to the fact that the new password change can not be effected from WHM.  It can only be effected from cPanel logins.  This is a flaw or bug that we did not realize was present in cPanel until our reseller clients started running into it.  We are working with cPanel in an effort to correct this issue.

As many of you are aware I take customer service very serious and I also do not beat around the bush with my clients.   It appears that this policy has become more of a nuisance for clients and that is something I would never want to promote.  It is with that in mind that I have chosen to revert this change and forgo this new policy.

I would like to thank all the clients that have reached out in both support of and against this issue.

As always thank you all for making TCH what it is today and most importantly….

Happy Hosting!

I would like to thank all the clients that have reached out in both support of and against this issue. Please review the latest update concerning this policy.

Dear TCH Family,

TCH takes security very seriously and we are always looking at ways to improve the security for our internal infrastructure and for our client’s web presence. In an ongoing effort to provide our clients with the most secure environment possible we are enforcing the following new password rotation rules.

Effective immediately, every shared and reseller hosting client upon their next login to cPanel will be prompted to change/update their user passwords.  You will not be able to utilize your old password during this new password update.  Once this is completed, users will then be prompted every 180 days to once again rotate their passwords.

We would suggest you implement at a minimum, an 8 character long password with combinations of upper/lowercase, number and special characters.

We believe this new policy will insure that the hosting environment provided to our clients will become more secure.  This will also promote password security that will hopefully pass down to the way all users handle their passwords not only here at TCH but overall.

As always we appreciate your business and look forward to providing our clients many more years of happy hosting!

Thank you

Bill Kish
TotalChoice Hosting

This does not effect any servers across the TCH network.  This only effects the main totalchoicehosting.com pages.

This issue cropped up at 2:38PM EST.

We are working as quickly as possible to restore services in full.

Update 3:56PM EST: All databases have been restored from backups. The website and back end services (billing, forums, helpdesk) are all back online.

Enough of my rambling, everyone that hosts at TCH knows about our budget hosting with excellent up time and services.  Even with the two outages of last week our network up time is still 99.982%.  In fact, TotalChoice is ranked #1 by Hyperspin for hosts with more than 50 servers monitored.  Number #1 uptime of all monitored hosting companies.  This is quite a feat for a budget hosting company.  I am really proud of TCH and its staff.

Whilst our up time is excellent , we have learned something from our short periods of down time.  We need to move our network status page and blog to another Data Center.  Currently if the TCH Data Center goes off line, so do our status pages.  This is going to change.  We are in the process of preparing to do just that.  I have a friend that owns and operates another data center on the West Coast.  He was looking to do the same and wanted to host his status pages off network.  So we have struck a deal.  A server for server type of barter.

We expect to roll out these changes in the next 48-72 hours.  I will be sending out a mass email to our client base informing them of this after we have done a bit of testing on it to make sure everything is working as it should.

Happy Hosting!



The first thing we need to understand is what is being attacked; as the post subject implies, it is primarily Joomla being attacked as the software has had a series of 9 vulnerabilities released since the 1st of September of which a number of more in depth attacks have formed around. The intended purpose of most of these attacks is to taint web sites with injected javascript, that code takes advantage of a number of client side browser vulnerabilities that if not patched or stopped by an antivirus can cause further issues for web site visitors.

Now, at a glance you might be thinking that if someone fails to patch web site software then it is there own problem, how does this affect me? That is where the dl() function comes into play, the dl() function is essentially a dynamic loader for php modules or 3rd party extensions. To simplify this a bit, the dl() function when enabled allows anyone to add extensible features onto php, generally these are all well to do features but if someone so desires they can create a dynamic loader module with malicious intent.

The scenario we are looking at is that attackers have gained entry to vulnerable web sites, primarily through joomla then they upload a series of malicious scripts including a dynamic loadable module for php that once enabled through dl() has the ability to inject javascript code into pages. The code usually finds itself placed before the body tags and executes its payload on a visitors first visit to a site, a cookie is then set that expires every 2hours then the payload executes itself again on a new visit.

This attack though had far reaching implications, only affected 4 servers on our network (denver, dantooine, alderaan, chewbacca) of which only about half the sites on the given servers or in some cases less were being tainted by the attack. As alarming as this situation is, we need to stress that no content was actually modified on sites except the joomla sites themselves that were compromised.

The way we have come to deal with this situation is a layered approach, we have first and foremost made increased efforts to identify compromised sites on our servers and suspend/remove them. The next step was to cut off the enabling function of the attack, which is the dl() function. This function was actually something we used to disable on servers for its malicious implications but over time that procedure was phased out in the interest of allowing users to install custom dynamic loadable modules from their home directories such as ioncube. However, now that ioncube is standard server-wide on all servers, there is little in the way of other commonly installed packages that depend on dl(), php.net has even went as far as to declare dl() deprecated as of php 5.3.

With dl() disabled on servers, the effects were immediate and all reports of tainted sites stopped, now when I say stopped I do not just mean that that lightly. We literally sat around all evening bashing the f5 key on our keyboards trying to get the javacode injections to reappear on sites, between myself, Bill and Dick we must have done over 6 hours of combined keyboard kungfu in this effort. It was with great relief that we were not seeing anymore reports or issues ourselves first hand but it was still not quite enough to actually be confident that we had done enough.

We are continuing to be extra vigilant with compromise assessment on the servers to prevent any further malicious content from being injected into sites, in addition to this we have on some servers started to use suPHP as a basis for new php security standards. Essentially, by using suPHP we enforce php code to run as the user who executed it instead of as the web server but it goes beyond that by enforcing strict permissions on content and not allowing anything to run above mode 755 (such as world writable data) and also making sure that executed content is owned by the user. This might seem problematic however since the code is now executing as the user, there is no longer a need for data to be set to mode 777 (world writable) or its ownership set as the web server user, which reduces support issues and vastly increases security. The suPHP changes are something we have only rolled out to about 6 servers so far but the support issues it has generated are minimal for the advantages it provides, in the future we will be looking to roll this change out to more servers on a slow but steady basis.

That is where we are at, if you have any questions or concerns regarding this blog or the topics discussed please feel free to comment or head to the TCH forums for further dialog.

Here at TCH we operate a network that is dominated by Linux, to say we took this matter very seriously would be an understatement. It was decided after evaluating the threat this vulnerability poses to our network, dedicated servers, and shared/reseller clients, that waiting any longer on an upstream update was not reasonable. Originally there was an estimate of Saturday 1900GMT for upstream updates but this fell through prompting us to take action. In addition to a lack of a reliable upstream update for this issue is the fact that this vulnerability is being actively exploited in the wild with publicly available attack code on many security and underground web sites.

At this moment, we are rolling out to all Linux servers on our network an updated kernel version that will close this vulnerability while maintaining version compatibility with future upstream software updates. This effort in retaining version support will allow our dedicated clients in addition to our own support team to resume normal update practices with tools such as ‘yum’ or ‘apt-get’ and not have to worry about conflicting versions against our in-house kernel update.

Please do not be alarmed if you experience an outage temporarily on dedicated, shared or reseller servers, we thank everyone for understanding the urgency of this matter and if you have any questions or comments please feel free to submit a help desk ticket at https://www.tchhelp.com.

UPDATE:Aug 18, 2009
We will be conducting reboots again this evening to push out a revised version of last nights kernel that corrects issues with r1backup agent, local firewall services and the network driver on certain servers. In addition, this new kernel revision is binary compatible with CentOS/RHEL 4 kernels being that it was built off the same kernel source tree as the standard kernels.

Update: Authorize.net is now back online. You will be able to make payments. We will update you if this or any other changes arise.