We were just informed that the data center where our DNS server dns4.totalchoicehosting.com / dns4.snhdns.com is located is going to discontinue the facility. Since we are being forced to move our server, we are taking this opportunity to move the server into our own Data Center.
This migration is scheduled for May 25, 2013 after 11 pm.
If your domain is pointed to us using the above DNS Host Name you will need to take NO action. Your domain will continue to work after our migration to our new server. If you are using the DNS Server IP Address to point your domain name to us then you will be required to update the IP at your DNS registrar.
The new IP for this nameserver will be 18.104.22.168.
We will be updating the forum thread located here as this process gets going. As always, our techs are available to attend any queries you may have.
On March 7, 2013 at approximately 3:20PM our network monitoring detected a large amount of inbound traffic to several segments of our public facing network. This large inbound traffic tripped our alerting system and we started to investigate.
Within just a few moments we watched a large Distributed Denial of Service attack unfold upon our network. The attack was a targeted UDP attack at our edge. Typically we can quickly identify and null route these types of attacks, however this attack was very large and coming from thousands of sources.
The above network graph represents just one port on a switch that was targeted. As you can see the 100mbps port was 100% saturated. You can also see when the attack was stopped by our network admin team.
During the attack, I grabbed the above graph from one of our gigE switch fiber feeds, as I wanted to show the family just how large of an attack this was. It was a very short sampling but shows how much inbound we were taking. At one point the attack was pushing over 1.5Gbps of traffic into our network. This combined with such a large amount of packets per second simply overwhelmed our routers.
The actions we were taking against the DDOS starting showing progress at around 4:15:20PM EST and complete network stability was returned at 5:39:10 EST.
While we never like to endure such events, every provider on the internet faces these issues and it is just part of doing business online.
I am sorry for the network instability from yesterday.
Thank you for your business.
The above photo was taken last night. I don’t know how everyone else feels but seeing that stack of Cisco Switches gave me goose bumps, so I was inclined to include the photo with this blog post.
Over the next few weeks, TotalChoice will begin a mass upgrade to much of its network hardware. We have been busy negotiating with our vendors and have taken delivery of all the gear needed to make the upgrades that our customers deserve!
We will start by replacing every Rack Distribution Switch with new Cisco Switches. These switches are a large upgrade from our current switches and will give us a nice boost in performance and should provide years of reliable service. A rack by rack announcement will be made via our Social Media outlets and via email to affected clients. We anticipate very little downtime per rack as we work our way thru the switch replacements.
In addition , we are adding another layer of redundancy to our network. We are in the process of adding additional up-links to each distribution switch from a redundant secondary gigE line card. This means we will be able to sustain a line card outage on our router without any downtime to the switches. Over the past few days we have been installing the secondary up-links from the routers to the distribution switches. So as we put up new the switches we will bundle the secondary redundant link to the new switch.
Last week we installed an additional fiber drop from our upstream providers. This installation is part of our bigger plan for increased fiber redundancy and an improved automatic fail over system. Automatic fail over occurs when loss of connectivity occurs from our main fiber connection. We had a brief network outage a few weeks ago and when the system automatically failed over to our redundant fiber path the fail over simply did not work. After some investigation this failure was determined to be caused by a bad optical device at our upstream provider. The optical device has since been replaced and lessons learned. However, by adding this third fiber it gives us another path to our upstream and gives us more redundancy in our network and that means better uptime for everyone. We have also set up a monthly test that will be conducted to test our redundant/backup fiber connections.
I will be posting more about these upgrades as we progress along. I know this was a long, technically winded post. If you got this far in reading it, thank you.
In a never ending quest to improve the reliability and quality of services that we provide to our clients, TotalChoice has expanded its number of available Internal Backup Servers.
Over the past few weeks we have been tracking a slight degradation in the level of services provided to our clients from R1 Backup Software. This was tracked down to the current cluster of backup servers simply becoming to busy with traffic. It was decided to add another 64GB RAM, 50TB of RAID drive space and 16 CPU’s to the cluster.
The parts were ordered and the additional capacity was brought online over this past weekend.
Services were never interrupted, the TCH team did a really nice job. Specifically have to thank TCH-Bala for his endless push for new backup resources and all the hard work he did once he was given the resources he asked for.
Today the American Registry for Internet Numbers (ARIN) granted a request from TotalChoice Hosting for additional IPv4 address space. This new pool of IP’s will be made available to our clients within the next 72 hours, as we need to get the block advertised to the internet and routed within our network.
With the pool of available IP addresses dwindling at a very fast rate, it has become increasingly difficult for providers to obtain new allocations directly from ARIN. As of this posting there are 2.69 /8’s in aggregate available for allocation from ARIN.
We are very pleased to have accomplished this.