We have the last 3 days been tracking a local root vulnerability in the Linux kernel, the core element of all Linux operating systems. This vulnerability is unprecedented in scope, effecting Linux versions going as far back as 8 years which prompted extra consideration in how we handle it.
Here at TCH we operate a network that is dominated by Linux, to say we took this matter very seriously would be an understatement. It was decided after evaluating the threat this vulnerability poses to our network, dedicated servers, and shared/reseller clients, that waiting any longer on an upstream update was not reasonable. Originally there was an estimate of Saturday 1900GMT for upstream updates but this fell through prompting us to take action. In addition to a lack of a reliable upstream update for this issue is the fact that this vulnerability is being actively exploited in the wild with publicly available attack code on many security and underground web sites.
At this moment, we are rolling out to all Linux servers on our network an updated kernel version that will close this vulnerability while maintaining version compatibility with future upstream software updates. This effort in retaining version support will allow our dedicated clients in addition to our own support team to resume normal update practices with tools such as ‘yum’ or ‘apt-get’ and not have to worry about conflicting versions against our in-house kernel update.
Please do not be alarmed if you experience an outage temporarily on dedicated, shared or reseller servers, we thank everyone for understanding the urgency of this matter and if you have any questions or comments please feel free to submit a help desk ticket at https://www.tchhelp.com.
UPDATE:Aug 18, 2009
We will be conducting reboots again this evening to push out a revised version of last nights kernel that corrects issues with r1backup agent, local firewall services and the network driver on certain servers. In addition, this new kernel revision is binary compatible with CentOS/RHEL 4 kernels being that it was built off the same kernel source tree as the standard kernels.