Some of you have requested why we have had so many issues with some servers and why the reasons of those reboots that we had to do this late November.
It has been found, with the help of vendor engineers and different monitoring devices, that the root cause of the problem was the local power company, TECO. The problem is that we are not being supplied with the 480V of power from our two feeds. Instead, we are being provided a fluctuating source of power, but to the degree of over 60V in variance. This has caused our UPS systems to kick on continually to cover what it sees as a brown out. The bad quality of the power supplied has end up burning up batteries in several strings that are tied to both primary and redundant UPS systems. We have replaced the bad batteries, but each replacement was one of those reboots that you did notice. A pitty that Saturn server as as well affected, because Saturn was not able to repeteat the 412 days uptime, our uptime record for one server box, and the counter had to start over again.
Now, we are fully aware of the actual cause for our continued plight. We have also engaged TECO to have them work to rectify their issues so that this will not be a recurring issue. Power from TECO is now more stable and they are working to solve it for good and forever.
Please direct all additional questions to our helpdesk and we will answer them to the best we can. We thank you for your patience and understanding while we work to get this resolved.
Monday, December 29, 2008