FDL down !….Now back up
I have been involved with IT since the early 1970s. From a DEC PDP8i running a memory resident language called FOCAL with no disk, just 2 ASR 33 Teletype terminals. Through various PDP 11s, a XEROX Sigma 7, and IBM 1130 and Harris and a couple of IBM Mainframes. Up through SUN SPARC stations and i386 boxes and servers running various flavors of Unix™.
In the old days recovering from a disk crash or other system failures was not nearly as evolved as today. But then neither were the systems. Replace the SYSRES Volume (OS system residence pack) with a back up and you were nearly there.
All this has changed extensively. Oh we have RAID for disk back up and extension. And numerous automated back systems but all of this is no better than the weakest link. The use of Virtual Machine technology has also made recovery from most glitches a lot easier.
But a major failure of the underlying hardware for the server(s) is at best a time consuming task to recover from. In nearly all cases the underlying OS has to be reinstalled in order for it to configure itself for the current hardware. And this takes time no matter how it’s done. Whether off of a DVD or network or a combination. After the OS has been installed then it needs to have any updates applied and there can be a slue of these as well. When I update my Linux box ti generally requires around 100 updates as well.
Then there is the manual configuration for the applications and the applications them selves need to be installed and configured.
All of this takes time. I have spent around 2 days on each sever I have had to set up. About average.
Nearly all web sites and most big companies use off site sever companies for their IT needs these days. Just like FDL does here. If they have the money, they will have what is know as a Hot Site. A place at a different physical location that has a duplicate of their main computer/server systems. But these are very expensive to set up and maintain as the Hot Site to be of any value must be in sync with the main site at all times. Not an easy task. And if the communication lines between the two go down before they can activate the Hot Site, things can get very complicated.
Since nearly all of our communications – cell phones, email…etc. – financial transactions and records are kept these days on various servers at various locations, you can imagine the havoc that would result if even a few of these were to be heavily damaged.
It took the systems people at the server company that FDL uses a little over a day to fix the server(s) FDL uses. If the whole or even part of the building were to be damaged, it would take far, far longer. And if it also involved a fiber cut, longer still.
Making us virtual sitting ducks.