Why you should keep your tech up to date.

“An ounce of prevention is worth a pound of cure.”

- A lot of smart people.

Dylan and I (Travis) used to work at the same Managed Services Provider, an outsourced IT department for dozens of businesses. One day (in ~2012), I arrive at the office to a commotion - one of our clients' main server was down. This server held their primary line of business application, a CRM that the whole client ran on. This was an old server (Windows NT End of Life in 2004), running old software, and they so far had refused to upgrade due to cost. So we did the best we could to keep it alive, and alive it was - until that day.

As it turned out - the server had completely crashed, and was not powering on. Something we had warned them would happen sooner or later. This server held all their data and was critical to their business function - without it, there just simply wasn’t much of a business.

Our team had valiantly tried to get it online but to no avail - no amount of tech wizardry was going to breathe life into this remotely. We suspected the problem was with the motherboard itself, and since this server was so old,it was long out of support. The only real chance it had was to replace the motherboard - and as luck would have it, we found someone with a used computer parts store that had one and only one. But this was a long shot - and the odds were against us, but someone had to go try to swap the motherboard. That someone was me.

“I don’t expect this to work, but we need to try anyway, this is the best chance to try to get that thing online. So here’s the new motherboard, good luck.” Paraphrasing what Dylan said to me, he handed me the “new” motherboard we had acquired from a used parts store, something that somebody probably had in a dusty closet somewhere.

I’ll be frank - I’d never performed this kind of tech surgery before. Swapped out lots of parts - hard drives, batteries, etc. - but not an entire faulty server motherboard before, especially one that finished its prime before I left high school. Not only was I not confident in the replacement, but I also wasn’t really that confident that I’d get humpty dumpty back together again in the same way that I’d have to take it apart. And, further, the rest of the components of this server were old and dusty, too, who was to say even if we got it booted, if there wasn’t damage from the motherboard going, or if there would be some incurred in the attempt to swap it. Needless to say, the odds were seriously against us.

Nothing ventured, nothing gained, right? We were going to try anyway. So there I went, motherboard in hand, and headed over to the client’s office. It gets even more fun, because not only was this server old and dusty, but its primary seating space was on top of the client’s office refrigerator (don’t do this). So, I climb up to get that, pull off this giant box and set it on their breakroom-table-turned-operating-table.

I’ll spare you the technical details, but needless to say this is much like swapping a car’s engine - a lot of stuff has to come out and a lot of stuff has to go back in the same way. Hours went by, sweat going down my brow (but not on the server, we don’t want moisture around electronics!) - thermal paste applied, static wristband worn, screws coming out, and then going back in. The moment of truth arrives: pressing that little power button.

Taking a deep breath, and closing my eyes while sweat poured down my face (okay, maybe not) - I pressed the power button, expecting to hear nothing. Instead, I hear the spinning of a fan, the familiar sounds of electricity reaching capacitors and a server coming to life. “Holy shit it turned on.” I said out loud, and then looked to see if anyone in the office heard my surprise (they didn’t).

Immediately, I called Dylan and asked him if he saw it come online through our remote monitoring system. It had. The server came back to life, and not only was it online, it was fully functional. Data was accessible, services were running, and software that the business needed to function was operational. It felt like we pulled off a victory at the last second of the fourth quarter from across the field.

I took the old motherboard and headed back to the office, and was met with congratulations and surprise - all of our hard work to get that thing back online paid off, and much celebration was had from us engineers and our client. We had just saved them (and ourselves) a lot of time and headache trying to get data off of a dead server, and trying to get a solution in place to get them functional while we implemented a new system. I went home that day with a real sense of satisfaction, and probably the greatest victory I’ve ever had in tech - I still have the old motherboard as a trophy. We won big that day - but what if we hadn’t?

We tell this story not to celebrate the incredible skill Dylan, myself, and our team had that day (though, seriously) - but to tell you the other side of this story - the more likely scenario that would’ve happened had we not pulled off a tech miracle: this company could’ve lost their entire business and had to shut down.

I know that sounds heavy and, well frankly, it is. So it should be said again for emphasis - this single point of failure could’ve cost this entire company their business. Let’s get into the why by taking a step back to the beginning - imagining that you’re them and your server just crashed and it won’t turn on, and there is no techno-magic to breathe it back to life, nor techno-wizards to work their magic.

First of all - their entire business ran from this server. All of their client information, contact info, projects, notes, billing, invoicing, contracts, sitting on the fridge. Imagine everything you have right now: your email, your social media tools, all your accounting software, everything you use today to run your business, all on one device, and suddenly that device doesn’t turn on.

Sure, you have backups of your data, and you still have your local workstations. But what do you restore that data to? There isn’t anything. What do those workstations connect and “phone home” to? Nothing, because there isn’t a device that serves as the central hub for all those applications to function. It’s like a baseball bat without the ball and the field - it’s basically just a stick now. Those workstations can’t do a lot because the main thing facilitating their function in a networked environment (which is a whole thing in itself to explain) no longer exists.

Okay, so now what? Just get a new server, right? Not that simple. You were already operating on a shoe-string budget when it came to technology, and thus weren’t updating things and spending money on the proper hardware redundancy to prevent this calamity. Hardware is expensive, and you might not be at a point where you can afford it, much less get it (especially now, in 2022, hardware is really hard to get quickly.) And at the time, “the cloud” was still in its infancy and you couldn’t just spin up a virtual machine and get that started that quickly either. And remember - you’re a business and have services to provide for many clients, so the phones are ringing to get things done (or are they? Did your server control your phones, too?).

So it’s entirely possible you can’t get a server quickly, so now what? Things might grind to a halt, right then and there. You might have to lay people off just to afford the hardware. But can you get it? Maybe not. Windows NT was long out of support and couldn’t even run on modern hardware and do you really want to go back to where you were, right back to what caused this problem in the first place?

But let’s say you find the money in the budget and can buy some servers - well, you want to do this properly. Multiple servers for each primary thing - controlling your network, your email, your main applications, etc. This might mean at least 3 servers. Cool - you’ve got some servers, somehow. You might have waited a few weeks or months, but you’ve got them. Restore the data and away you go, right? Maybe not that simple.

Let’s say you have your backups, are they good? Have you ever tested them prior to this all happening? Further, just moving your entire configuration that you had from your old server to the new ones isn’t just like putting the same hat on a different person - this is an entire operation to attempt to restore functionality. We’re talking hours to days of work here. And if you’re doing it right, you’re building something entirely new that functions far better. But for arguments sake, let’s say your backups are good (cause if they aren’t, it might be game over right here).

You can’t just restore the configurations and assume things will function and you can resume your business - that’s just not how hardware and software works. You’d have to first install new operating systems, get the servers online, and configured for their purpose. Then you get to your main applications - do you have licenses for those applications? Were they all on the old server? Did they get backed up? Uh oh, you don’t. That means purchasing new licenses, and knowing what you had to begin with. Another place that you can grind to a halt.

Phew, you did find your licenses and got software installed and activated. Is the data going to migrate back into them (assuming this is something you even can or should do)? Not that simple - new versions and new hardware might not support it. Sure, some vendors have conversion tools, but a lot don’t. And things can get lost very easily during a backup and restore - or worse, get corrupted. Emails won’t come back, documents go missing, and data from spreadsheets or other items won’t convert from old to new versions so you’re left with potential mess. All this still while trying to serve your customers.

But again, let’s say it all works and you have a properly configured system, and your data is relatively back. Are you still even in business right now? This took days if not weeks, maybe even months to properly come back from. This isn’t even including all the other types of mess this can cause, such as insurance or legal - can you imagine the pitfalls if you lost a bunch of your customer’s data? Depending on your business, this itself could lead to ruin.

The point of all this is - though this is a bit of a doomsday scenario, this is almost (barring extreme circumstances) entirely preventable - so fear not! Just keeping up to date with your technology, prioritizing hardware refreshes, confirming the ready state of your backups, security setup and awareness training, and ensuring that there is redundancy wherever you can have it will be the prevention that prevents the need of the cure. Because the cure might not always be available.

If this struck a chord with you, please reach out, we’d love to help.

-Dylan and Travis