Chekhov's Gun Saved Me.

Another week later into fall, and life goes on and on.

I got great gifts from everyone on my birthday- and all have been greatly appreciated. My wife bought me a pair of roller blades, though- and I haven't gone a bladin' since probably ought nine! In any case, it's been a good moment since I last had, and I was still a youth. I got to dawn my new stylin' wheels and go out on the town for a quick lap around the park in the dark. I was pleasantly surprised to find I still had more skill in me than I thought! In any event, I only fell once, and it wasn't bad at all. Thanks, love!

Probably the coolest achievement I had this week came from a strange series of solutions to a mysterious problem. The scenario: at my office, we have a physical host that is responsible just for our in-office lab. Although it is mostly of little consequence, it does host the servers that host our guest network, and it also hosts a program that monitors uptime for most of our clients. The problem: for about two or so weeks, we noticed notification after notification that the servers were going and coming offline. And indeed, it was made manifest by the fact that, on occasion at the office, the lab and guest networks would suddenly cease working for a few minutes. I finally had the time to look at it this last week, and that's where it got fun.

I decided to look at the virtual servers on the host, first. Finding nothing, I gave up the chase for a moment. Then, when I noticed the network go offline, I sprang to and, in hasty diagnosing, noticed I couldn't ping the firewall on that network, and assumed it was the firewall appliance. I had even started building a new one, before the problem repeated itself, and in a new train of thought, I decided to look on the host's logs. And indeed, I found a very important clue in doing so- it seems that the host was rebooting itself- under mysterious circumstances.

Talking with my manager, we figured we'd leave it be for a bit, and see if the same symptoms reappeared consistently. I had intended to do so, but then something else funny happened- Chekhov's Gun style. I remembered we had a TrueNAS connected to the host via iSCSI, and knowing this particular brand of hardware has had issues in the past with iSCSI, and seeing that the datastore said degraded, AND seeing as how we weren't even using it, I decided to decommission the old thing. That was that, until I noticed that the host (running VMware ESXi) said it was running a Dell-customized image- on a non-Dell piece of hardware. I figured there would be no harm in sidegrading the host to the correct image.

And so I went to do exactly that. I went physically to the host and ran an "update" to change it back to a vanilla image. Funny enough, I never did get to accomplish that goal, because in the process of trying to do so, the host suddenly blipped off and back on. And then it did it again. And again. And it wouldn't even boot before doing it again. It was at this point I knew I had found the source of the issue- a bad power supply! Oh, but where would I find a replacement power supply for this host? ...[The camera pans stage right to the now-decommissioned NAS from earlier, sitting ready on a table nearby] ...That'll do it! And yeah, despite the two hosts being 7 years difference in age, the power supply inside was the same, and it worked!

It was a very accomplishing feeling to know that by accident, I had accidentally planned ahead like that. It's a rather interesting dynamic, because all too often, unfortunately, I find myself on cliff's edge, wanting to ride right on the edge of updating things, and keeping things clean, organized and simplified. In the past, I've learned the hard way from it, by deleting something I thought I would never need again, only to find I needed it 30 minutes later. Or there was the time when I noticed an expired certificate on an exchange server and did the dumbest thing possible and deleted it. That definitely didn't bite me right in the butt immediately afterwards. At all.

Sometimes it's nice to reflect on moments like these, where I get to look back and see just how many times I've gotten wrong, wronger and downright stupid- and through gradual learning in ways ranging from quick and easy to painful and disastrous (seriously, I'm never living that cert thing down), I've finally reached a point in life where I'm not only learning more and getting better, but also learning how to make better mistakes, and think ahead more clearly. In short, I still fail some of the time, but I have found ways to do it better!

I don't think I ever would've pictured a day, as a child, when my grown-up self would've been happy for his failures. Maybe that's what was holding me back at the time from being a happier person- being happy that I am learning to accept failure, and improve on it, and accept shortcomings. BOY do I have a way to go, though... I really have a hard time accepting a lot of shortcomings. One day at a time, right?

Until then, I hope you have the opportunity to do the same for yourself. Until next time, see ya!

Chekhov's Gun Saved Me.

Adminisk8or

11/13/2023