no work related but am overfill sink with water changer because forgot to remove drain cover
is flood
am get in trouble also ._.
Strap in friends, because this one is a wild ride.
I had stepped into the role of team lead of our IS dept with zero training on our HP mainframe system (early 90s).
The previous team lead wasn’t very well liked and was basically punted out unceremoniously.
While I was still getting up to speed, we had an upgrade on the schedule to have three new hard drives added to the system.
These were SCSI drives back then and required a bunch of pre-wiring and configuration before they could be used. Our contact engineer came out the day before installation to do all that work in preparation of coming back the next morning to get the drives online and integrated into the system.
Back at that time, drives came installed on little metal sleds that fit into the bays.
The CE came back the next day, shut down the system, did the final installations and powered back up. … Nothing.
Two of the drives would mount but one wouldn’t. Did some checking on wiring and tried again. Still nothing. Pull the drive sleds out and just reseat them in different positions on the bus. Now the one drive that originally didn’t mount did and the other two didn’t. What the hell… Check the configs again, reboot again and, success. Everything finally came up as planned.
We had configured the new drives to be a part of the main system volume, so data began migrating to the new devices right away. Because there was so much trouble getting things working, the CE hung around just to make sure everything stayed up and running.
About an hour later, the system came crashing down hard. The CE says, “Do you smell something burning?” Never a good phrase.
We pull the new drives out and then completely apart. One drive, the first one that wouldn’t mount, had been installed on the sled a bit too low. Low enough for metal to metal contact, which shorted out the SCSI bus, bringing the system to its knees.
Fixed that little problem, plug everything back in and … nothing. The drives all mounted fine, but access to the data was completely fucked,
Whatever… Just scratch the drives and reload from backup, you say.
That would work…if there were backups. Come to find out that the previous lead hadn’t been making backups in about six months and no one knew. I was still so green at the time that I wasn’t even aware how backups on this machine worked, let alone make any.
So we have no working system, no good data and no backups. Time to hop a train to Mexico.
We take the three new drives out of the system and reboot, crossing all fingers that we might get lucky. The OS actually booted, but that was it. The data was hopelessly gone.
The CE then started working the phone, calling every next-level support contact he had. After a few hours of pulling drives, changing settings, whimpering, plugging in drives, asking various deities for favors, we couldn’t do any more.
The final possibility was to plug everything back in and let the support team dial in via the emergency 2400 baud support modem.
For the next 18 hours or so, HP support engineers used debug tools to access the data on the new drives and basically recreate it on the original drives.
Once they finished, they asked to make a set of backup tapes. This backup took about 12 hours to run. (Three times longer than normal as I found out later.)
Then we had to scratch the drives and do a reload. This was almost the scariest part because up until that time, there was still blind hope. Wiping the drives meant that we were about to lose everything.
We scratched the drives, reloaded from the backup and then rebooted.
Success! Absolute fucking success. The engineers had restored the data perfectly. We could even find the record that happened to be in mid-write when the system went down. Tears were shed and backs were slapped. We then declared the entire HP support team to be literal gods.
40+ hours were spent in total fixing this problem and much beer was consumed afterwards.
I spent another five years in that position and we never had another serious incident. And you can be damn sure we had a rock solid backup rotation.
(Well, there actually was another problem involving a nightly backup and an inconveniently placed, and accidentally pressed, E-stop button, but that story isn’t nearly as exciting.)
Imagine the difference trying to get that kind of support these days. Especially from HP
I have a small PC I use for exposing a private PC to the wider web via nginx proxy. It had two accounts on it: mine, and one I called “remote” with some basic password I set up to forward the proxy connection.
One day, this machine started making 100% CPU noises, for several hours. Wtf? I check the processes and a Tor node had been setup and was transmitting gigabytes to some Russian IP.
My brain goes into panic mode, I kill the process, wipe the remote user, and eventually pull the Ethernet plug.
I wish I hadn’t wiped the user directory as I wanted to know what was being sent and where. Nonetheless the logs showed that several Russian IPs had been attempting an SSH brute force for literally months and one finally guessed “remote” and weak password I set for it.
I have decades of experience on Unix system, and I cringe having made such a rookie mistake.
Lesson learned: change the default SSH port to a transient port, have one dedicated SSH user with a non-standard username, and use auth-key entry only.
I still wonder what was being sent over that Tor node, and why it required all the CPU cores. My best guess is crypto mining, or it was used for a DDOS attack net somewhere.
Obfuscation is not security, changing the port doesn’t increase your security
I see this claim all the time, and it bugs me every time. Obfuscation is a perfectly reasonable part of a defense in depth solution. That’s why you configure your error messages on production systems to give very generic error messages instead of the dev-centric messages with stack traces on lower environments, for example.
The problem comes when obscurity is your only defense. It’s not a full remediation on its own, but it has a part in defense in depth.
Changing the port isn’t really much obfuscation though. It doesn’t take long to scan all ports for the entire IPv4 range (see masscan)
I hear you, but I disagree:
It buys you enough time to check the journals and see that a group of IPs have attempted various ports giving you enough time to block the IP altogether.
It also buys you disinterest from the malicious host, since probably there’s a hard limit on how many ports they will test, and they will flag your machine as “too much work” and try another.
Again, I agree with you that obfuscation is not security, but it sure does help.
From what I understand you obfuscate the port in order to limit the amount of incoming attacks. But then fail2ban would be a much more effective tool.
The disinterested aspect you described is the actual problem. Because it’s based on the assumption your port won’t be found, but it definitely will, and as soon as that happens you’ll end up in a database such as shodan and the entire effect is GONE.
I think they were either computing crypto-hashes and passing on the results back home (via Tor), or they were using my machine to send out several ping/fetch requests over Tor to DDOS some unknown target machine.
So can this pretty much always be shut down by having sufficiently complex + long pw?
I once pushed a git commit with youtube link as the commit message. Nothing terrible, some completely random video. Still, it looked really weird in the commit history. Turns out you can edit this if you have access to the server and I did have access to the server.
One time in the same company I found a random youtube link in the middle of a java class. Yes, it was still compiling. No I didn’t commit it.
Urgh. I sadly do this all the time
Interactive rebase, amend the commit message for your commit, continue the rebase, and force push.
Thank heavens for Magit which simplifies this process.
