Jordan_U
More than a decade ago a user came into #ubuntu-server on Freenode (now libera.chat ) and said that they had accidentally run “rm -rf /* something*” in a root shell.
Note the errant space that made that a fatal mistake. I don’t remember how far it actually got in deleting files, but all of /bin/ /sbin/ and /usr/ were gone.
He had 1 active ssh connection, and couldn’t start another one.
It was a server that was “in production”, was thousands of miles away from him, and which had no possibility for IPMI / remote hands.
Everyone (but me) in the channel said that he was just SoL and should just give up.
I stayed up most of the night helping him. I like challenges and I like helping people.
This was in the sysv-init (maybe upstart) days, and so a decent number of shell scripts were running, and using basic *nix commands.
We recovered the bash binary by running something along the lines of
bash_binary_contents="$( </proc/self/exe)"
printf "%s" > /tmp/bash
(If you can access “lsof” then “sudo lsof | grep deleted” will show you any files that are open, but also “deleted”. You may be surprised at how many there are!)
But bash needed too many shared libraries to make that practical.
Somehow we were able to recover curl and chmod, after which I had him download busybox-static. From there we downloaded an Ubuntu LiveCD iso, loop mounted it, loop mounted the squashfs image inside the iso, and copied all of /bin/ , /sbin/ , /etc , and so on from there onto his root FS.
Then we re-installed missing packages, fixed up /etc/ (a lot of important daemons, including the one that was production critical, kept their configuration files open, and so we were able to use lsof to find the magic symlinks to them in /proc/$pid/fd/ and just cp them back into /etc/.
We were able to restart openssh-server, log in again, and I don’t remember if we were brave enough to test rebooting.
But we fucking did it!
I am certainly getting a lot of details wrong from memory. It’s all somewhere at irclogs.ubuntu.com though. My nick was / is Jordan_U.
I tried to find it once, and failed.
It’s a stretch to say that going to war in the middle east indicated “care” about/for Arab people.
Also, I haven’t checked but I’d bet good money that we’ve gone back on more promises than we’ve actually honored WRT interpreters.
Meaning, to be clear:
We’ve promised a lot of interpreters U.S. visas / citizenship if they helped us in Iraq and Afghanistan, and have probably blocked more from entry to the U.S. than we have allowed.
That is utterly fucked up, and I don’t see why anyone would trust such promises from the U.S. in the future.
It’s at least gotten a bit better.
There was a time when Photoshop and other programs used a copy-protection scheme that overwrote parts of grub, causing the user not to be able to boot Linux or Windows.
They knew about it, and just DGAF. I don’t remember their exact FAQ response, but it was something along the lines of “Photoshop is incompatible with GRUB. Don’t dual boot if you use Photoshop.”
Grub still has code for BIOS based installs that uses reed-solomon error correction at boot time to allow grub to continue to function even if parts of its core.img were clobbered by shitty copy protection schemes for Windows software.
I can’t vouch for this particular playlist / series since I haven’t watched it, but the channel (Crosstalk Solutions) is great, and so I expect that their home networking 101 is as well.
https://youtube.com/playlist?list=PL1fn6oC5ndU9l3eYa7S_s206JUpbIa-8m
I was helping you there and asked you to back up configs and post some information.
Once you’ve done that I think actually getting things back the way they should be will go fine.