I know, I know, clickbaity title but in a way it did. It also brought in the situation in the first place but I’m just going to deliberately ignore that. Quick recap:
- I came home at 3pm from the city, my internet at home didnt work.
- checked multiple devices, phones worked out of wifi, I figured I need to restart the router
- I login to the router and it responds totally normal but my local network doesnt. (Its always dns, I know)
- I check the router log and see 100s of login attempts over the past couple of days.
- I panic and pull the plug, try to get into my server by installing an old monitor, works, many errors about dns
- Wife googles with her phone, seems I had https login from outside on and someone found the correct port, its disabled now
- Obviously, local network still down, I replug everything and ssh into the server which runs pihole as dns
- pihole wont start dns, whatever I do
- I use history and find I "chmod 700"ed the dns mask directory instead of putting it in a docker volume…
- I check the pihole.log, nothing
- I check the FTL log, there is the issue
- I return it to 777, everything is hunky dory again.
Now I feel very stupid but I found a very dangerous mistake by having my lan fail due to a less dangerous mistake so I’ll take this as a win.
Thanks for reading and have a good day! I hope this helps someone at some day.
Did you expose your router login page to the open internet? How’d they get access? Why are you chmoding anything to be 777?
There was an option that I had enabled years before and forgotten so yes, I didnt know but it was, on some obscure port.
And yes, pihole in docker makes its files be 777 which is pretty disgusting, I know. Thats why I tried to make it 700 and broke my whole network.
Doubt. You probably need to set the file owners in your volume to the same user running in the container.
You can doubt all you want. I changed it from 777 to 700 and back again because it broke. Couldnt find the user in the container immediately. Will probably just migrate it to a volume and be done with it.
I think you are still learning… What you say doesn’t make sense, so I think you may have misunderstood what happened.
Imo we are all constantly learning. Otherwise we stagnate. What I say makes perfect sense, you just dont get it. So let me explain it again, in more detail:
I was going through my docker compose files to sanitize them and upload them to my private forgejo instance.
While doing that I found a directory in my filesystem, a remnant of the early days of my server where my knowledge was severely more limited, that was a docker volume mapped to a regular directory, something I wouldnt do today for something like this.
It was owned by root:root and had 777 permissions which is a bad idea imo. So I changed it to 700 since I dont think I had any other users in group root and others, well.
Nothing bad happened, until today when my unattended backups triggered a restart at noon and the tragedy started. I put it back for now to 777 but I‘ll try and integrate it in a real docker volume which resides in the docker folders.
Wipe and start fresh
Because you don’t have a way to know what’s been compromised. Take your data only and make sure to verify nothings been tampered with.
Trust me it will be better in the long run.
Yeah, I dont feel like setting up a whole cloud infrastructure on a hunch. I‘m running like 15 different services and they are all compartmentalized. It would take weeks to reset all this. So far nobody got anywhere from what I can see.
If you have everything on docker compose migrating to another host is pretty easy. I could probably migrate my 11 stacks of 36 containers in 2 to 3 hrs
If everything works well, I could probably do that too. But I‘ve had too many obscure little things happen that 10x the amount of time needed so I always plan for the worst case.
Also, my point was that people are being massively overreacting due to the fact that my logs showed signs of attacks, not intrusion.
I run many servers and the commercial ones I am much more slow and careful with. Every public facing service has attacks in their logs and I deal with them. I know what experience you guys have but its not hosting public services.
the arrogance with which people suggest someone is incompetent is baffling. Not talking about you but quite a number of comments where condescending af.
Thanks for the advice with ansible. I might actually give this a go.
Why would I start fresh?
If they had access to a machine, the first thing you do is install some kind of root kit so you can get access again later. This could be as small as modifying an existing binary to do things it isn’t supposed to do.
If they didn’t access any machine, your fine.
777? Bruh just set the owner?
I’m referring you to my quick “self-hosting guide”: https://lemmy.world/comment/7126969
One of my fears of starting up my homelab.
All you have to do to avoid this is just not open any ports except one for something like wireguard, and only access your network using it externally, and you will never have this problem.
