Hi, recently (ironically, right after sharing some of my posts here on Lemmy) I had a higher (than usual, not high in general) number of “attacks” to my website (I am talking about dumb bots, vulnerability scanners and similar stuff). While all of these are not really critical for my site (which is static and minimal), I decided to take some time and implement some generic measures using (mostly) Crowdsec (fail2ban alternative?) and I made a post about that to help someone who might be in a similar situation.
The whole thing is basic, in the sense that is just a way to reduce noise and filter out the simplest attacks, which is what I argue most of people hosting websites should be mostly concerned with.
Unrelated to your actual post (plan to read later), but is your RSS busted? The rss link on the webpage gives a 404 and my RSS reader is erroring on it as well…
Oh, it looks like! Something went wrong with Zola build and I must have not noticed. Thanks a lot for pinging me about that, I will fix it today!
EDIT: Fixed! That’s what you get when you forget to bump the Docker image version after you upgrade zola version locally with a breaking change in the config! :) Thanks for letting me know, it would have taken me a long time to see it was broken!
Can you use CrowdSec to track logs from a k8s pod? Say I have my website and some other services hosted on a k3s cluster, do I need to spin up a new pod for CrowdSec or should it be installed on the host?
Hey, the short answer is yes, you can.
I would elaborate a little more:
- First, you have the problem of sourcing the data. In essence, Crowdsec won’t be able to go and fetch those logs for you dynamically, but can go and take those logs from a file (you can do a dirty solution like a sidecar deployment) or from a stream. You can deploy crowdsec in multiple modes, and you can have many instances that talk to each other. You can also simply have some process tailing the pod logs and sending them to a file crowdsec has access to or serving them as a stream (see https://doc.crowdsec.net/docs/data_sources/intro).
- The above means that it doesn’t really matter whether you run Crowdsec inside your cluster (it does have a Helm chart) or on the host. Ultimately all it matters is that crowdsec has access to your pods logs (for example, the logs of your ingress controller).
- The next piece is the remediation component. What do you want crowdsec to do, once it is able to detect bad IPs? If you want to just add IPs to the firewall, then it might make more sense running it on the host(s) you use in ingress, if you want to add the IPs to network policies you can do it, but you need to develop your own remediation components. I am planning to write a remediation component that will add the IPs to Hetzner firewall, some other systems are already supported, but this would be a way to basically block the IPs outside your cluster. For nginx ingress controller there is already a pre-made remediation component .
In practice I personally would choose a simple setup where the interesting logs are just forwarded (in Syslog format for example) to a single crowdsec instance. If you have ingress from a single node, I’d go for running it on the host and banning via firewall, if you have multiple ingress nodes, then I would run it inside the cluster and ban via a loadBalancer/cloud firewall/whatever you have in front.
In essence, I would spend some time to think about your preferences, and it might take a little bit to make the setup clean, but I think you have plenty of flexibility to do what you prefer. Let me know if you want to bounce some more ideas!
Some people also swear by other measures, like changing the SSH port to something else. Most people end up using 2222 to easily remember. This is borderline useless, as you can see for yourself.
While being useless against a sophisticated attacker, there hasn’t been any bot activity in my sshd logs since changing my ssh port to a different one.
Yeah, what I mean is that it’s useless using ports like 2222, that’s like the unofficial SSH port! Bots are generally harmless (once you move to key auth), and you get functional the same result with the automatic IP ban on failed auth, minus the bother to change client configurations to your custom port. Anyway, if someone does want cleaner logs, changing port works :)
I recall hearing that openssh has something like fail2ban built-in now. I forget the name of the feature.
AFAIK I know that SSH has MaxAuthTries and LoginGraceTime, but all it does is terminating the SSH session (I.e. slow down at most), it won’t block the IP via firewall or configuration.
Not sure if there is a recent feature that does the same.
It’s this (excuse formatting): https://www.openssh.com/releasenotes.html
sshd(8) will now penalise client addresses that, for various reasons, do not successfully complete authentication. This feature is controlled by a new sshd_config(5) PerSourcePenalties option and is on by default.
sshd(8) will now identify situations where the session did not authenticate as expected. These conditions include when the client repeatedly attempted authentication unsucessfully (possibly indicating an attack against one or more accounts, e.g. password guessing), or when client behaviour caused sshd to crash (possibly indicating attempts to exploit bugs in sshd).
When such a condition is observed, sshd will record a penalty of some duration (e.g. 30 seconds) against the client’s address. If this time is above a minimum configurable threshold, then all connections from the client address will be refused (along with any others in the same PerSourceNetBlockSize CIDR range) until the penalty expire.
Repeated offenses by the same client address will accrue greater penalties, up to a configurable maximum. Address ranges may be fully exempted from penalties, e.g. to guarantee access from a set of trusted management addresses, using the new sshd_config(5) PerSourcePenaltyExemptList option.
Fail2ban + key-based SSH + self-hosted WAF if you can spin up another machine == 80% of your Web hosting problems gone
