What You Get After Running an SSH Honeypot for 30 Days | Hacker News Reader

What You Get After Running an SSH Honeypot for 30 Days

blog.sofiane.cc

I self-host a (non-critical) mail server and a few other things and occasionally look at live firewall logs, seeing the constant flow of illegitimate traffic hitting random ports all over the place, some hitting legitimate service ports but others just probing basically anything and everything. I decided to setup a series of scripts that detect activity on ports that aren't open (and therefore there's no legitimate reason for the traffic to exist) and block those IP addresses from the service ports since the traffic source isn't to be trusted.

Something that came out of analysis of the blocked IP addresses was that I discovered a few untrustworthy /24 networks belonging to a bunch of "internet security companies" whose core business seems to depend on flooding the entire IPv4 space with daily scans. Blocking these Internet scanner networks significantly reduced the uninvited activity on my open service ports. And by significantly I mean easily over 50% of unwanted traffic is blocked.

Network lists and various scripts to achieve my setup can be found here: https://github.com/UninvitedActivity/UninvitedActivity

Internet Scanner lists are here: https://github.com/UninvitedActivity/UninvitedActivity/tree/...

Large networks that seem responsible for more than their fair share of uninvited activity are listed here: https://github.com/UninvitedActivity/UninvitedActivity/tree/...

I'm semi-aware of the futility of blocking IP addresses and networks. I do believe, however, that it can significantly reduce the load on the next layers of security that require computation for pattern matching etc.

Be aware: there are footguns to be found here.

One thing I do is I blocklist entire countries' and regional ISP' CIDR blocks. Believe it or not: straight to firewall DROP.

China, North Korea, so many african countries who's only traffic is from scammers, tiny islands in the pacific that are used for nothing but scamming...

Straight to DROP.

And I do not care about the whining.

Had a travel insurance do this and when I was in hospital in Asia I couldn't start a claim and the hospital nearly kicked me out. I'm sure the sysadmins thought it was a great way to reduce hacking attempts by blocking Asia.

That's so remarkably stupid for travel insurance, it's unbelievable.

I wrote a cynical take on "how it happened" at the time: https://joshua.hu/losing-sight-vision-mission-of-your-role

I think it comes from the divorce of what people are hired to do versus what their work actually contributes to. I also remember the countless cloudflare turnstiles that I've had to get through one way or another on airlines' websites which reset every minute (looking at you, airserbia, for being the worst).

If there’s one single business that I might expect to honor traffic from foreign countries, it would be the travel industry. I can suddenly envision using a VPN to route through Asia and check a travel agent’s site access before purchasing.

Why couldn't they fix this with a phone call? So much suckage.

Mental note 1: Verify whether I can file a claim over the phone before I give a travel insurance company my money.

Mental note 2: Don't travel without being able to VPN through a U.S. endpoint. Preferably something sitting in my house.

Mental note 3: Verify you can call international numbers?

GP said he does "not care about the whining".

Ironic that GP commenter said "I do not care about the whining" about regional IP blocks and the first reply is just someone whining about it.

That’s awful but why is the onus on random sys admins around the world to deal with this correctly and not the government hosting the problem entities?

I would say because it’s their job to serve their customers, even if they’re abroad? Especially for a travel insurance company.

You don't think a travel agency selling policies covering china should have their sysadmins ensure that their customers can actually make use of those policies? They can always explicitly exclude china if they don't want to deal with this but then they wouldn't have gotten GP's money.

It's not a random sysadmin. It's a sysadmin of a travel insurance company.

if the government in question is supportive of said problem entities, they won't "deal" with it

If the government in question has free reign on regulating said traffic, it's an avenue for repressions and censorship

Otherwise it's a legal matter to seek action against such entities, which is already how it works

(... but I'm afraid we're actually mostly talking about "scenario 1 entities" here, which makes it futile to seek action from the very offices that already play a role in making it harder to use existing legal means)

Government needs lobbying to act

That's like asking why don't we expect burglars to not burgle, they won't, but that doesn't mean walling off a whole neighborhood is the solution either.

As a Russian, I hate it when people do this. It's extremely annoying when you just click some random interesting-looking link from HN or Reddit or Twitter only to be greeted by a 403 or a connection timeout. Then you turn your VPN on, and magically, it loads just fine.

For many services, the expected value of letting people from Russia access their service is negative. The reality is that Russia contributes a large portion of hacking attempts while providing very little to no revenue for the service. At the end of the day it is just business, and sometimes letting countries access your service is bad for the bottom line.

I think you and the person above you can both have valid concerns at the same time. If someone said "~50% of theft is from <insert minority group> while they only account for 5% of my business, so I'm not going to let them in the door", assuming the absence of social and legal consequences which would realistically occur, it could be argued that it's the right move for their "bottom line" or whatever. Does that mean it's right, or good, or equitable?

Of course at the same time, if you hold yourself to a much higher standard than what's socially or legally acceptable, there's the inevitable fact that your competitors aren't. So it's a fine balance.

Your country is a bad global citizen. If they started taking action against the groups trying to break into my systems every minute of every day then I wouldn't need to block the entire jurisdiction.

Geoblocking all sanctioned countries was the best thing I ever did

Your annoyance is a feature, not a bug. You are supposed to get annoyed enough as a group to lobby your government to fight the internal problem

You're very naive to assume that this government takes any feedback.

I'll just leave this thread here: https://twitter.com/IrineKuklina/status/1578339408801304580

Ah, yes, the remaining English speakers in Russia will overthrow the literal millions of the silovik class whose entire job is to repress (with violence) any independent political activity. There is no "lobbying" in Russia, if you didn't know.

If you hate all Russians just say you hate all Russians. No need for this "lobby your government" euphemistic BS.

people here are not thinking in whole systems-- roads have dual purpose.. there is security AND there is trade .. a world without trade is a poor world.. that includes the intellectual arts, civilian institutions cooperating, common issues like Climate.

The voices here that say "I block everyone, don't bother me with your whining" .. it is a security practice.. OK. security is not the whole story of civilizations; obstinate thinking leads to ignorance, not evolution.

The topic is SSH, an administrative and secured access. Yes security applies. to be on-topic

Of course one can obfuscate and secure their own SSH access as much or as little as they want. Run sshd on a different port, require port knocking, ban IPs after failed login attempts, all that kind of stuff.

I'm, however, specifically talking about public-facing services like HTTP(S), which also get blocked with this "I'll just indiscriminately blacklist IPs belonging to countries I don't like" approach.

Had a reddit clone. The amount of Russian spam coming in was nuts.

Blocking the ru language blocked all spam. And since it didn't have Russian users, it was an easy choice to make.

I think it’s harmless though if say it’s a business site or mail site that is only meant to do business with a subset of people, like a country or region. That said, I think it’s of highly limited value though because any hacker above Lvl 1 will know how to use a bot, remote box, or VPN from a more local IP.

> It's extremely annoying

Now imagine how annoying is russian traffic to world's sysadmins. Then could you please point your finger to who's more wrong here: your government or sysadmins of the world?

I assume you don’t host anything that could be useful to the 1.5 to 2 billion people that you’re blocking.

Or they host a business site that doesn't do business in those countries and so nothing of value is lost to them. For example, it's literally illegal for me to accept payments from .ru, so why bother wasting their time and my bandwidth?

I live in EU,and a bunch of american sites just block the whole EU due to GDPR laws.

Then someone in US uses my email by accident to subscribe to some newsletter (not the first time, I also get personal emails for that person, since it's just one letter difference, and i'm guessing it's someone old, considering the emails I get), i try to click "unsubscribe", and it just redirects me to "<site> is unavailable in EU, blah blah" page, without unsubscribing.

I make sure to report that site to every goddamn spam list possible.

Same here. I country-block I think 4 countries and my "not-me" ssh login attempts dropped 90+%. As I run funzies sites, I couldn't care less about the reduced legit traffic.

I'd do this too except by far the most scam traffic I see are US in origin. I'm in the EU.

> so many african countries who's only traffic is from scammers

Which countries specifically? Asking from Africa, and not sure I've encountered this.

Personal page.. sure.

Business? You're a pain to many people and don't care.

I live in EU and many US pages just block the whole EU due to GDPR laws... then someone (by mistake) subscribes me to their newsletter, and the "unsubscribe" links leads to "this page is unavalable in EU"? I'll goddamn make sure your domain ends up on every goddamn possible antispam filter I can find.

That's often worth an FTC complaint for a CAN-SPAM Act violation: https://www.ftc.gov/business-guidance/resources/can-spam-act...

The FTC wouldn't accept "we didn't want to deal with GDPR" as an excuse for a business violating that law.

Why? Are they spam pages?

For me? Sure. I never subscribed to them. Ans the unsubscribe links doesn't work, probably illegal, although not sure if they can spam an EU citizen from usa, and which/whose/what law are they breaking.

> I'll goddamn make sure your domain ends up on every goddamn possible antispam filter I can find.

Honestly, individuals can't really do much to change the reputation of a domain.

Maybe petition your representative to adjust the GDPR so they don't claim it applies globally?

> Honestly, individuals can't really do much to change the reputation of a domain.

Your hosting provider and ISP will see this differently. So will the FTC.

> Maybe petition your representative to adjust the GDPR so they don't claim it applies globally?

Your butthurt about the GDPR doesn't absolve you from your obligations under the CAN SPAM act.

That's very computationally inefficient.

> That's very computationally inefficient.

It's O(1) with iptables/nftables ipsets. Moreover as I blocklist entire CIDR blocks, there aren't that many entries in those ipsets.

You can trivially maintain a list of the size of the whole ipv4 space by using bitmaps

The Biden administration needs to explain why they allow ISPs to import data from these countries.

I'm not sure I understand what you're suggesting. Are you saying that the US govt should make it illegal for people in its borders to communicate with people in those countries?

> and block those IP addresses from the service ports since the traffic source isn't to be trusted

Don't get me wrong, I want to do the same, I run a lot of servers and see all the automated nonsense aimed at public servers. However, you should consider the fact that today blocking an IP is akin to blocking a street, a village or sometimes even a town. For ~better or~ worse we now live in the age of CGNAT.

If your threat model and use case means you only care about a known subset of users with static IPs who are lucky enough to not share IPs then fair enough; but if you are running services intended for wide spread consumption you are likely blocking legitimate users without even knowing it.

I have thought about that and, as you say, my use-case is entirely "hobby" so there's nothing I host that's of much interest to others (if things break, which they have, it inconveniences me rather than other people).

Having said that, the websites I host are behind Cloudflare and so port 443 allows Cloudflare's ASN, but blocks everything else. This way, any of the IP addresses that are blocked from direct access to port 443 can still access the websites, just through Cloudflare's added layer of protection.

Try running some of your blocked ips through greynoise, they usually have some interesting information about them

Thanks for the tip. Looks like greynoise use ipinfo.io for IP metadata.

I use https://www.abuseipdb.com/ for any manual IP address checks, and https://hackertarget.com/as-ip-lookup/ for finding what ASN an IP address (range) is a member of. I'll check out greynoise and see what extra info may be provided.

I (DevRel of IPinfo) run Fail2Ban on a VM as well. Protip use the CLI.

- The CLI has the `grepip` command that extracts all the IP addresses from a text. You do not have to parse your logs.

- Analyze your data. After you have extracted your IP addresses from your logs, pipe them to the `summarize`, `map`, and `bulk` commands on the CLI.

- If you are doing bulk enrichment with the `bulk` command, you can use some kind of CSV query tool like CSVtoolkit, DuckDB, or Python-Pandas.

- Look into the ASN data. ASN data is always going to be the more interesting IP metadata for honeypots IPs. Summarize the IP addresses with the `summarize` command; it will give you a high-level report. If you want a web-shareable report, make a POST call to that endpoint. Docs: https://ipinfo.io/tools/summarize-ips

https://github.com/ipinfo/cli

You can always send your logs to me and ask what I think of them, and if I can find common patterns based on IP metadata. I am running our API and database services 24/7 and enjoy looking at logs. I can suggest firewall configurations based on country and ASN information provided by our free data.

Good idea. What I do is, I disallowed password login in my ssh server, and I permanently ban whichever address that tries to log in using a password.

I use a bastion host on a VPS as the only source IP address allowed to ssh into my systems, so any attempts to connect to ssh (from any IP address other than the bastion) are both blocked and logged into "the list" to be blocked from connecting to any other service ports.

I did this but added an "escape hatch" that allowed password logins from the local network only.

Just be aware that with your strategy “blocking 50% of unwanted traffic” means blocking non-attack traffic, as these Internet security companies are mostly legitimate. The automated attack traffic that you actually want to block is in the other half and will frequently change IPs.

> these Internet security companies are mostly legitimate

This is both subjective and highly dependent upon the scope of services being run. My setup would probably progressively create more hassle than it saves as on a scale from small business to large business. For the setup I have, I quite specifically want to block their traffic.

I'm possibly overly militant about this, but they keep databases of the results of their scans, and their business is selling this information to ... whoever's buying. I don't want my IP addresses, open ports, services or any other details they're able to gather to be in these databases over which I have no control and didn't authorise.

To steal an oft-used analogy, they're taking snapshots of all the houses on all the streets and identifying the doors, windows, gates, and having a peek inside, and recording all the results in a database.

I believe all of them are illegitimate. They 'do' because they can, and it's profitable. "Making the internet safer" is not their raison d'être.

Happy for any else to form their own opinion, but this is my current stance.

Yes - Anyone who's FAQ answer to "How to avoid being scanned" is "We don't have an opt-out, you must block all these addresses" isn't behaving like a legit business.

"Nice network you've got there."

"We noticed something might be open. We're not telling you what it is."

"It would be a pity if something happened to your business."

"Give us lots of money."

Sounds like a movie strong-arm thug.

Would be cool to have a "don't scan me bro" list of IP's that engage in this that we could share - is there such a thing?

The problem is that becomes a concentrator of IPs behind which privacy conscious individuals exist, which probably has higher value to "whoever's buying". It's a conundrum.

There's a comment downthread discussing something similar; I haven't tried it though: https://news.ycombinator.com/item?id=40695179

You're being sarcastic, right? We did this for telephone numbers and saw how it turned out...

> these Internet security companies are mostly legitimate

Act like a bot, get treated like a bot.

> Just be aware that with your strategy “blocking 50% of unwanted traffic” means blocking non-attack traffic

You don't block them forever, just enough for them to move on to someone else.

they dont move on to someone else, they scan entire internet on a regular basis, just like gogle crawls web pages

My experience is that after blocking Censys, unwanted traffic on non-standard ports from other IP blocks has basically gone to zero. It appears to me that some bad actors are using Censys scans for targeting.

i get similar results

> (...) as these Internet security companies are mostly legitimate.

Note that you're basing your assertion on the motivation of random third parties exclusively on the fact that they exist and they are behind active searches for vulnerabilities.

Lol legitimate. As legitimate as door to door salesmen. OP just put up a proverbial "no soliciting" sign.

Have you considered using crowdsec?

I set it up in a fairly superficial way, and there are only a handful (two or three) rules that can be applied on the free tier, and I'm a tight-ass.

It's still running, but it doesn't seem to block much - but that might be because I didn't put enough time into "doing it properly".

Are there any downsides to crowdsec?

You end up sharing signals (IPs) to their crowd-sourced bad IP databases, but only get 3 free IP lists on the free plan. To get some of the bigger IP lists you need an enterprise plan at $2500 a month.

Essentially they use the free customers to build the lists that drive their enterprise sales, which is fair enough as you get to use their free dashboard and open source software. But to me it seems they're really only targeting enterprise customers as a business.

Hi all and @snorremd, (Philippe from the CrowdSec team)

The $2.5K / month was for enterprise, but we didn't correctly understand the need and converted it to 2 optional prices: $1K for LTS and $1K for support. This will be reflected in an update on our pricing page this week; thanks, everyone, for your patience in this matter.

It took us time to segment our four products properly. We wanted to avoid pivoting later, as it happened to so many other open-source tools recently.

* The Security Engine (IDS+WAF+IPS) is for everyone. (Free / MIT license, three free blocklists)

* Its SaaS companion is made for anyone with a security engine. (Generous free tier, $31/engine/month for pro industrialization features, 3 premium blocklists + all free ones. Volume discounts avail. We'll soon merge SecOPS and enterprise plans, all features at the price of the SecOPS plan)

* Blocklists are made for M/L entities to use. (In the range of a few ten of K$ yearly, all blocklists, unlimited)

* The Full CTI database is intended to be used by L/XL Corps. (It contains 32 fields about ~25M IP, with industry targeted, country targeted, tech stack targeted, AS and range reputation, etc. Local replication at your place, several updates/day. 10 to 20K$ / month, depending on some parameters)

PS: As we did for the Olympic Games 2024, we'll also give away a blocklist for the US presidential election of the most aggressive IP against US assets. With a quarter of a million machines running CS, we have a fairly good overview of this, in real-time.

Safer together.

I was about to say out loud that it was a (kind of) relief not finding Google in your lists, then I found https://github.com/UninvitedActivity/UninvitedActivity/blob/...

I need to check my exact configuration, but whilst I've got 1e100 in a list, I think I've got an exception for it elsewhere.

Ie. Whilst it's been detected as uninvited activity, it causes issues when blocked, so it's excluded from the blocking.

Just install fail2ban.

For SSH, changing to a random port number resulted in zero connection attempts from bots for months on end. It seems bots just never bother scanning the full 65535 port range.

For most of my VMs there's no ssh running. I use wireguard to connect to a private IP. I haven't done this on the bare metal yet but I might. Though barring exploits like we had recently nobody is getting into a server with either strong passwords or certificates. Fail2ban in my eyes is a log cleaner. It's not useful for much else.

it bans the bad ips, isn't that worth running?

> the full 65535 port range

Note that putting SSH on a high port has security implications.

What security implications?

A server with fail2ban can be DOSed by sending traffic with spoofed IP addresses, making it unavailable to the spoofed IP addresses (which could be your IP, or the IP of legitimate users).

That is typically a bigger problem than polluting your logs with failed login attempts.

What would spoofing the IP of a packet when the underlying protocol requires a two-way handshake accomplish?

With CGNAT, a prepaid sim card and some effort, you can make them block a whole legit ISP in a few days without spoofing anything.

Don't most isps check the source address before relaying traffic nowadays? I know at least one of mine started a few years ago (and we had no idea we were asymmetrically routing our traffic till then...)

fail2ban is another layer which is susceptible to abuse and vulnerabilities. It might keep noise out of your logs but at a huge cost. I'd rather just change the SSH port to something non-standard and write it down.

Add it port knocking and this is how I do it. nftables ftw

> and block those IP addresses from the service ports since the traffic source isn't to be trusted.

This means that you are locking out anybody using a paid VPN service, if any other customer of that same VPN service does any kind of scan.

Something I didn't mention in my original comment, but have mentioned in another reply somewhere, is that I have the websites running behind Cloudflare, and I allow Cloudflare's ASN into port 443 but block everything else.

Essentially outsourcing the security of port 443 to Cloudflare.

My use-case is "hobby / enthusiast", so I believe I'm losing nothing and the "world at large" is losing nothing from this setup. Having said that, all policies on this kind of thing need to be strongly thought about in terms of their applicability to the use-case.

Were I running a small or even medium business, I'd probably do it exactly the same with maybe a bit more of an eye on what's being blocked and the ownership of the IP addresses, and I'd have some stats to point to on the range of sources of legitimate traffic. It'd have to be a pretty big, international business for it to cause much of an effect (although I'm talking well out of school here because I don't have anything at stake).

Flipside, though, I have my outgoing traffic routed through a couple of different exits, and I've had to make specific rules for some websites that block traffic from VPNs and VPSs, which is annoying, so I'm not completely dismissing your point.

Lastly, however, at all scales I'd still block the Internet Scanners for reasons I've given elsewhere. Blocking them massively cut down on the uninvited activity - again, it's not about making clean logs, but it really helped clear a lot of the noise.

Good grief. A couple days ago I re-enabled password logins on a server that normally only accepts private keys, just to check something from a third location, and then forgot to turn it off. Two days later the server's logs were full of thousands of failed login attempts that started a few hours after I enabled passwords and then ramped up to dozens per minute.

Just because it didn't instantly say "Goodbye".

I checked ip locations on the biggest offensing addresses; all were in China.

I don't know what to call the idiocy and amorality that leads people to scan port 22 for a living (or the stupidity that leads them to guess random passwords for random usernames that don't exist), but I suppose that for every gardener there are a billion ants.

There's a cottage industry of shitty mass-scanning attacks that continue onto getting root on badly setup fresh installs of various linux distros and drop a rootkit on them.

Some other common targets are websites to be reused for spam (hello, Wordpress!) or to hijack things like gitlab (again to drop a rootkit.

The rootkits are then usually used either for DDoS extortion rackets (usually against game servers, including online gambling), spam (might be less big today than it used to be), and cryptocurrency mining (from my experience mainly monero).

One time it happened in a network I set up due to miscommunication and misunderstanding of how vendor's install scripts worked (by vendor technicians!). During investigation, we found out that this particular "kit" was sold cheaply on a chinese forum (used to be russian forums back in the day, eh), as complete package to run on Windows to attack linux hosts for DDoS botnet purposes.

I always install fail2ban or something like it on servers I want to have SSH on. Really cuts down on the log volume, even if I have locked myself out occasionally. The thing about port scanning is that it's cheap as hell. There's less than 4 billion IP4 addresses and zmap can hit them all within an hour on a decent network connection.

I have SSH access to my server behind a VPN. Not opening port 22 makes life a lot easier.

The name for it is "authoritarian government"

Coincidently, I recently visualized the scanners for fun by plotting them on a globe[1]. It gives a more comprehensive view of the locations and ASNs of the scanners. The demo data is generated from 1 day of logs.

[1]: https://github.com/simonmysun/where-are-the-scanners

Amazingly there's no request from same ASN. I believe this is because the VPS provider has a quite strict validation process, e.g. you have to upload a photo of yourself with your ID and your handwritten username, etc. I would suggest we consider the reputation or credibility of the data centers so that the data centers have the motivation of banning such users. In my case, a lot of the requests were sent from Tencent or Alibaba data centers.

If you have only public key authentication enabled with SSH I honestly don't understand why people bother with things like fail2ban. It just adds more moving parts with very little security gain.

The real risk is a zero-day in OpenSSH and fail2ban probably isn't going to protect you from that. In that case you are better served by putting another layer of defense in front of SSH like a VPN.

fail2ban is the kind of pseudo-security applied just because someone's cousin mentioned that in his blog.

It provides zero security. If your endpoint uses default usernames you will be shot anyway because of IP spread. If your security is good you add something that will block your legitimate connection when you are in the middle of nowhere and, shit, cannot access your <some service>.

"security" is a term that has to be defined in relation to a threat model. If your threat model is an attacker with a static IP hammering your server, fail2ban does provide some security against that sort of attacker.

No it does not. If the packet is at your door it is too late already. Then either it does not matter in which case you do nothing, or it matters (DoS) and then you have other problems.

You are right that security works in the context of a threat model. There are however useless tools that give a false sense of "security" that do not fit in any reasonable model.

I have cases where I block whole ranges of IPs for "legal" reasons - it does not make sense but there you are, the ones who write the rules are not the ones who actually know the stuff.

If your server is on the internet with a public ssh server then it is probably providing some sort of internet service. That internet service is almost always easier to DoS than your openSSH server. If you are not providing a internet service then why is your SSH open to the internet?

You're not wrong, but I'd say fail2ban still has value for junior operators seeking to reduce load and increase stability. If you don't know how to harden SSH, fail2ban is offers a much friendlier way to reduce the volume of logspam, CPU burn, and network traffic. It's just a pity that it's understood/documented/pitched as something that substantially increases security.

> If you don't know how to harden SSH

then you do not open it to Internet. Otherwise you patch aggressively, you use ssh keys and not passwords and you move it to some random port to hide it a bit (it actually helps)

> logspam

you can filter this out in your log management tool

> CPU burn

if this is your concern, then you have a hep of issues you need to address. I have never seen a CPU perf hit because of such behaviour (there are cases where it happens, butthis is due to a vulnerability of the service)

> network traffic

the packet is here already, there is nothing to reduce

fail2ban increases your server performance. It cuts down on enormous amounts of logging from failed attempts, and reduces the CPU used to deal with the failures.

Some sites get a mind boggling amount of attempts. For example I sysadmin some Jewish sites, and they get exponentially more hacking attempts than the sites not mainly used by Jews. (This was before the current war mind you, I'm sure it's worse now.)

People don't believe it's possible for software to be secure, and need a secondary defense to "protect them".

> People don't believe it's possible for software to be secure

Rightfully so. You'd statistically be almost always right considering a software unsecure given enough time (for the vulnerabilities to be introduced then found).

> need a secondary defense to "protect them"

Nothing wrong with that. It's called Defense in Depth and is rather advised. Once you understand that security measures are not bulletproof, stacking them proves to be an easy way to increase protection.

The case of fail2ban is not trivial: reducing log noise is a great perk, and can indirectly help with monitoring (you'd more easily notice suspicious behaviour if it's the only thing on your logs), but it comes at the small cost of setting it up, and accepting the risk of having a shared IP unwillingly blocked.

Except that it explicitly doesn't protect against security bugs.

I always read the main use case had nothing to do with security, but was to reduce log spam.

Repetitive log is something you appreciate by reducing and you don't have to give it unnecessary CPU cycles too.

Fully agree. Limiting the networks which can access your server will help, e.g. limit access to just your local provider or your workplace and you'll see no attempts from Brazil, China, ... unless you are located there, of course ;-)

It's all fun and games, until you travel outside of your country, and try to access stuff at home.

That's manageable with a bit of preparation: when I'm travelling, I allow access from other networks, e.g. those from phone providers. Or add a web form where I activate the IP address with a cryptographically signed "token" which the server can verify and then add the IP address to the set of allowed ones.

Used one or the other every now and then in the last 10+ years and still have my attackable footprint small the rest of the time.

Tailscale (et al)?

How do you protect your vpn?

use a vpn that does not advertise its presence, like wireguard.

I am not sure why this should keep anyone from hosting their own servers and services.

I find it positive to know that whatever and whomever expose anything on the Internet someone will try to exploit it.

For 443 and 80, why the concern ? Outsiders can try all they want bit if you are certain the software you use is secure, there will be no cigar.

I'd much rather have these things out in the open than hiding things away with some obscure thought about that should help anything.

If something is difficult do more of it. The same goes for understanding security.

  > if you are certain the software you use is secure

This is the problem right here. You can be certain that the software you use has security issues.

And who will fire a 10k+ exploit on your server? So you could record it and resell? In the early days, surfing shady sites with Internet Explorer, you could net a lot of interesting js that exploited the browser.

My server is an attack vector for my 10k+ users, and all their contacts. A 1% ransomware infection rate could net them $1 million USD worst case, and potentially an order of magnitude more if one of my users is browsing from a work machine in their network.

Don't underestimate the security value of people hitting your servers, even if all you think you're serving is emojis.

I'm not underestimating. All I'm saying if someone pays 10k or more for an exploit against ssh/nginx/whatever, nobody is gonna pepper your server with it. They will sell it to a broker and pocket the money, end of story.

You will be targeted if your server seems to be the lowest hanging fruit or most easily exploitable or the target is most easily reachable through your site. Otherwise noone will bother with your setup.

The question isn't does the software I run have some sort of yet-undetected security issues, but am I a valuable enough of a target for someone to waste their yet-undetected exploits specifically targeting me?

If the answer's no, then your only job is to keep up with software updates.

If you’re exposing your software to the external internet, you’re potentially valuable enough to get a drive by.

Assuming your software is fairly up to date and/or you haven't badly misconfigured it, they're not gonna do anything. There are a ton of routers and IoT devices that are a much easier catch than a machine run by someone that actually gave a thought or two about securing their server.

Sure. And so what ? Should I stop using it ?

> if you are certain the software you use is secure

The entirety of the problem is that you can't be certain the software you use is secure.

Exactly. And to overcome this you as a user of that software has to be aware of that specific software.

Most people doesn't give a shit, they pull down or introduce dependencies and think "wauw that was easy and fast".

Of course there is secure software, otherwise we wouldn't be able to live as we do.

As history has shown repeatedly, there is no secure software - just software that folks have not yet discovered how to exploit widely and effectively yet.

Then why bother? I'm sorry, but where did this meek, defeatist attitude come from? It pervades software now. Sure, you're right, I guess I could get hit by a bus today, but that won't stop me from crossing the street, because there are a lot of things I can do to minimize my risk, like looking both ways, listening, and crossing at a signal. Software is similar. "Nothing means anything, all is chaos" might poll well on Reddit, but it's not good engineering.

That gives the misleading impression that it is impossible to create and maintain a truly secure software system.

Haveibeenpwned paints a pretty good picture. Breaches, breaches everywhere. The average piece of software cannot be trusted with keeping any data secure for any notable amount of time.

It's funny that password managers and random generated single use passwords are so popular now, because the greatest risk to one's credentials isn't direct attacks, but having them leaked by someone's half assed backend. It gets even funnier when the service that gets breached has some arcane password security rules with two symbols or whatever, the ultimate hypocrisy.

Almost all stories you read about data leaks are some variation of "I installed XXX database and forgot to limit access" or even "and I wrongly supposed it wasn't listening to an internet exposed port". Breaches are just queries.

To be blunt, those breaches are the result of software written by people I wouldn't trust to bag my groceries. I've never had a database get leaked, because I'm not a hack, and I know how to do the bare minimum above professional negligence to secure internet-facing services. I wish I could say the same about most of the industry.

A “breach” usually means they got access to the database, which is much different to access to the underlying server. We aren’t talking about databases, we are talking about servers.

It really depends on the architecture. At least I think it's fairly common for people to have some sort of database proxy running beside the static serve, so there isn't any direct public access and to do some caching, but once you're there it should be pretty wide open.

Common the web servers like Nginx, Caddy are not secure? If they found a zero day in these application whole Internet will go up in flames.

The whole internet keeps patching those flaws as they are found. The problem with self-hosting is patching.

This is a non-problem since the invention of unattended updates. This whole subthread spreads uncertainty and doubt over simple things like nginx or ssh. Service providers don’t patch their software by hand either.

20 years ago, when I was still young and naive, I took these concerns way too serious, remapped ports, believed in pwn, set up fail2ban and knocking, rotated logs. Later I realized it was all just FUD, even back then. You run on 22, 80 and 443 like a chad, use pw-based auth if you’re lazy, ignore login attempts and logs in general and never visit a server until it needs reconfiguration. Just say f* it. And nothing happens. They just work for years, the only difference is you not having tremors about it.

The only time a couple of my vpses were pwned in decades was a week after I gave a sudoer ssh key to some “specialist” that my company decided to offload some maintenance to.

What changed from back then is that software became easier to set up and config and less likely to do something stupid. Even your dog can run a vps with a bunch of services now.

Even OpenSSH almost got a fatal backdoor recently.

What planet are you on? Nginx had a 0 day as recently as April 2022 https://www.accuknox.com/blog/nginxday-2022-nginx-ldap-zero-...

This happens _all_ _the_ _time_

A very specific one that doesn't affect 99.99% of nginx servers.

This seem hopelessly naive just after the windows php bug bit?

https://arstechnica.com/security/2024/06/thousands-of-server...

in the early 2000s I kept an anonymous ftp server open and would routinely get the latest cracked software delivered right to my hard drive. It was very convenient.

Cracked software can contain extra features. Especially when delivered in this way.

Ooo like that awesome techno music on startup, or maybe bee movie during install

I like the idea that someone embedded an entire movie as a malicious payload in an installer.

Well not a hollywood movie, but cracktros are a thing.

I'm sold, send me the link

In the early 2000s it was pretty much expected that each and every computer you encounter is full of viruses. That is, viruses on top of viruses that come by default from everyone running a cracked version of Windows XP.

Most people on here didn't use Windows in the early 2000s, or ever.

I remember a very popular windows xp key that always circulated on the net. it was funny as hell doing some contract work and a client giving me this code his former IT used to install windows.

Im like is this what I think it is.. anyways he ended up buying a legit license since you know lawyer but still funny that I saw it in a business environment.

A very optimistic estimate of this websites demographic.

Oh, when you needed specific ftp clients, because most of them couldn't handle special characters needed to access the directory containing the LOOT :D.

serv-u and cuteftp baby!

Back in the early 90s my first "job" on the Internet was for a pirate group -- I had to scan and find open FTP directories where the group could upload and store their output. There was a specific job title for this position, but I no longer remember it.

"H2O, try before you buy..."

Over 90% of the ssh logins come from just a few China Telecom addresses. They just keep trying random ssh accounts over and over all day. I just geoblock China now. Maybe occasionally unblock it for a few minutes if the kids want to buy something from Shien. Then I honeypot the rest with the continuous ssh banner script.

What's a continuous ssh banner script?

It's a tarpit that slowly sends a message to bots to keep them (and their bandwidth, memory, and CPUs) occupied: https://github.com/skeeto/endlessh?tab=readme-ov-file

I have a utility that parses ssh failed attempts and creates iptables blocklists:

https://gitlab.com/mtekman/iptables-autobanner

For those just wanting the blocklist, here is a table of malicious IP addresses, with columns of: address, number of ports tried, number of usernames tried.

https://upaste.de/bgC

I simply run fail2ban with a whole bunch of customer filters that will ban people very quickly. There's no need to request php or malformed urls when php is not used for example.

I used to run fail2ban, but I found it (or at least its defaults) ineffective against discouraging further requests. With iptables, you can specify the connection to hang for a period and then drop

Defaults are set to reject. Just configure the jails or a global config.

A iptables hashlimit rule can do the same. Your firewall rules get to be more readable and you don't end up relying on the security of a log parser.

The biggest win comes from just disabling password authentication in sshd though.

I run endlessh, I always giggle when I see some connection that last for 2d

a lot of ppl thought this would be a good idea at some point

upaste link is 404

Some time ago I set up a server for a website and I was appalled, like many others, by the number of SSH connection attempts. I decided to open SSH only in a randomly chosen port number above 1024 and now I have essentially zero probing attempt. It is trivial but for me is a satisfying configuration.

This was true in 2018. In recent years I get 100s, sometimes 1000s of login attempts a day on high addresses.

My servers are on AWS addresses. If someone searches for servers (as opposed to routers, phones etc.) AWS might be a preferred address range. No experience whether scan rates depend on the address used.

It appears to be two-stage process.

There are open port scanners that just check what ports are open on which IPs, and there are separate ssh login brute-forcers. Once your machine gets picked up by the former, the latter will pile up.

I have two servers on adjacent IPs, both with ssh listening on a high port. One gets hammered with login attempts and the other does not.

A lot of these seem to use zmap (https://github.com/zmap/zmap) or masscan (https://github.com/robertdavidgraham/masscan) for the initial scan.

Often with default parameters such as zmap setting ip id to 54321, having tcp initial window at 65535, having no SACK bit set and masscan with no SACK bit either, tcp initial window at 1024, tcp maximum segment size 1460 (which is strange to put below initial window size!), (older versions having fixed src port 61000 or 60000 from documentation examples and no MSS set), all of which are extremly uncommon in legitimate traffic and thus easily identified.

Even those so called "legitimate" scanners (emphasis on the "") seem to use these tools with little or no extra configuration.

With this setup the last time my high-port ssh (key-only) has got an attempt on it was 2023-07-26 (previous intruders get permanently firewalled).

This might not matter for your setup, but I would have thought it's bad in general to have sshd listening on a high port because then any non-root user who finds a way to crash it can replace it with his own malicious ssh server on the same port.

You mean non-root local user? We don't have non-trusted users on the system.

Well, unless the http server or our dns resolver has a remote code execution vulnerability.

So directly I don't see the risk you describe. Of course considering maximum defense in depth you might have point.

That's a good point, though you could use some firewall rules to rewrite the port number so that the local daemon is listening on the normal port but accessible via an alternate high numbered port.

Maybe that's the case. The machines where I am seeing a lot of ssh login attempts on high ports have been on the same IPv4 address for years. Some since 2018.

Interesting to know. For the moment, several months, I still have no login attempts but so that means my server didn't get picked up by any port scanner.

addresses == ports in your view?

Yeah, sorry about the mistake. Too late to edit the comment :(

  > 1016 cd ~; chattr -ia .ssh; lockr -ia .ssh

Does anyone know what the "lockr" command is? I can't find any references to it besides people saying they observed malware trying to run it, usually (as is the case here) right after a chattr command with the same arguments.

https://blog.netlab.360.com/icnanker-trojan-downloader-shc-e... has:

    cp -f /usr/bin/chattr /usr/bin/lockr

https://www.lockr.io/

I think that's something totally unrelated that just happens to have the same name. I don't see anything in their docs that even hints at a UNIX command called "lockr", let alone one that makes sense to call like that.

> 8181 root

In 30 days? That's tad unrealistic.

Just checked and there are dozens root login attempts per minute on my colo'ed server in the EU. Virtually all from the Chinese and post-Soviet IP space. But mostly Chinese.

I see ~1000 unique IP addresses hitting SSH every day.

> In conclusion, these commands represent a clear strategy to infiltrate, assess, and establish control over targeted systems.

Oh hello, ChatGPT. You seem to be everywhere these days.

Amazing article!

It is actually amazing how fast and thorough the connection attempts happen as soon as you put anything online.

I've been playing around Hetzner and Coolify recently, and notice that, as soon as port 22 is opened, it is bombarded by those attempts. Several per second. It might be due to Hetzner IPs being reused, but happened to me every single time. Same with Postgres default port (those were the ones I've seen).

I have defaulted to use Terraform and bash to only open those ports in the Hetzner firewall (and more common ones like 3000 or 8000) to my own current ip. It does mean I'll get drift and need to reapply the Terraform code if I change ips, but seems to be at least one way to defend.

I fear that a lot of devs jumping into the "you only need a VPS" crowd on Twitter will end up with a huge attack surface on their apps and machines and most won't even know they are being targeted like that most of the time.

To this day I still find it hard to find a comprehensive security guide for those newer Linux fresh boxes (and the ones you find are all so very different with different suggestions). If anyone knows of a good one, please share with me!

You just need to turn off password authentication so it's keys only. They can attempt logins all they want and never get in.

Also if you run ssh on a nonstandard port you get many fewer attempts. There are several groups that constantly scan all of ipv4 for open ports, if you use ipv6 they cannot scan that space anymore.

Optionally you can set up fail2ban but I find it's not a big deal.

I changed my SSH configuration to only listen on an IPv6 address 6 months ago and since then the number of SSH attacks has fallen from 1000+/day to less than 10/week.

Thanks!

That is usually what I already do. Good to know I'm on the right path.

When possible I disable root login as well (though Coolify seems to need it on, even if without password).

I would recommend just using a VPN, like tailscale, for all non-public resources - rather than IP whitelisting.

Ed: including private web services like self-hosted gitlab not used for publishing public projects.

It's on my list to try. Haven't sat down to actually try using Tailscale with servers yet but seems like a good option. Thanks!

Not sure if op will see this, but with regard to his comments on MikroTik routers and frequently seeing in his honeypot logs, the command: /ip cloud print

he is correct, This is a MikroTik command- although mikrotik has this feature, disabled/ off by default, a lot of users make use of it, and running that command will (if cloud dns enabled), will show the dynamic DNS entry of the device he is connected to. Ie if the cloud DNS is enabled, the output from that command will be something like: Detected public ip: 34.2.82.3 DynDns: djwisyehd.clouddns.mikrotik.com (which will always be updated to the detected public IP address of the router)

So I assume the attackers run this command so that they can still reach the router in case it’s public IP address changes at some point. (And assuming that the device will still be accessible after any public IP address changes).

(or perhaps they run that command to see if the cloud DNS service is disabled, which is the default, in which case they will then enable it so that they will have a dynamic DNS entry for the device).

What does `echo -e "\x6F\x6B"` do?

If you say it 3 times in front of a mirror, it summons Stallman

With or without the swords?

Only one way to find out!

Haha

Maybe I should create a honeypot where cat, echo, sed, and curl/wget all drop random bytes in all commands they execute

Would be fun

Better would be to just subtly change the output...

Like do a +1 on the byte every 7 bytes. Bonus to do it only on every 7 printable chars.

And you can even do A/B testing on the constant 7.

Tests whether `echo` supports the `-e` option.

It prints "ok" and shows they got in (it relies just on a shell, nothing else).

Why not do 'echo "ok"'?

As shown by someone having to ask what it does, it obscures what it does.

This look like a simple test to see if remote command execution works.

It echos "ok".

Prints out `ok`

A bit tangential but is there a service or self hosted solution that would take a list of IPs and then keep scanning them periodically and alert me if any new ports have suddenly open?

I think shodan could br useful in this regards

https://www.shodan.io/

hmmm....you could do that with nmap script and a cronjob.

I just scanned my domain for all 65k ports and it took 20 seconds with a 10gbit pipe. i could scan yours for you and shoot you an email if a new port is discovered. Would charge you Like $100/year or something.

I simply block traffic from countries where I do not do business in.

I used to see constant attempts to mess with Wordpress URLs, which I know is not legitimate because I don't run Wordpress.

Cutting out Russia & China basically removed this problem. I really hate locking up my tiny corner of the internet but I don't see another way.

If you want to try this yourself, although he carefully doesn’t mention his software, it appears he is running the Cowrie honeypot (https://github.com/cowrie/cowrie)

(Note, I’m the maintainer)

This story reminds me of the time I accidentally/naively set up a ssh honeypot when I configured the router I was using at the time (this was a long time ago) to forward incoming SSH connections to a ReadyNAS (which was using a Sun SPARC processor) in my house. I did that so I could log into it while I was away from my house. One day, I ssh'ed in and notice that the ReadyNAS was running very slow which surprised me because I thought it was idle. I checked the CPU usage and the sshd was using 100%. First, I thought it was a bug but it occurred to me I should check my incoming ssh connection attempts in my router log. Turns out there were a ton of ssh connection attempts coming from an IP address assigned to China. In response I changed the router port forwarding for incoming ssh connections to use a non-standard port number, like 55,243, and after that my ReadyNAS was no longer bombarded with ssh connection attempts. Lesson: try to avoid forwarding standard port numbers.

There's a project for running Honeypot as a Service: https://haas.nic.cz The data is public and you can register your router too

What is a `lockr` command? Is it file system specific or something? Never seen anything like this. It probably should lock permissions on .ssh, but how?

Somewhat related due to a weak password a mail server from a community I'm involved in send out lot's of spam mail, after analysing the log files I've had over 1500 different IP addresses that logged in to send spam, about 10 mails for each address. ASN and subnets where spread across over the whole world. It seems like these attacks are coordinated using vast botnets and the use of single ssh public key here seems to confirm this. I had similar experiences going after attacks on WordPress instances and there I've also found attacks spread out across lots of hosts.

I'm wondering if it's possible to pin down those behind these attacks, there must be mistakes.

Interesting article, sadly due to my exposure to LLMs I couldn't help but notice that the parts about "oinasf" and sakura.sh are AI-edited at least. Kind of a weird choice considering that a lot of the article was clearly human-written.

3 simple sshd config lines remove a gigantic amount of worries if you run open ssh servers:

   Port NN
   PasswordAuthentication no
   AllowUsers user1 user2 user3

change those, sleep at night.

Check out https://viz.greynoise.io/ especially the trends > anomalies tab is very interesting

How do you use that information?

Had a own server in university during mY PhD. Most request were trying to download scientific papers from large journals using absolute and not relative URLs after request.

I once tried hosting a web server at home by exposing ports 80 and 443 to the Internet. Hours later I reviewed the logs, thousands of attempts to hack into my lil Linux server. It spooked me to say the least, so I switched to using cloudflare tunnels instead.

Exposing ports on the Internet is dangerous, especially SSH. You're much safer using a proxy or gateway of some sort, or better yet a VPN if it doesn't need to be publicly accessible.

Is it? If you've got `PasswordAuthentication` disabled, only allow public key logins and keep your system up to date. Honest question.

I self host my email ( docker-mailserver ) and host my personal website on an old laptop with a static IP. Have done for years now without issue.

The keyword is diligently keeping your system up to date! That said you’ll still have exposure to zero day vulnerabilities and DOS attacks.

But an attacker with one of the biggest vulnerabilities on earth (hell, ssh noauth 0day) would very likely use it against big cloud providers and infrastructure (isps and others) and not burn it on your home server! Keeping it reasonably up to date with your distro's cycle is probably enough for most people doing this home server thing.

So of course, as things always are with security this is a matter of risk assessment and understanding your attack surface, a server with only public key and maybe on a special port goes a very long way, add fail2ban on top and i'd say it's probably fine for quite a while.

But that does make me think... what if... a wormable noauth 0day like that on ssh or some other popular system... how fast could it replicate itself to form the biggest botnet.. how long would it take, to take over all visible linux servers on the internet (so that your little home box ends up being a target)?

I guess at that point you are limited by bandwidth, but since you can scale that with every compromised server... hope someone does the math on that one day!

https://wiki.debian.org/UnattendedUpgrades Most distros have something like this.

"PasswordAuthentication disabled" not sure I can even do this on my shared BSD server. I have ssh access via pw and need it. Is this really dangerous?

Yes, it's risky to accept password auth if someone sharing the box with you has a poor password. They could do things like:

. Install a spam or brute force password bot, which could get the machine kicked off its internet connection (in addition to whatever havoc it causes first)

. DoS the server by filling up the disk or using too much RAM (are quotas enforced?)

. Exploit a local vuln to get root, if such exists on that box. (Is the kernel promptly patched and the box rebooted?)

. Explore other users' directories (are permissions locked down correctly across users?)

…and more thrilling possibilities!

Embrace key auth. Future you will thank you.

It is, if for no other reason than you never know when some other user has a guessable password. You should switch everyone to ssh keys. It's a good excuse to learn :)

Yes. Authenticating with passwords is obsolete and dangerous. Use keys and disable password auth.

How good is your password? If it's long, with special characters, it's fine. Install fail2ban. The problem with auth keys is you can't get into the server if you don't have your laptop/phone/NFC device because you got pickpocketed/mugged?

"I self host my email "

Is this still possible? Are your emails getting delivered?

Downvoted. I don't know when the downvoter tried the last time to "host their own email". Yes, DMARC, DKIM und SPF. Good luck trying to get your email deliverd to t-online or something.

https://forum.hestiacp.com/t/t-online-curious-story-about-th...

They may even check if your domain has an "imprint". I kid you not. I use my own domains too, but I piggyback with infomaniak.com

> Is this still possible? Are your emails getting delivered?

Mine are. Although it probably helps to have a static IP with a 25 year long clean history.

Are there very occasional glitches? Sure. But I've seen ISPs drop everything from GMail on the floor for no obvious reason. I've seen GMail drop GMail email before. Same for every other large email provider.

To date I haven't seen any reason strong enough to push me to switch to a centralised email host. That day may yet come of course.

I self-host my email, and have not really had problems delivering normal quantities of personal email (except a bit of pain for Microsoft to accept mail in the first place, but it can be sorted quickly) - as long as you do DMARC / DKIM / SPF.

I've never heard of t-online before or tried to send an email there to my knowledge... if one provider I've never heard of would refuse to accept my mail if I ever sent something to them, that's more of a them problem than a me problem - but it certainly isn't the norm for other providers.

> Is this still possible? Are your emails getting delivered?

Yes and yes (if DMARC/DKIM/SPF configured correctly).

> Good luck trying to get your email deliverd to t-online or something.

People who say it cannot (or should not) be done should not interrupt those who are doing it.

The dismissiveness is likely why you are downvoted, I'm guessing. The suggestion that because it's hard for you and therefore you're surprised others are doing it isn't a good look.

Self hosting email isn't that hard, and there are many solutions for all sorts of self hosting issues. That's a topic for another discussion, though.

I fo it too and can deliver to gmail/office365 etc addresses no problem.

yes and yes.

Selfhost does not imply residential IP.

Don't worry, they are usually Russian/Chinese ips scanning for 5 year old php exploits. I've been exposing ports to the internet for decades with no issues. Always block ssh password and keep software relatively up to date. If you are very paranoid, make a vps beacon and remotely tunnel ports from your lab to it. That way you only expose the beacon.

I wonder, what is the issue with authenticating by password. If you choose a password of lets say 64 random chars, shouldn't it be pretty safe? Or is there something in the password method itself, that is inherently weak?

> Or is there something in the password method itself, that is inherently weak?

Your 64-character high-entropy password might be safe; other users on your system might baulk at memorising/typing in 64 random chars, and choose a less-secure password instead. With SSH keys, that can't happen.

Sure, they probably won't crack that, but there are other things to consider as well. A sshd on IPv4 port 22 that accepts password auth attracts attention, and you'll spend CPU cycles constantly checking credentials from very large database dumps that float around. In my experience it leads to more log noise too, it seems many bots will discard your IP and stop pestering it if passwords aren't accepted.

So in practice you'll probably also use something like fail2ban, firewall rules that only allow connections from certain IP blocks, things like that.

The first benefit is some bots won't bother testing passwords as the SSH error message tells them the server doesn't use password auth. The second benefit is if your server is compromised it's quite easy for a rootkit to hijack SSH and steal your password when you login (and then abuse that on other servers you use it), the same is not true with a key and it is much harder for a rootkit to abuse as long as you only use the key on your local machine (there are strong protections against SSH handshake MITM attacks afaik)

> Or is there something in the password method itself, that is inherently weak?

You have to send your password/hash. With PKC, your private key never leaves your device. It can even live on a separate security key. All you ever send are signed messages, never your key.

There are still advantages to public key auth. Sibling comment mentioned resource use, but also consider ease of use: are you setting a random 64-character password on every machine that has SSH server installed? Would it not be easier to generate one ed25519 keypair, apply a reasonable passphrase (and/or use disk encryption), and then you have secure auth on all your machines without a password manager?

If you're _not_ setting unique 64-character passwords per server, then you should consider what happens if your super strong password is discovered -- an attacker would have access to all your boxes. Compromising a key is harder than compromising a password.

I've been doing it for 25 years. It's fine.

”Works for me.“ does not really answer the question.

Having a 25 year history might be why your mail gets delivered, while many people trying to self-host have constant and unpredictable deliverability issues.

It's more an advocacy against security paranoia.

You will always get automated attacks, constantly. But they're almost all doing stuff like trying to exploit a 12 year old bug in Wordpress or IIS.

They're about as sophisticated as any other scammer on the net.

I checked the logs for May for one website I run---65% of failed requests were for PHP scripts (mostly Wordpress). I don't run PHP so I don't worry. The rest of the requests were bots that can't parse HTML [1] and other weird requests. I've been running a webserver, SMTP, SSH and DNS for over 25 years and only once had an issue due to an inside job [2] twenty years ago (hard to protect against those).

[1] https://boston.conman.org/2019/07/09.1

[2] https://boston.conman.org/2004/09/19.1

Out of curiosity, what are the ramifications of exposing ports 80 and 443? Can these ports even be 'hacked'?

It doesn't seem terribly unsafe to me, especially if you're serving static pages.

In my experience, most of the noise on my web server are bots with spoofed iPhone or Google Chrome user-agents. I see three kinds of traffic patterns.

1. bogus /wp-login.php requests, or endpoints of presumably insecure wordpress plugins. These bots are pretty dumb and do it non-stop, even if the server constantly responds with a 404

2. testing recent Apache vulnerabilities by POST-ing to something like /cgi-bin/.%2e/.%2e/.%2e/.%2e/bin/sh . Even if your web server clearly communicates that it's not Apache, the bots still insist on testing Apache vulnerabilities. They also occasionally test vulnerabilities that exist in ancient Nginx versions.

3. less common, but bots that exist to scrape something from the internet. I remember two years ago seeing a bot whose sole purpose was to document as many registered, valid domain names as possible (I found out about this since they linked a website explaining who they were in their user-agent string)

Overall, I would say the background noise of HTTP servers is tame compared to what you see for SMTP servers and, to some extent, SSH servers. I happen to also self-host e-mail; logs record failed login attempts about every second. They always pick a username like "admin" or "adm". There's also people who try using your SMTP server as a relay for spam.

For me the biggest source of noise in logs for a small site is the referrer spam. At some point like 12 years ago I enabled webalizer stats with a public link to the stats page. Soon I had to deal with massive amount of bot requests with http referrer pointing to porn and farmacy ads. That has not stopped after the public link was removed and the stats has started to use a public spam database. And the spam is still there after 12 years.

> testing recent Apache vulnerabilities by POST-ing to something like /cgi-bin/.%2e/.%2e/.%2e/.%2e/bin/sh .

Are they really recent vulns though?

Gotcha, thanks for the detailed response. I've seen the WordPress login attempts in my own web server logs, and that seems to be corroborated in your comment.

I've added a /wp-login.php and friends that firewall-blocks the IP of the requester for a week. It greatly cuts down the bot noise.

https://arstechnica.com/security/2024/06/thousands-of-server...

99.9999% of issues on 80/443 are apps run on the server not webserver itself.

It is applications that you run on web server that are exploited.

So serving static pages is safest thing you can do.

> Out of curiosity, what are the ramifications of exposing ports 80 and 443? Can these ports even be 'hacked'?

These are the ports usually employed to serve HTTP and HTTPS traffic, which mean public-facing servers.

Having a server listening to those ports is the precondition to have web servers running specific types of services, some of which have known vulnerabilities that can be and are exploited.

Ports can't be hacked but the application listening on them can ;)

You can have vulnerabilities on the server software and its configuration even if you are serving only static content. This should be unlikely if you use up-to-date battle-tested software like nginx without making crazy config changes.

If you serve dynamic content, that may also have vulnerabilities that hackers can exploit.

> I once tried hosting a web server at home by exposing ports 80 and 443 to the Internet. Hours later I reviewed the logs, thousands of attempts to hack into my lil Linux server. It spooked me to say the least, so I switched to using cloudflare tunnels instead.

Isn't this hypothetical risk mitigated or outright eliminated by using stateless apps and periodically redeploying them in the spirit of cattle?

Depends, If they get into the stateless app and hoist that to penetrate into other stuff in your network, they might be able to install an APT.

> (...) they might be able to install an APT.

As you're periodically doing clean redeployments, that's not a concern isn't it?

I noticed earlier this year while deploying a CoreOS VPS with terraform that sometimes you'd get an interesting IP that would receive incoming HTTP requests for interesting domains such as theguardian.com. I of course destroyed and re-deployed the VPS several times so the interesting IPs are lost to me, but it might be worth running a HTTP honeypot as well as an SSH one.

The traffic doesn't matter if you are sure your setup is secure. Key auth only for SSH, reverse proxy in front of your actual web server and use secured containers or VMs for each service. Throw in fail2an or crowdsec and that's more than enough for a little home linux server.

Every things on the internet is doing exactly this "dangerous things", with the exact same means you have at your disposal.

Exposing a service is not dangerous.

It is the same thing when you go to the sub and many people ask you for money : they keep asking, but that will not lead you to your bank account.

So you have log, this is not an issue, this is not something to be scared of or even cared of.

Just ignore them, as they are worthless and part of the v4 internet.

Yeah this is what keeps me away from self-hosting public facing stuff. To me its like opening a new pipe into your home that is open to the whole world. And I'm too carefree to get the settings down right. So I avoid it all with complete process isolation. Don't shit where you sleep!

But couldn't, you, within your home, separate it from everything else? I don't see how it's any more dangerous really.

I should clarify. When I mean self host it’s for public facing applications that generate revenue. It involves some transaction in currency?value? between the user. Once money is involved you become a target. I don’t want anything that could be traced to my physical address. I told you I’m careless, I’ll eventually slip up on installing the patches or configuring something right.

Public facing like serving some static webpages or blog, text content. Yeah do it.

Obviously you need to know how and if you don't then it's always going to look very daunting.

We run internal sites that are on the public facing web - the logs from Akamai are a daily list of mostly the same requests to find unsecured Wordpress and MySQL installs, .cgi and php files and paths like "..%C0%AF../..%C0%AF../..%C0%AF../..%C0%AF../..%C0%AF../..%C0%AF../etc/profile"

In 24 hours theres anywhere from 7000-9000 log events just from the CDN

How do people feel about using docker as a way of avoiding 0 day vulnerability

It's all for personal use and maybe I'm just cosplaying as a sysadmin but I have apache proxy-pass ing to sets of docker containers. So as long as apache and ssh are kept up to date (on nixos), even if all my services are 0 day'd, they have to also escape the docker containment

If anyone is looking to run IP metadata based reports on their honeypot, I can suggest IPinfo's CLI (https://github.com/ipinfo/cli). Here is my summary report from Fail2Ban:

``` Summary - Total 1490 - Unique 153 - Anycast 0 - Bogon 0 - Mobile 52 - VPN 91 - Proxy 12 - Hosting 1003 - Tor 0 - Relay 0

Top ASNs - AS132203 Tencent Building, Kejizhongyi Avenue 409 (27.4%) - AS14061 DigitalOcean, LLC 148 (9.9%) - AS135377 UCLOUD INFORMATION TECHNOLOGY (HK) LIMITED 72 (4.8%) - AS16276 OVH SAS 72 (4.8%) - AS206264 Amarutu Technology Ltd 44 (3.0%)

Top Usage Types - Hosting 955 (64.1%) - ISP 418 (28.1%) - Business 49 (3.3%)

Top Routes - 43.134.64.0/18 (AS132203) 48 (3.2%) - 183.81.169.0/24 (AS206264) 44 (3.0%) - 43.156.192.0/18 (AS132203) 36 (2.4%) - 43.130.0.0/18 (AS132203) 36 (2.4%) - 43.134.0.0/18 (AS132203) 36 (2.4%)

Top Countries - United States 276 (18.5%) - Singapore 221 (14.8%) - China 141 (9.5%) - France 88 (5.9%) - Japan 86 (5.8%)

Top Cities - Singapore, Singapore, SG 221 (14.8%) - Santa Clara, California, US 100 (6.7%) - Hong Kong, Hong Kong, HK 80 (5.4%) - Tokyo, Tokyo, JP 61 (4.1%) - Amsterdam, North Holland, NL 50 (3.4%)

Top Regions - Singapore, SG 221 (14.8%) - California, US 134 (9.0%) - Tokyo, JP 86 (5.8%) - Hong Kong, HK 80 (5.4%) - New Jersey, US 70 (4.7%)

Top Carriers - Africell 12 (0.8%) - Claro 12 (0.8%) - Vivo 12 (0.8%) - WINDTRE 12 (0.8%) - Telekom 4 (0.3%)

Top Privacy Services - TunnelBear 36 (2.4%) - Best Proxy Switcher 12 (0.8%)

Top Domains - ovh.net 60 (4.0%) - googleusercontent.com 24 (1.6%) - prod-infinitum.com.mx 16 (1.1%) - poneytelecom.eu 12 (0.8%) ```

Here is the command I used:

``` cat /var/log/fail2ban.log | ipinfo grepip -o | ipinfo summarize ```

The CLI is free to use. You can also do `bulk` enrichment.

``` cat /var/log/fail2ban.log | ipinfo grepip -o | ipinfo bulk -c > fail2ban_ips.csv ```

Disclaimer: I work for IPinfo. However, the CLI is free to use, and the bulk feature will usually work within your free tier limits. Ping me if you have any questions

I somehow found myself in charge of a computer lab two decades ago... and idiotically set up admin controls via SSH.

The entire lab was down for almost a week [immediately hacked], and then I suddenly moved a few states away.

Why is 345gs5662d34 the 2nd most tried username after root? Bizarre.

I opened my personal server's 22 to the world because I screwed up my vpn config a couple weeks ago. I just had a look at the auth log and closed it again. It is non-stop.

A perfect example of why one should use SSH over a mesh network like Tailscale, and don't expose over the public internet. No attack surface means no attack.

I love TS just for this reason. All ports are locked and ssh-ing is possible only via TS. And for public facing web apps I open only 80 and 443.

Does anyone have any experience with CF tunnels on free account? Is it actually free for smaller apps with less than 1TB of traffic per month? I was wondering about switching to CF tunnel which would mean I could also close 80 and 443 ports and block China (because I read somewhere that most of DDOS attacks come from Chinese locale botnets).

For some additional peace of mind, you could also use something like Authentik in front of your web apps, so you don't expose the apps themselves, only Authentik. You can then use the IDP of your choice within Authentik for authentication.

Thanks, I was thinking about small but public project.

Yes, CF tunnels are $0 for very small users. I have this, as do many others, as a reverse proxy for stuff like Home Assistant and it works great.

Thank you, I'll have to try them

I was always wondering, how do you guys geoblock entire countries/ip ranges? Do you just use free ip lists you find on the internet?

My new VPS got an SSH attempt in 5 minutes after I purchased it. I'm now in the progress of running a similar honeypot experiment.

If you push it you can scan the entirety of IPv4 in about five minutes.

I've been running self-hosted servers for the last 25+ years without an incident and its less complicated than it might seem if you learn a bit about securing unix-based systems (ok, I already had 10+ years of server admin knowhow for various systems, but anyway, it's not rocket science ;-). Yes, an hour or so after you connect any machine to the Internet, you'll see attempts to "talk" to your server. So don't wait to set up basic security. But it actually has never been so easy to "just give it a try" (see below), with all the virtual offerings today. So here's a short/raw sketch of basic things you'd need to do:

1. 25+ years ago I used http://easyfwgen.morizot.net/ to generate an iptables based local firewall. Still works fine (then and now tweaking some things) and allows only certain ports too be accessed at all. I just open email, ssh and a web server.

The generator is well documented and still works, although it would be nice to see an updated version to newer firewall software like pf.

2. server configs:

edit /etc/hosts.deny --> restrict all by default

  ALL: ALL

edit /etc/hosts.allow --> allow your service providers networks, e.g.

  sshd: .t-dialin.net
  sshd: .dip0.t-ipconnect.de

So you can connect to your machine for further setup, but not the whole world.

3. set up sshd:

edit /etc/ssh/sshd.config

  # allow key-based access only
  PasswordAuthentication no

Maybe change sshd's port (reduces log file entries) but don't forget to allow this port in your iptables setup and your /etc/hosts.allow

People have opinions an key-based access, I know. But my private and public key is stored in various secure locations, including my phone (password safe) and I can access my server even from my Android phone or tables via Termux.

4. set up email (I suggest postfix as an MTA):

configure restrictions in /etc/postfix/main.cf, e.g.

  # restrictions in the context of the RCPT TO command
  smtpd_recipient_restrictions =
        reject_invalid_hostname,
        reject_non_fqdn_hostname,
        reject_non_fqdn_sender,
        reject_non_fqdn_recipient,
        check_sender_access hash:/etc/postfix/sender_access,
        reject_unknown_sender_domain,
        reject_unknown_recipient_domain,
        permit_mynetworks,
        reject_unauth_destination,
        [...]

  # restrictions for clients connecting
  smtpd_client_restrictions =
        reject_unauth_destination,
        check_client_access hash:/etc/postfix/access_client,
        reject_unknown_client,
        reject_unauth_pipelining

This heavily reduces the amount of spam you'll see. I add greylisting too, as this even nowadays reduces even more unwanted traffic. Combine that with spamassassin if you like. This setup gives me maybe one spam per day reaching my inbox (actually the spam subfolder).

5. Learn by doing (not just reading stuff on the Internets ;-), that is, set up a machine, e.g.

If you'd like to experiment a bit, take a look at Hetzner's unexpensive cloud servers, these are easy to set up (incl. a virtual firewall in front of it) and take down after some experiments of a failure. You can do this in Hetzner's web interface, even if you misconfigure your server to be unaccessible. Cf.

https://docs.hetzner.com/cloud/servers/overview/

Tip: Hetzner's web interface allows you to pre-define an ssh key which they'll install automatically on your new machine (but they leave password login enabled, so change that asap).

Disclaimer: I'm just a happy customer, no other relation. And it might be as easy to do this with Digital Ocean, which have some nice tutorials too, for example on the set up of a web server:

https://www.digitalocean.com/community/tutorials/how-to-inst...

Last but not least No Starch Press overs some nice books like "How Linux Works" or "The Linux Command Line" (if you're not sure about that) or even "Linux Firewalls: Attack Detection and Response" ...

You learn most by trying.

I'm now heading for the beach to enjoy some offline adventures and will answer questions later if needed.

I guess may the compromised host was probably also use same weak password as it's Brute force other host.

My website backend APIs get repeated attempts at javascript prototype injection, all day, every day.

(Long shot) I really would like to USA a spare machine for web serving a Jupyter Notebook server, but I did not found a single resource that blocks everyone except a single IP or something like this. Supper annoying to pay some cloud providers to have a resource that I already have.

We use port knocking and haven’t had a single hack attempt in many years.

How did he expose his honeypots and make the bots aware of his existence?

If your server has something that listens on port 22, you just have to wait for like 5 minutes

internet is cool and badass

I wanted to read more about the interesting part!

dont ever run publicly exposed production SSH. If there is vulnerability in your version of ssh, you risk getting pwned.

Most of this nonsense disappeared when I adopted wireguard and later Tailscale.

I am amazed we have not yet said "Hands off!" and coordinated physical interventions against the scum who attack our electronic brains.

Is it so hard to kick in the doors of those whose IP addresses are used to try to hack honeypots?

This lack of action is why I oppose all law enforcement. Until they do their jobs, they do not need to be paid.

Reply on news.ycombinator.com