Cowrie: a medium-interaction SSH and Telnet honeypot | Hacker News Reader

Cowrie: a medium-interaction SSH and Telnet honeypot

github.com

127 pointsFazel9454 comments

Caution for folks: I ran this for a few weeks and it was very informative. You see how simple many attacks really are, how many corporate assets are comprimised, and where most of the source attacking countries are.

But, there is a downside.

The downside is any site you put it on will be flagged and attacked 100x more since it registers as a vulnerable destination. After turning it off, the deluge of attacks for the site continues and I ended-up rigging a fail2ban setup to just deny if a destination port is hit. (See >> https://mergy.org/2019/08/setting-up-a-killswitch-for-attack...)

But, it is fun and interesting to see. Highly recommend but know the legacy on that as well.

You should not put this alongside your site, at least not sharing resources. You should apply forwarding rules to separate this into a different vlan or vpc (ip based routing for instance) and completely isolate this from your site, while the two ssh destinations would be indistinguishable from the outside, preferably behind a load balancer with a firewall to mitigate ddos attacks and just log different attack types

Here is what a playmaker session looks like if you are interested

https://vimeo.com/345068825/86f8c8f97e

Agreed, it's mostly old bots trying to take over wordpress sites and serve malware. That's about it.

For a while, I gathered attempted usernames. I created accounts for all the usernames, set a null password on my chroot sftp server. I was really hoping they would try to upload something interesting. Nope. If they can't get a standard shell, they just keep retrying in a loop, forever. I've had the same bots hitting my server every few minutes for several years. No harm, no foul, I let them have at it.

> records the interactions of hackers

Before you get too excited, I would say that a common mistake of inexperienced sysadmins is assuming all those brute-force attempts in the logs are results of those "hackers" and "crackers", it's not, not even a scriptkid.

All you can get by running a honeypot, like this one, is pretty boring activities by soulless, ancient worms and viruses, or automatic global Internet scanners running 24x7, not humans. Most of those are not even worth your time to block (e.g. if you only use strong password or pubkey, there are few reasons to fail2ban).

It's nothing personal. Any machine will be port scanned, vuln probed, brute forced, blindly hit with ancient "1 shot" exploits (e.g. ?file=../../../etc/passwd ). This is how the Internet works.

Another lesson is: never run any unsecured webserver/service on the public Internet, never ever, not even for debugging, period. Don't listen 0.0.0.0:80 if you have just installed your PHP management system, don't reset your forgotten MySQL password by disabling privilege checking before turning off networking first, if you just installed a new VPS with root password 123456 in a morning, don't wait until afternoon, change it immediately, etc.

The reason is exact the opposite, not because of "hackers" but those stupid worms. Ordinary life is boring: if you run a webserver with password 123456 (e.g. for debugging an issue on a disposable server - just for 20 minutes, you think, then you forgot it), you won't (or unlikely) to see someone hacking into your system, but it's a certainty that one of those stupid worms/viruses would infect your machine within hours, sometimes it's as quick as having your lunch. And it probably won't do much damage, but you would spend your time to reinstall the system again...

Indeed. After running this for a few weeks, most of the playbacks where a couple of seconds and consisted of a script that, once it obtained access, would just try and pull down payload sitting on a vulnerable host (in my case on the OVH Euro ISP) and then try to trigger a cron setup and exec in stealth mode on linux.

Rarely did I ever get an interactive shell show someone trying to move through the process. When I did, it was fun to watch. I would say automated 1-2 sec script attacks outnumbered human logins 1000 to 1.

Thanks for sharing.

On the other hand, I heard that running sandboxes to obtain the latest samples of IoT viruses and tracking their evolution is fun.

> I would say automated 1-2 sec script attacks outnumbered human logins 1000 to 1.

After writing my own honeypot to get up to speed with some SSH details and satisfy curiosity I was kind of disappointed by that fact tbh. Though I expected the level of automation and boring stuff, I let a similar fake environment run for weeks in the hope of some human "coming back" for details but it honestly did not reveal more exciting information than you get from honeypots that make their data public.

Nobody else ever got to see the forced little ncurses chat I've built in to talk to the human on the other side, shutting that down was not the happiest of moments. But yeah, fun to watch it was.

Wasn't it just a couple minutes at the hight of Windows XP?

Indeed, /cough/ ransomware /cough/.

Seconds. Back when NAT routers weren't everywhere because you only had one computer and Windows XP didn't yet have a built-in firewall, unless you installed a third party firewall like ZoneAlarm before connecting a new installation to the Internet, it would be infected very quickly.

Oh yeah, the "infected before you can install patches" time

Thanks for the memory. Your comment reminded me about the suggestion "You must disconnect your machine from the Internet before/during the installation of patches." from 2000s computer magazines.

Worked IT in the college I went to. One of the more permanent members told me that if you plugged a fresh XP install it'd be fucked in about 30 minutes. MS apparently came to campus to look at some of the stuff coming in.

When I stopped working there, they were still assigning everything an ipv4 address, no firewall. Great for running game servers in dorms, not as great for getting your printer hacked.

> if you only use strong password or pubkey

Don't use passwords. Disable passwords. It's tempting to say "Oh we'll require 20 character long random strings, it'll be fine". If you've dotted every I and crossed every T that would work, but such passwords aren't memorable so they're no better than a private key - you'll need to store them somewhere and make sure to generate them properly, you might as well use keys.

More often something will go wrong. Maybe it will be a local policy failure. "Oh I didn't realise devops needed those goofy passwords too, I thought that was just the backend team - we've been setting them all to pass1234, should I change them or is it too late now?". Maybe it will be in the password checking code, as it has happened way too often already on a broad spectrum of operating systems. Just use pubkey auth everywhere all the time.

When I setup a new server or a temporary account, my usual workflow would be simply generate a new random password locally from my password manager, before considering proper authentication. A few routers at my home with public Internet access are still password-only, and I don't see a big problem.

But I have to agree with you, the suggestion "use strong password" itself is prone to failures, thus bad, and should be avoided.

I remember setting up my first VPS a few years ago and finding automated login attempts in the logs within twenty minutes of it going live. Since then, I always install fail2ban as one of the first things I do on a new server. (Not that I don‘t use good passwords, but at dozens to hundreds of attacks a day, I don‘t want to run any risks!)

I once set up a raspberry pi, didn't finish the entire installation, figured I'd finish where I left off the next morning. Discovered after a few hours of sleep that some ancient port forwarding rule I had set up in my router exposed my rasppi's SSH port to the outside world. Hadn't changed the default root password yet - so obviously it got infested with a buuuunch of malware. So I took it upon me to unplug the damn thing and carefully analyze the content of the SD card.

Turns out, it only got infested with x86 malware (as far as I can tell anyways) - None of which worked on rasppi, obviously. Various malwares attempted to connect, and once they succeeded, they uploaded their payload, and repeatedly tried (without success) to run it.

Also hilarious: Various attacks tried to write to the same file location, resulting in neither of the payloads being successful. `mktemp` is your friend kids.

Don't use passwords. At all.

Pubkeys only.

Upvoted, Why would someone downvote this comment?!

Downvoted by malware worms. They target me too.

When Slammer hit, I had just setup a new SQL Server instance, and was going to head home and finish configuring from there... I didn't get out of the office for 5 minutes before they entire network was flooded with traffic from the worm and completely unusable.

It doesn't help too much, but I generally move even my SSH port and tunnel everything not meant to be public through that, or over a full VPN. It's a shame that you have to do this now.

You need to teach the Docker devs that. By default docker localhost is 0.0.0.0 so when you do docker run container_name -p 80:80 you are mapping to 0.0.0.0. I always explicitly write out the full port map i.e. -p 127.0.0.1:80:80. This is especially problematic if you are running e.g. postgres in Docker. The last thing you want is to have it exposed to the internet. One mitigation is to change the default localhost, you can do this by setting the IP variable in daemon.json to 127.0.0.1.

    /etc/docker/daemon.json (check your OS, for Mac use the GUI)
    {
       "ip" : "127.0.0.1"
    }

Sane, secure-by-default configs are important. Don't let your users go shocked_pikachu.jpeg because of perceived notions of end-user freedom. (See for example how badly a certain famous lisp-based text editor effed up TLS)

> Don't let your users go shocked_pikachu.jpeg because of perceived notions of end-user freedom. (See for example how badly a certain famous lisp-based text editor effed up TLS)

On the other hand, don't lock your users up because of perceived notions of end-user privacy and security. (See for example how badly a certain famous browser named after a metal blocked adblockers)

Not disagreeing with you at all, I'm all for strict TLS defaults, just ranting about the fact that so many restrictions meant for control and domination are marketed as "security", it hurts the public support of actual security. Many people I've read online cannot distinguish Google's push of secure TLS and Google's push of artificial control, one decided to run the website on plaintext HTTP diehardly. And many others still see ALL hardware security modules as evil DRM / mandatory code-signing scheme. How unfortunate it is, even more unfortunate is that their estimation has a better chance to be correct.

> Another lesson is: never run any unsecured webserver/service on the public Internet, never ever, not even for debugging, period. Don't listen 0.0.0.0:80 [...].

This is why Ubuntu's "ease of setup" policy of listening on interfaces with things running on installation is such a poor choice. It may not be terrible on a workstation[1], but it's awful on servers.

[1] Actually, it is.

I opened up port 23 telnet and had it connect immediately to a playable Crowther & Woods' original adventure,

YOU ARE STANDING AT THE END OF A ROAD BEFORE A SMALL BRICK BUILDING. AROUND YOU IS A FOREST. A SMALL STREAM FLOWS OUT OF THE BUILDING AND DOWN A GULLY.

In a year and a half I had logged over a million probes. No humans at all. I would have known because they would have typed something to do with the game or tried to play it. So I must conclude no one is checking the logs of the worms either.

That's a really interesting idea for a test, thanks for sharing that.

Yes it was fun to do, though disappointing in the end. To help throttle attacks and give the right historical ambiance I limited output to 30 characters per second... just like I first played the Fortran 77 version of C&W Adventure on a Texas Instruments Silent 700 with acoustic coupler.

Wow -- you've impressed me! (and yep, I also played adventure on a Silent 700...)

Keep the roll all together on the ground and feed the thermal paper in again to print upside-down on the other margin! Good times. Saw your old calculator video with the 'sonic spiral memory'. I had NO IDEA such a thing existed!

There is great value in the doggedly determined engineering of yesteryear, and there are some among the young who continue to carry the torch. Check this out if you have not seen it,

Apollo Guidance Computer Restoration

https://www.youtube.com/playlist?list=PL-_93BVApb59FWrLZfdli...

In a strange way, it's a bit sad that there aren't more humans out there.

Of course there are. They are all over the place. But the internet is _vast_ so you narrow down your targets using tools before you waste even a second of your time connecting manually.

If he had his telnet service return something that looked interesting (and no, not Crowther & Woods interesting), then he'd have a lot of people connecting.

Or any humans didn't read English?

My team and i run some different honeypot solutions, and we base a lot of them off of cowrie. As pointed out by previous comments, most interactions are not so interesting, except for the fact that many cowrie based honeypots imitating IoT devices have their attackers running a simple script that pulls down a number of second stage binaries, for a variety of cpu architectures.

One downside to running software like cowrie is that generally speaking crawlers like shodan will be able to figure out that you are running a honeypot, and will have you fingerprinted in a hurry.

A better strategy for increasing the cost of an attack is actually implementing something i read about on HN called a ssh tarpit, where one can "hang" an incoming ssh connection indefinitely. A lot of the attacks on honeypots are automated, so instead of having a 3 second attack, one can waste the attackers time for about 30s to 1m on average as these scripts have very generous timeouts (and sometimes no timeouts at all).

> ssh tarpit

Interesting, like slow request attacks in reverse? Is that actually useful for something? It seems like that would just needlessly burn resources on your end. The majority of attacks on my instance seem to have come from other infected routers/devices/etc. that pretty much perform these attacks for free.

Depends on your goals! If you are defending a network, increasing the cost of attack is something we actively try to optimize for. It costs me next to nothing to hold a socket open and send a keep alive every 15 seconds or so, in addition to the extra threat intel from the initial connection.

You might have a point, and maybe i should try to turn these subjective feelings into harder metrics in terms of cost, but we have figured at this point it has a net good. If we slow the scanning down by a magnitude, in my opinion its a good thing!

Ah, sure, if it works why not, I would have expected most scans being too distributed for this to have noticable impact. Thanks for the explanation.

Is there any research on what proportion of services (ssh, telnet, FTP, WordPress) have to be honey pots or tar pits for it to make a difference?

I've been running honey pots or tar pits for years out of a belief that anyone who can has an ethical duty to do so, to slow down the attacks on those who can't.

I havent seen anything terribly relevant, most of the thesis projects i have seen are more interested in creating realistic and believable honeypots for specific protocols, eg RDP.

In my experience, honeypots and tarpits are not the same sort of thing, and fufill different goals. Tarpits get you more utilitarian good, honeypots get you more representative threat intel.

Thank you, and good points. From the view point of increasing the utility of scanning for weak-password ssh ports, a honeypot and a tarpit are both entities the human setting up the scanning would like to avoid. I think that ultimately a human looking for easily-guessed ssh or telnet or whatever passwords would want to avoid tarpits and honeypots equally. They might have to code differently for a tarpit than a honeypot, but the goal would be to detect and avoid instances of both things. What proportion of "something to detect and avoid" would cause a scanner to be less than profitable, or just give up?

To illustrate: I've been giving the people that staff robocaller's "service centers" a hard time for years. I believe that my phone number is in some of their systems as a "bad actor" - I've occasionally heard an audible, computer-generated voice telling the "service rep" that this is a known troublesome number. They also occasionally hang up on me a sentence in to the script. I usually tell them I'm Edward Snowden, but you can call me Ed. That gets a hangup maybe 5% of the time. So giving them a hard time wastes their resources enough that at least a few boiler room/"service centers" put effort towards avoiding me, and the few others like me. What proportion of resource-wasters would it take to make them quit?

I believe you're thinking of "endlessh" - https://nullprogram.com/blog/2019/03/22/

I have endlessh running on my internet-facing server. Here's the current suckers:

p222149-ipngn200203toyamahon.toyama.ocn.ne.jp:45763

14.33.133.188:59757

121.204.198.213:53240

I think all those have stayed connected for some days. But a lot scanners don't fall for it anymore. Since July 17th, I've had 60868 ssh connections, mean 195.817 sec, median 19.124 sec, max 886283.605 sec. The distribution is skewed much shorter than 10 days, however. Half the connections last less than 20 seconds, by eye around 15. 25% of the connections last between 30 and 40 seconds, and about 10% last around 50 seconds.

Wish that this were available 33 years ago...

Mr. Cuckoo's Egg, I presume? Well met!!! I really enjoyed your sleuthing tale in the age of modemy Compuserve and Telenet-with-an-e ... which I also did some exploring on. The telephone NPA plus a couple digits was an explorer's dream. Hope you are doing well in this whacky 21st century.

Well, I daresay you created your own :-D A bit more complex than this one, too...

Didn‘t know you visit HN - I‘m a great fan of your book!

any examples of replay logs on asciinema or even youtube?

Just curious about what these look like

I played back a friend banging on my cowrie setup a while back. Here you go.

https://vimeo.com/345068825/86f8c8f97e

I want to give this a try. Years ago I stumbled upon a similar project written in python which I don't remember the name of, but something about its handshake must have been fishy as most clients disconnected again right away without trying to authenticate. (The initial version string sent by the server looked fine but I wasn't motivated enough to dig any further.)

You might be thinking of kippo? Honeypots are fun to think about and toy with - I just never have any idea what to do with the logs and data?

Write a report or paper or blog or something about it I guess.

The major use case for them nowadays is to help filter out the noise in your alerts. See for example a company such as GreyNoise (https://www.greynoise.io) to help companies focus on real attacks and not just background noise/ automated attacks.

Alternatively, run on your internal network and alert on every attempt....

Yes that's the one!

Toying around was exactly my plan. See if any interesting sessions show up. I have a couple ideas like trying to automatically group or classify them but there are way too many things on my "would be cool to try" todo list already..

I believe kippo is the ancestor of Cowrie.

That's likely an artifact of the client side I'd think. While not most of them, a significant portion of clients exhibited similar behaviour on mine. Disconnects before the handshake were quite common (due to simple port scanning), others seemed to perform the handshake for discovery without actually completing/attempting a login. I've chalked that up to things looking for vulnerable server versions (e.g. home routers).

About the only interesting thing I've seen on mine was login attempts using a single compromised key instead of the old brigade of admin/admin password attempts. Although I guess that might just be some known backdoor of some popular network equipment that I was not aware of at the time.

Reply on news.ycombinator.com