Hacker News Reader

Comment by jffry | Hacker News Reader

In fact, the Unicode consortium provides a report and extensive list of "confusable" symbols, which you could use alongside Unicode normalization tables to map adversarial back into more ASCII-equivalent text before running it through anti-spam mechanisms that are interested in the content of the message.

https://www.unicode.org/reports/tr39/

View on HN

I built a Python library for finding strings obfuscated this way. Was critical when moderating our telegram channel before an ICO.

https://github.com/wanderingstan/Confusables

E.g. "𝓗℮𝐥1೦" would match "Hello"

I should have had better eyes while searching, could find this and saved some hours.

I only learned about it myself after spending too long building my own half-baked version. I think it's in pretty opaque language that makes it hard to find even if you know what you want.

Maybe somebody else on here will see it and learn about it before they need it, and at least you still have a new tool to reach for in the future.

You should post this to Show HN. Also you have a typo on your README ("characgters")

Thanks, I've fixed the typo! It was such a simple project, hardly seems worthy of a "Show HN".

I've seen crazier things get to #1

Test for the library: would it catch that that typo still refers to characters?

The one built-in to Python will get you most of the way there:

    >>> import unicodedata
    >>> unicodedata.normalize('NFKD', '𝓗℮𝐥1೦𝗵𝗲𝗹𝗹𝗼')
    'H℮l1೦hello'

Obviously it isn't going to remap leetspeak characters like 1 -> l but it covers a lot of cases.

Obviously you're saying it doesn't cover everything, but a big thing it's not going to catch beyond leetspeak-type situations is the kinds of thing you (used to) see in internationalized domain spoofing: legitimate non-Latin-script letters that just look the same or nearly the same.

NFKC/NFKD will handle "this is another form of the Latin letter A" type stuff but not "Cyrillic A looks like Latin A."

Thank you for making this!

What I don’t understand is who sees spam that looks like an old school ransom note and thinks ‘seems legit’, and then actually buys something?

There should be a 'bad symbols' list you can block like you do expletives. There's surely zero need to support that kind of thing in comments.

One term used to describe this is homoglyph, as in homoglyph attacks for phishing.

Back in 2015 I did some work just using simple bitmap rendering plus OCR to find text that looked like a small selection of known words. It was actually reasonably effective.

Reply on news.ycombinator.com