unicodedammit

You're being unnecessarily argumentative. The web is filled with badly-encoded/re-encoded/mixed-encoded text. One of the worst offenders is Microsoft Outlook, which sends emails using local 8-bit windows codepage by default, instead of utf-8. Pass that through several mail gateways, display those messages on some php forum, and you get yourself a bad mess that cannot be decoded successfully by any python codec. chardet is useless in that case - there is no 'valid' encoding per se.

This library is taking the only possible approach, which is to segment the text and try to convert each segment into its most probable unicode representation. It seems to cover a larger number of encoding mixups compared to other libraries, that's great!

Comment to rspeer: in fix_text_segment(), I would limit the number of recursive passes on the text to 5-10. Right now it's using 'while True', which might take a very long time to converge on corrupted/binary data.

from __init__.guess_bytes(): "This is not a magic bullet. If the bytes are coming from some MySQL database with the "character set" set to ISO Elbonian, this won't figure it out. Perhaps more relevantly, this currently doesn't try East Asian encodings."

The world is a very large place, there are many codepages in use besides latin-1 and "ISO Elbonian". All central european contries use latin-2 (1250), or cyrillic codepage (1251). Since they are all single-byte codepages, they cannot be detected by try: convert() catch: try_another_codepage() and must be distinguished statistically. LTR/RTL language and asian encoding detection is even worse. https://en.wikipedia.org/wiki/Code_page https://en.wikipedia.org/wiki/Windows_code_pages

Another python Unicode conversion module which is slightly less US/English-centric: https://github.com/buriy/python-readability

Very common problem in web scraping, for example forum site might contain a mix of MacRoman, Windows codepage and various european codepages in a single page (yes, even in 2014!). Seems like a more advanced version of UnicodeDammit module ( http://www.crummy.com/software/BeautifulSoup/bs4/doc/#unicod... ).

Note: this module, like UnicodeDammit, is very US/English-centric, and is practically useless for worldwide web scraping. For non-english pages, it is necessary to statistically estimate the codepage and language of each page segment, and then try to normalize each segment to unicode.