Hacker News Reader

Comment by ianbertolacci | Hacker News Reader

  >>> int("۲۶۷۹")
  2679

Thats pretty cool.

View on HN

Indeed. Seems to be a thing of python 3.

    $ ruby -v
    ruby 2.2.6p396 (2016-11-15 revision 56800) [x86_64-linux-gnu]
    $ ruby -e "puts \"۲۶۷۹\".to_i"
    0

    $ python2.7 -c "print(int(\"۲۶۷۹\"))"
    Traceback (most recent call last):
    File "<string>", line 1, in <module>
    ValueError: invalid literal for int() with base 10: '\xdb\xb2\xdb\xb6\xdb\xb7\xdb\xb9'

    $ python3.5 -c "print(int(\"۲۶۷۹\"))"
    2679

    $ node -v
    v6.9.1
    $ node -p "parseInt(\"۲۶۷۹\")"
    NaN

It works in Python 2, you just have to ensure that your string is unicode. I can't get it to work directly from the command line for some reason, but if you open an interactive Python session it works:

  Python 2.7.12 (default, Dec 14 2016, 13:32:53) 
  [GCC 4.9.3] on linux2
  Type "help", "copyright", "credits" or "license" for more information.
  >>> print(int(u'۲۶۷۹'))
  2679

Hmm, interesting. What's up with the terminal here? The shell is doing something to the text.

    $ python2.7 -c "print(int(u'۲۶۷۹'))"
    Traceback (most recent call last):
    File "<string>", line 1, in <module>
    ValueError: invalid literal for int() with base 10: '\xdb\xb2\xdb\xb6\xdb\xb7\xdb\xb9'
    
    $ python2.7 
    Python 2.7.13 (default, Jan 03 2017, 17:41:54) [GCC] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> print(int(u'۲۶۷۹'))
    2679

Looking at what the string is:

    $ python2.7 -c "print(list(u'۲۶۷۹'))"
    [u'\xdb', u'\xb2', u'\xdb', u'\xb6', u'\xdb', u'\xb7', u'\xdb', u'\xb9']

    $ python2.7 
    Python 2.7.13 (default, Jan 03 2017, 17:41:54) [GCC] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> print(list(u'۲۶۷۹'))
    [u'\u06f2', u'\u06f6', u'\u06f7', u'\u06f9']

DB B2, etc. are the UTF-8 encodings of U+06F2, etc. So Python is seeing mojibake: U+00DB (Û), U+00B2 (²), etc. which are not digits. Well, one of them kinda is, but it’s No (“Number, other”), not Nd (“Number, decimal digit”).

Yeah I get that, but why is that happening?

I’d guess because CPython is assuming the input to -c is ISO-8859-1 (Latin-1) when it decodes it using Py_DecodeLocale():

    main()
      …
      setlocale(LC_ALL, "")
      …
      argv_copy[i] = Py_DecodeLocale(argv[i], NULL)
        …
        mbstowcs() or mbrtowc()
        …
      setlocale(LC_ALL, oldloc)
      …
      Py_Main(argc, argv_copy)

While the REPL’s encoding (sys.stdin.encoding) is set to UTF-8 due to LANG/LC_CTYPE settings. You can get the same error when invoking the REPL as:

    LANG="en_US.iso8859-1" python2.7

So the shell isn’t doing anything to the text—it’s providing UTF-8 bytes in both cases, it’s just that Python is interpreting them differently.

That's what I thought as well when I first found it – once the initial surprise passed, it seems nice to have something just work for anyone using another script. The Unicode consortium has put an enormous amount of work into building that database and it's nice to reuse that to make something friendlier for humans.

Reply on news.ycombinator.com