Hacker News Reader

Hacker News

Comment by adpirz | Hacker News Reader

Full thread

adpirz

Interpreted as: [0xf or x in (1,2,3)]

0xf = Hex for 15

Expression is evaluated as a boolean

View on HN

Here's the AST:

    >>> ast.dump(ast.parse('[0xfor x in (1,2,3)]'))
    "Module(body=[Expr(value=List(elts=[BoolOp(op=Or(), values=[Num(n=15), Compare(left=Name(id='x', ctx=Load()), ops=[In()], comparators=[Tuple(elts=[Num(n=1), Num(n=2), Num(n=3)], ctx=Load())])])], ctx=Load()))])"

Shouldn't this be considered a bug in Python? Why does it even try to evaluate 0xfor without the space? Trying a few other things..

* 0xfor1 evaluates.

* 1or 2 evaluates.

* 1or2 doesn't.

* ''or'foo' evaluates.

This is gross.

Not by design, fortunately: https://bugs.python.org/issue43833

That hasn't been confirmed. Are we sure that it's not an inherent ambiguity in the grammar?

That's not totally clear. A bug being filed doesn't mean it's accepted. And this has been (ab)used for quite some time in various python codegolf. See https://codegolf.stackexchange.com/a/56 from 2011.

It is 100% by design. It's even documented.

https://docs.python.org/3/reference/lexical_analysis.html#wh...

This is not a bug.

The cited documentation says:

> Whitespace is needed between two tokens only if their concatenation could otherwise be interpreted as a different token (e.g., ab is one token, but a b is two tokens).

The two tokens in this case are "0xf" and "or". Their concatenation cannot be interpreted as a different token, because "0xfor" is not a valid token. Therefore, if I'm reading the rule correctly, whitespace is needed in this case.

"0xffor" is another interesting case. It's also not a valid token, but it could be interpreted as two tokens in two different ways: "0xf" "for" or "0xff" "or". (Python does the latter. I presume it uses something like C's "maximal munch" rule.)

Your conclusion is the exact opposite of what the documentation explicitly states.

> Whitespace is needed between two tokens only if their concatenation could otherwise be interpreted as a different token (e.g., ab is one token, but a b is two tokens).

Because the concatenation of "0xf" and "or" can't be interpreted as a different token, the whitespace is not needed.

You're right, and I was wrong.

I dislike the rule, and I strongly think that "0xfor" should require whitespace between "0xf" and "or" (I'm sure that influenced my reading), but you're right about what the rule says.

(Apparently I can't edit my previous comment.)

Probably a lexer bug. "foo"or should never be processed as "foo" and token OR

why not? "or" is an operator, like "+", "foo"+"bar" should be valid. Why have special inconsistent case for "or".

You know what? You’re right.

I guess it’s an operator token after all.

That’s the normal way lexers work, given “tight” token definitions. They continue adding to the current token until an invalid (for the current token type) character is reached, and then begin parsing a new token starting with the “invalid” (but now valid for the next token) character (or the next non-whitespace character).

“1or2” is lexed into “1” (integer) followed by “or2” (identifier), which is valid on the lexer level but then fails on the grammar level.

The lexer unfortunately is a greedy token matcher. As soon as the 0xf "made sense" to it, and 0xfo did not - it did the same thing it would do in the case of something like 0xf+3. Except the + was an `or` in this case which is kosher. There is an idempotent step you can take where extra spaces are added before the AST is formed to make this sort of thing easier. The good news is, with a decent lint / format flow - these sorts are easy to catch.

It is not strictly speaking a bug, since it works as intended. But it is clearly a counter-intuitive behavior and could be improved. Making 0xfor a syntax error would definitely be an improvement.

But requiring whitespace between all tokens is not an acceptable solution, since "2+2" should work. Always equiring whitespace between alphanumerical characters in different tokens would make sense.

Why is it parsed like that and not as invalid integer literal "0xfor"?

edit: 'cause it's a bug! https://bugs.python.org/issue43833

For pretty much the same reason that "1+2" is not parsed as an invalid integer.

"1+2" is a valid expression consisting of the three tokens "1", "+", and "2". It's permitted to omit whitespace between those tokens.

The question is whether "0xfor" is a valid way to represent the two tokens "0xf", "or". My reading of the rules is that it isn't, and that whitespace is required.

I read the rules incorrectly. It appears that "0xfor" is a valid way to represent the two tokens "0xf" and "or".

The rule, which I initially misinterpreted, is:

> Whitespace is needed between two tokens only if their concatenation could otherwise be interpreted as a different token (e.g., ab is one token, but a b is two tokens).

Since "0xfor" cannot be interpreted as a different (single) token, whitespace is not needed.

(I'm not a fan of this particular consequence of that rule, and I suspect it was not intended, but it says what it says.)

This illustrates why lexer tokens should preferably not be defined as exactly what the language allows. Instead the internal lexer definition should include invalid tokens (like “0xfor”) that are only sorted out in a later step (in this case, when actually converting it to an integer value).

Yes, that is basically how C works, though the C standard describes it in a weird way. There is the notion of a “preprocessor number” which starts with a digit and allows any characters that can appear in a number or identifier. At a later step in parsing it is converted to a strict number, which would fail in a case like 0xfor.

Just to elaborate, it short circuits, so x need not be defined.

    [sielicki@dogfruit ~]$ python3.10 -c 'print(0xf or some_undefined_variable in some_undefined_set)'
    15

This is the kinda thing that helps illustrate just how powerfully dynamic Python is. Almost nothing is statically bound. Super cool if you want to write code that inspects some dynamic state and has special behavior. But the cost is that there's very little static checking capability.

This code "should" result in a NameError or something similar. But resolving that identifier to a name won't happen until we decide to evaluate the right-hand side of the `or`.

Thankfully, most python static checkers make some mostly-sane assumptions and will flag this code as an error, despite the subtle possibility of legally creating this name using some other Python/CPython features.

Type annotations are a great idea but in practice I don't see them used much. Should be great for dev teams working on bigger Python projects.

A bit of a miss for a language that puts whitespace central in the syntax.

"2+2" and "2 + 2" is the same in Python, so whitespace is not significant between otherwise unambigous tokens in Python.

Whitespace in Python is significant to indicate statements and block.

I initially assumed the missing space was a typo in the headline, so I tried [0x for x in (1,2,3)] and got "invalid hexadecimal literal" as I would have expected. Took me a few minutes to realize the typo was intentional.

I still can't figure out what the author was trying to do when he stumbled onto that, though...

My guess is that they were trying to do soemthing like [x for x in some_list], but had some unfortunate typos.

Yeah maybe he was converting a tuple to a list, but that still seems like a weird way to do it.

Think of it as a simplified version to demonstrate the issue.

This is the kind of thing you could end up with from an automated test case reducer. Start with a typo that causes odd behavior (which might be interpreted as a CPython 'bug') and use something like creduce to whittle it down to the minimal case.

Anyone know why it's legal to use logic operators like that in Python?

That is, 0or, 0xor, 0and,etc. in my view, this should clearly be a syntax error.

edit: as noted by the parent comment, 0xor != 0 xor, but 0x or. But this seems to hold true for all data types.

Other languages don't enforce whitespace around logical operators. Like `0xf||1+2` is perfectly valid javascript.

I can see how this could slip by, it's possible that original python used a non-alphabetical token as the logical operator, then it was swapped out at some later time for or and and.

There's no ambiguity though, right? [0x for x in (1,2,3)] is nonsense (one typo off from something legitimate, but nonsense regardless). I can't think of another way someone might expect this to parse.

I would have expected the english boolean operators to require whitespace to parse correctly, personally. "0xf||x" makes sense. "0xforx" makes no sense.

"0xforx" would be parsed as "0xf orx" which would be a syntax error.

Yeah, I realized after my edit window had expired. Should have been "0xf|| x" and "0xfor x".

If you want confusion with a valid expression:

    [0x1for x in (1,2,3)]

looks like it should parse as

    [0x1 for x in (1,2,3)]

but instead parses as

    [0x1f or x in (1,2,3)]

This is better than the original post.

Reply on news.ycombinator.com