Hacker News Reader

Comment by gbin | Hacker News Reader

The optimization is awesome but IMHO the key ordering "benefit" in the implementation but not the spec is a so-so move: It can cause some future bugs in the code assuming it is there. Some languages like Go added some key ordering randomization to maps to be sure to avoid people counting on any specific key order.

View on HN

It's not so-so, it's a terrible idea. At a minimum, the implementation should cough up keys in (say) reverse order, or some cheap to implement different order than insertion order.

Two years down the line, the amount of code that depends on insertion order will make it impossible to do further optimization.

[Many years ago I wrote a platform with an unintended ordering scheme in its dictionary. The documentation stated in bold that order was not deterministic and could change. Users started reporting "bugs" when I changed the order. That sucked.]

Ordered kwargs is in a PEP[0] that is in "Final" status, meaning it's been accepted and implemented and is making its way into the language proper.

[0]: https://www.python.org/dev/peps/pep-0468/

EDIT: On rereading, I see that the parent and GP aren't specifically talking about the kwarg-ordering benefit, but the general dict-ordering benefit. I would be surprised if dict did not become a synonym for OrderedDict in the spec in the near future.

The only potential disadvantage of dict==odict is performance, as anyone relying on the current arbitrarily-ordered dict behavior by definition isn't worried about the underlying implementation moving to a deterministic-by-insertion-order dictionary.

Still a terrible idea. Speaking from hard experience.

I've never run into a case where the difference in performance between Dict and OrderedDict was important.

I have dealt with legacy code (especially code that needs to interact with JavaScript libraries) where strict-ordering was critical and it was great that at the time I was using Ruby 1.9 in which dicts are explicitly ordered.

I agree that waffling about it is bad, but I think ordered is a better default than unordered. All this stuff about deliberately trying to shuffle order to catch bugs just says to me that most code is a lot simpler to get right if dicts are ordered-by-default.

I've seen some python 2 code "made use" of the deterministic ordering of dict. Migrating to python 3 sucked, it took a while to realize this was the reason for some failing tests.

It seems strange that that's what you'd be worried about. Your users found bugs in their own code. That's GOOD, isn't it? Does it matter that they blame you?

Like if I'm using pointers after freeing them and just happen to get usable data back... then you go an change it so that memory is reallocated and my code breaks. Yeah, it is a bug. In my code. It's pretty clear that I should not have been doing that. If I'm to stubborn or stupid to figure that out, then I'm really just a shit programmer.

As long as the language specification is updated to specify ordered keys in dicts then this is good. The big complaint about key ordering I remember hearing is that it was basically a side effect of the CPython implementation (back in the 2.X days), but not part of the spec.

If the spec won't be updated to specify that dict keys must be ordered, then I agree with you.

EDIT: ganduG clarified below that this is just an implementation change. No bueno.

Many, many people won't read the spec. And their code will break when the implementation changes.

And what's worse, this kind of issue is hard to detect. At least with unicode the code was crashing most of the time.

If the ordering of the keys is not in the spec, a programmer should not assume the ordering of the keys. If none of the training material says or implies that the keys will always be ordered, why would a programmer?

The new and improved algorithm happens to preserve ordering. Future algorithms might not. The dictionary data structure does not specify order preservation so that, in the future, a better algorithm that doesn't have this same side effect could be used.

> If none of the training material says or implies that the keys will always be ordered, why would a programmer?

Because people make mistakes.

Perhaps "should" is the better word. That is, a programmer should not assume the keys will be ordered.

Programmers should not try to access memory beyond the end of an array. They should not attempt to add ints and strings. They should not give a function a float when it expects a struct.

But they do. So we try to design languages in such a way as to minimise the likelihood of these mistakes going unnoticed.

Designing a language to minimize the likelihood of these mistakes is a motivation for language design, but usually far from the primary.

"Programmers should not try to access memory beyond the end of an array" ^ Right, and when they do, it's called a logical bug, something to be fixed. A programmer relying on the incidental but unspecified behavior of a function would be making a similar mistake.

I agree.

But, we learned from C that the possibility of trying to access arbitrary parts of memory means that programs will end up trying to access parts they shouldn't. So, higher level languages like Python solve the problem by having memory-safe types like lists, which preclude the possibility of making this mistake.

Similarly, if we randomise the ordering of keys in the dict then it's impossible for a programmer to make the mistake (consciously or unconsciously) of relying on ordering.

At the cost of an additional processing step to scramble the keys. The balance between safeguarding ignorant or overactive programmers against the performance of a critical builtin type is not going to tip in that direction.

The whole reason we can even have this discussion is because a new algorithm was developed to give us the increased performance in the first place.

> Programmers should not try to access memory beyond the end of an array. They should not attempt to add ints and strings.

In Python, both of these raise an error to make it very explicit and obvious to the programmer, while this implicitly ordered keys bit can't.

Given how much Python I learned just by opening the interpreter and seeing what happened, I can definitely go along with the idea that this is bad.

> why would a programmer

How many programmers have you worked with? (People will assume all sorts of things that are not true if it makes their job slightly easier, or even harder)

why would a programmer? Because the programmer would just run the code and check what different methods return as result. Then if it looks like the ordered keys, said programmer will rely on this behavior.

For instance, I'm writing some javascript recently. I try a little array in the console, if Array.keys() returns the orders keys, I will assume it is, and would check the docs (where?) only if I had a problem. Something unstable should not look stable. I agree with the Golang way, make it look like what it is by the specs.

> check the docs (where?)

Thankfully that's more obvious in Python.

I have trouble understanding a programmer who takes a couple samples of a method's behavior and feels confident enough to trust that as the actual behavior of the function. At least guard you're flippant assumption with a check or something!

The problem is not to add some checks (asserts? No, just kidding), the problem is that you're quickly writing a little thing for a demo out an experiment, and faster than you'd expect these ten lines of innocuous code have grown in a gigantic ball of mud upon which many businesses are built.

To me the less worse way to handle this is a softened TDD, where a consequent part of the code is covered (described, checked, structured) by a suite of reasonably atomic automated tests.

"try a little array in the console, if Array.keys() returns the orders keys, I will assume it is"

That may work for some toy script you write for your own use, but must be avoided if you intend to engineer a program.

Certainly, that kind of thinking will lead to very bad results if you ever do low-level multi-threaded programming or try to write portable code.

For multi-threaded programming, "is this call thread-safe?" is not something answered in a test, and certainly not in a quick test.

For portable programming, you simply cannot run the test on all possible systems.

In general, when you program like that, you will not only rely on implementation details that may change (for example, this may change for arrays with lots of entries) but also forget to handle error conditions.

> a programmer should not assume the ordering of the keys

The Python community has many inexperienced programmers, even non-technical people. Particularly, it's so beginner friendly by making it very, very easy to get started with the REPL, etc. A lot of these programmers aren't even going to look at the docs and even if they do, might not understand them.

It is very easy to inadvertently add dependencies on key order. For example, if you are generating HTML, you might have something like:

    print("<a " + " ".join(k+"=\""+v+"\"" for (k,v) in d.items()) + "/>")

And d would be a dict of attributes for the anchor element. The code will produce different output depending on computer which is not what you want because it can cause subtle problems for other systems expecting a certain output.

Several compiler bugs have been of this kind so it is not only newbie programmers that make these mistakes.

How does HTML attribute order change output?

The code will produce a string which will be different depending on unpredictable factors. Such as the memory address of the 'd' object.

But how will that affect output? I'm not aware of any scenario where HTML attribute order matters.

It shouldn't affect the way a browser renders the HTML. But there are many possible scenarios in which the order is relevant for some other reason.

Suppose you have another system monitoring the web page generated by the above code. Like a cdn or something. That system would think the web page is constantly updated as the code of the HTML page would be different each time it accessed it, leading to lots of redundant work if it has to update caches, check outbound links and so on.

If a caching thing doesn't understand that order of HTML attributes doesn't matter, I don't want to be using that thing.

This was my thought as well.

I haven't really followed the discussion around the decision, so perhaps I am missing something, but my initial feeling is that either ordering should be required by the spec and implemented, or the ordering should be randomised.

But, that is the outcome of about 1 minute's thought :)

Reply on news.ycombinator.com