Hacker News Reader

Comment by clusmore | Hacker News Reader

Honestly I haven't looked into MyPy a huge amount, and haven't used it yet. But from reading the docs on it I feel like it's taking the wrong approach. In my opinion, Python is all about duck typing, so having a static type checker that seems to focus on concrete types feels wrong to me.

Consider the first example,

  def sum_and_stringify(nums: List[int]) -> str:
      return str(sum(nums))

The type here is way more specific than it needs to be. Not only will this work on any sequence type (anything that supports __iter__), the element type only needs to support __add__ and __radd__ (to work against the implicit seed value of sum).

I really feel like in order to get the best of duck typing, you need a good syntax for describing types by usage, not by explicit name. Please let me know if I'm missing something, and this is actually more expressive than the documentation suggests.

View on HN

Well, I like to put that another way -- I think Python is all about protocols (in the Python sense of the word protocol). I think that is what most people mean when they say Python is all about duck typing, and it communicates the concept more precisely.

The concept of 'type' is overloaded with too many meanings these days. It started out as a what-storage-to-allocate and which-instruction-to-generate (int v float) concept (FORTRAN I, variables starting with I..N are ints, others real). Deep class hierarchies muddied the waters by overlaying the idea of whether or not a class implements a particular property or method, and how to locate the property or method that you want.

Python shows us that types and protocols/interfaces are two very different thing. In Python, asking 'isinstance()' is usually a bad idea unless you are doing something very low level, low level as in next stop is a C extension. The missing predicate in Python is 'hasprotocol()'. Abstract base classes can be used to accomplish that: isinstance(foo, SomeABC), but I think the intent could be more transparent with different syntax.

Yeah I agree with you completely, I think we just use different terminology to describe the same thing. It sounds like you use "type" to mean "concrete type", i.e. the thing it actually is, and not the thing it acts like, and then "protocol/interface" to describe the behaviour, the thing we can use it as. As somebody who mostly uses statically-typed languages, I use "type" to mean "the thing my static type checker knows this as", which could be a concrete type or simply an interface, and then obviously I use the term "concrete type" when I need to be more explicit.

I like the idea of a hasprotocol or supportsprotocol predicate, certainly over usage of isinstance, since one of the beauties of duck typing is that it works without requiring the type hierarchy. The tricky part will be in allowing you to define your own protocols, like "has these attributes/methods". An example would be the difference (or lack thereof) between classes and namedtuples.

Interestingly, I think that's what isinstance() actually does when it's called on abstract base classes like collections.abc.Iterable -- it looks to see whether the type supports __iter__ and whatnot. See the abc module and it's implementation here:

* https://docs.python.org/3.4/library/collections.abc.html * https://hg.python.org/cpython/file/3.4/Lib/_collections_abc....

Pasting the relevant snippet from your link:

  def __subclasshook__(cls, C):
      if cls is Iterable:
          if any("__iter__" in B.__dict__ for B in C.__mro__):
              return True
      return NotImplemented

Wow, hadn't seen this before. That's exactly what I mean. If I can define my own, e.g. Addable, which does a similar check for the presence of __add__ (and __radd__) and MyPy uses the subclasshook, then that'd be perfect.

Get _Fluent_Python_ by Ramalho and read chapter 11.

> Python shows us that types and protocols/interfaces are two very different thing. In Python, asking 'isinstance()' is usually a bad idea

Interesting perspective. This would seem to be pretty well described by parametric polymorphism with constraints (Haskell-style polymorphism with typeclasses). In that context, too, we are often well served staying as generic as possible, and asking at runtime about the type of something is a bit of a code smell (and only possible where you have a Typeable constraint available).

Yes. Haskell's type deduction is exactly what I was thinking of as I read the article. You write a function using what you need and it deducts the requirements based on usage, giving you the most general type. I think this is actually very close to compile-time duck-typing. Actually I think if you took the class declarations out of Haskell and removed the instance lines (but left the subsequent implementations), and had the compiler deduce typeclasses for you, you'd get something that felt very much like compile-time duck-typing.

If you want an even closer example, it would be OCaml with its structurally typed object system with inferred types. For example:

    # let f obj x y = (obj#m x)#n y;;
    val f : < m : 'a -> < n : 'b -> 'c; .. >; .. > -> 'a -> 'b -> 'c = <fun>

Here I defined a function taking 3 arguments, called method m on obj, passing x as argument, then called method n on the result of that, passing y as argument.

Note the inferred type signature for this: the first argument is of type <m : 'a -> <n : 'b -> 'c; ..>; ..> - i.e. any object that has a method named m (and possibly some other unspecified members - that's what ".." means), with said method m having a signature that allows us to pass some 'a, and returning another object of type <n : 'b -> 'c; ..> - i.e. something with a method named n that accepts 'b and returns 'c. 'a and 'b' are then used as types of x and y, respectively, and 'c is the the result of f.

So effectively, this is full-fledged duck typing in the vein of Python objects, but captured on type system level.

>Note the inferred type signature for this Wow, that's really powerful. Thanks for showing me this!

Agreed. Haskell and Python have type systems that work. Stuff in between, not so much.

Or: Maybe haskelltyping

There's an open issue discussing the idea of an explicit protocol support somehow in Python: https://github.com/python/typing/issues/11 .

From the mypy docs linked in the article (http://mypy.readthedocs.io/en/latest/duck_type_compatibility...), it seems it supports a small number of built-in duck types currently, and has support for some protocols (http://mypy.readthedocs.io/en/latest/cheat_sheet.html#standa...)

> mypy considers an int object to be valid whenever a float object is expected.

> Use Iterable for generic iterables (anything usable in `for`)

Thus you should be able to get a more broadly correct (though still not 100% if you have your own types implementing __add__ and __radd__) like this:

  from typing import Iterator

  def sum_and_stringify(nums: Iterator[float]) -> str:
      return str(sum(nums))

It also looks like implementing an abstract base class SupportsSum (to check for __add__ and __radd__) may not take a whole lot of work, see `SupportsInt`: https://github.com/python/mypy/blob/d7f6ea617317103534d86686...

http://mypy.readthedocs.io/en/latest/class_basics.html?highl...

I had heard of the support for the hierarchy of some builtins like floats being more general than ints, but this isn't a very satisfactory approach because it's not extensible to user-defined numeric types. Perhaps what I'm suggesting is possible with either the existing classes in abc, or by defining more like them (and ensuring users can define their own). Would be nice to see more focus on this in the examples if that's the case.

> I really feel like in order to get the best of duck typing, you need a good syntax for describing types by usage, not by explicit name.

Not sure if this is of interest to you, but the Nim[1] programming language offers a feature that allows you to do exactly what you described. It is called concepts and is documented here: http://nim-lang.org/docs/manual.html#generics-concepts

As a bonus, Nim is also quite Python-like.

1 - http://nim-lang.org

Yep that sounds like what I mean. Haskell typeclasses, Go interfaces, TypeScript interfaces are further examples. Not Java/C# interfaces though, as they need to be declared at design time, and you can't force existing classes to implement interfaces they haven't already declared themselves as implementing.

Well yes, but

    # optionally: Number = Union[int, float, complex]
    def sum_and_stringify(nums: Iterable[Number]) -> str:
        return str(sum(nums))

also works. Except no one knows what the Number abstract base class is (or even the Iterable abstract base class for that matter)

Yeah but as I said it can still go far more general than just int, float and complex. It could be any arbitrary class defined by somebody else that I could not possibly know about at design time, and so a whitelisting approach of concrete classes is simply not going to cut it. It would be nice to have a way to describe the actual requirements (only that the elements support __add__ and __radd__).

Well, my joke was that numbers.Number exists, and I think supports those operations, but even someone complaining about this topic is unaware that it exists.

I find it a bit easier to treat it as documentation, instead of literal types

mypy will then tell you if you used your documented code wrong, according to your documentation

also sum excludes summing of strings if I remember correctly, so its any object that has an __add__ and __radd__ and isn't a string or subclass of string. which is just a bit insane to try and express on one line!

That's a fair point on the lack of support for strings by sum, but note this is only because the function is hard-coded explicitly to reject them and only because it's not the best way to build strings. By usage, strings should just work.

On the point of documentation, this is potentially a fair point if you consider that you probably wrote this function for a particular usage and you know at design time that you're really only ever going to give it a list of ints (if you strip the types away, the parameter is still called nums which suggests that's what was really intended). In reality, the function is actually much more powerful than that, and if included in a library, many people could use it for many more things than just that.

> In reality, the function is actually much more powerful than that, and if included in a library, many people could use it for many more things than just that.

I think all this says is that design and documentation considerations for functions in distributed libraries may be different than those for functions used locally. Something I certainly agree with, but not really a criticism of the parent point.

Fair point. I realise I sound quite critical in my comments so perhaps I should add that I like the tool itself. The problems that I see are more in the usage as shown in the examples. I am not necessarily suggesting that in all circumstances, the most general types are the best. But I am yet to see this tool used in such a way as to permit the most general types, as would be appropriate for library functions.

> if you strip the types away, the parameter is still called nums which suggests that's what was really intended

Correct and concise documentation is incredibly valuable.

For documentation to stay correct, it must be checked and updated as situations change. Types and assertions can both play this role. Tests can play this role.

Names also play this role - they are not mechanically checked, but "is my use here consistent with the name" is a check any programmer is applying as they work. This actually argues a bit against renaming refactor tools - if I am changing a name, I should be making sure that it squares with existing uses. From this perspective, checking of function parameter namess is weaker than checking of function names, since the names of the parameters typically do not appear at the use site - very different than the assignment of other variables.

Python's implementation of sum doesn't care about types, the function looks something like:

    def sum(iterable):
        r = 0
        for v in iterable:
            r += v
        return v

it always starts with an integer, and since int.__add__(str) returns NotImplemented it tries str.__radd__(int) instead, which raises an AttributeError because str has no method __radd__, so no summing of strings and integers.

Actually Python does explicitly reject strings.

  >>> sum(['hello', ' world'], '')
  TypeError: sum() can't sum strings [use ''.join(seq) instead]

But this is done explicitly against strings. They would _just work_ if not for this restriction, because based on usage they should work.

You can't say it is more specific than it needs to be without knowing the author's intent AND the semantics of all the __add__ methods of all types that have these methods.

Which is for practical purposes impossible.

Semantically, the presence of __add__ and __radd__ is a minimum, not sufficient.

Well sure but that's just the danger of duck typing vs explicit interface implementation, something Python already suffers from. Without MyPy, Python will already let you use this function on all sequences of addable things, not just lists of ints. It would be good if MyPy reflected this as closely as possible, otherwise you're throwing away the advantages of Python's duck-typing.

Edit to add: To be clear, sure the result of calling this on any arbitrary sequence of addables might be semantically nonsense, but that's not the purpose of a type checker. It's only to verify type safety, i.e. you won't get runtime type errors if you give me at least this kind of thing.

Part of the issue is the name, which is generic enough that with the generic implementation, it probably also wants a maximally generic type.

A more constrained type throws away "the advantages of Python's duck-typing", but it also throws away disadvantages, and that can be precisely what you want.

Yes, this is a good point too which I certainly agree with. Sometimes by using the most general type, you actually permit more than you intended and you might restrict yourself from making refactorings to the function because it would restrict the types.

> might be semantically nonsense, but that's not the purpose of a type checker. It's only to verify type safety, i.e. you won't get runtime type errors if you give me at least this kind of thing.

We either disagree about the purpose of a type checker, or we disagree about what can constitute a "runtime type error". If a type checker can catch inconsistencies that would lead to my returning "semantic nonsense", at low enough cost, I absolutely want it to. Obviously it won't be able to catch all such inconsistencies in all cases.

I understand your point, and perhaps we don't actually disagree on this. As an example, in Haskell (which you have shown in another comment that you are familiar with), this function would perhaps best be written as

  sum_and_stringify :: (Show a, Monoid a, Foldable t) => t a -> String
  sum_and_stringify = show . foldl mappend mempty

In this case there are additional rules (semantics, though not in the language) which we associate with monoids like associativity that don't apply to all "addables" (e.g. strings), that might make us feel more comfortable being so general. I think really it's just that we have more expressive language to describe our expectations. Though there's nothing stopping you from making String an instance of Monoid, or most any type for that matter, and producing nonsense. It's not a type error, so I say it's (in this case) not the job of the type system. But there are more expressive type systems that could let you guarantee these additional properties like associativity and turn nonsense into type errors, and I for one am all for that.

In the case of Python, though, coming from no static type system or verification at all, I would only ask of this tool to turn runtime checks (type errors caused by arguments lacking __iter__ or __add__) into static checks. Anything on top of that would be fantastic, I agree with you on that, but I wouldn't ask for or expect it.

Finally, thanks for all your comments - I appreciate the discussion.

> Anything on top of that would be fantastic, I agree with you on that, but I wouldn't ask for or expect it.

Ah, we probably do agree more than it seemed. "The MyPy team has done its job even if that's not possible/reasonable" is a statement I'd endorse - especially in the short-to-medium term.

Discussing the Haskell...

String concatenation actually does associate; in fact String does form a Monoid over concatenation, and has an instance visible from Prelude - specifically the list instance, because Haskell strings are just `[Char]`.

The only really "nonsense" thing about using a string here that I see is that we wind up with quotes and escaping in the result when we `show` the string. If the purpose of this is "gather results and format for logging", that might be fine.

Numeric types don't have an instance of Monoid, because there are two to pick. You can pick them with newtypes, but here that's a bit awkward (`sum_and_stringify . map Sum`? feels redundant), and then the Show will give you something like "Sum { getSum = 3 }", which is probably not what you want either...

Using Num rather than Monoid probably has better ergonomics, and may be closer to intent anyway. That is to say:

    sum_and_stringify :: (Show a, Num a, Foldable t) => t a -> String
    sum_and_stringify = show . foldl (+) 0

Of course, we have a `sum` defined (basically, perhaps literally) `foldl (+) 0` so why even pull this out into a function? `(show . sum)` is actually substantially shorter than `sum_and_stringify` and no less clear.

I think the biggest issue here is that the example is so devoid of context (probably necessarily) that it's hard to say what good style would be. In actual code, a function with that name should probably have a very general signature - but we likely shouldn't be writing a function with that name. Why are we wanting to "sum and stringify" as one action in this way? Is that something that might change? Will the uses want to change together or are we just Huffman coding? With a domain-relevant name, we'd have a sense of what ways it makes sense to restrict the types.

> Finally, thanks for all your comments - I appreciate the discussion.

Likewise!

>String concatenation actually does associate; Oh my goodness, my brain exploded apparently because I meant to say commute, although commutation is usually not something to be expected of monoids.

>Using Num rather than Monoid probably has better ergonomics, and may be closer to intent anyway. You're probably right on this point, in all honesty. I think I'm over-generalising the original intent.

>With a domain-relevant name, we'd have a sense of what ways it makes sense to restrict the types Yes, I think this is the classic helper function vs library function quandary. If you wrote it to help yourself right now, it's often good to be very specific with the name and signature. That way if later you need to add more functionality you can without changing the name or restricting the signature. If you wrote it as a library function, it's good to be generic, to let as many people use it as possible. And in this case there should never be a need to change the function, it does what it does. If you need something else you just create something else.

This resonates with some things I've found myself saying recently around misunderstandings of the DRY principle.

It's so catchy an acronym, and so seemingly self-explanatory, a lot of people don't realize it was originally framed in terms of knowledge, not syntax. So if syntax happens to be the same in N places, but in each of those places it is the way it is for a different reason, consolidating may not be "DRYing up the code". On the other hand, if I need to say "there's a text field here" in my markup and also in my logic, that could be more DRY even though the syntax looks nothing alike.

Tying this back here, we find as a corollary: when factoring out code we can ask ourselves "what piece of knowledge does this represent?" If it is knowledge about the language (this is a good way to sort things) it probably goes in a library, and should probably be very general. If it is knowledge about your domain (this is how we escape HTML), it probably goes in a domain-specific library, and it should probably be somewhat less general. If it is knowledge about your application, it probably goes in your application or an application-specific library, and should probably be pretty specific.

protocols (think Java interfaces) are becoming a thing for the type checking, so that's what you can use in places like this in the future. something like `Iterable[Addable]`

This is great provided that Addable is not just something hardcoded to know about builtins like int and float. If I create my own class that quacks like an Addable, then it needs to be supported. That's the essence of duck-typing.

I suggest you go and have a long hard look at Crystal. From your comments I am getting the feeling you could appreciate it.

Thanks for the suggestion. For the record, I'm a fairly happy C# for work, Haskell for hobby programmer, using Python as my go-to for quick-and-dirty scripts. My comments here are mostly based on my own thoughts on the static vs dynamic debate, in this case specifically how to keep the advantages of dynamic languages whilst absorbing some of the benefits of static checking. They have nothing to do with the best usage of MyPy in production-grade Python projects, of which I have zero experience.

Reply on news.ycombinator.com