Comment by BiteCode_dev | Hacker News Reader

Comment by BiteCode_dev | Hacker News Reader

Parent Full thread

Even before talking about the typing system itself, annotations existed by themselves and are not attached to any behavior in Python: they simply hold metadata.

When Python 3 came out, annotations were indeed added to the language, but no use case was decided for them. It was an experiment to see what the community would do with them instead of forcing a use case base on what the core devs would think would be useful.

In order to allow this, annotations where permitted to be any legal python expression.

Nowadays, type hints have won: annotations are widely used to add static type checking in Python. But the way the language treat them have not changed much appart from becomming lazy. While mypy cannot understand dynamic Python expressions in them, those are nevertheless legal syntax, and will result in correct metadata at runtime.

Therefore, were the typing system not turing complete, the annotations would have been anyway by nature of being basically an exec().

That’s not at all what’s going on here. While the Python interpreter permits and executes arbitrary expressions as annotations, Python static type checkers (mypy, Pytype, Pyre, Pyright) do not. This isn’t just an implementation limitation; the typing specifications (PEP 484 and successors) don’t allow it. For example, although these two definitions evaluate to the same result at runtime:

    def f() -> int: return 1
    def f() -> (lambda: int)(): return 1

only the first is correctly typed by PEP 484. mypy rejects the second with “error: Invalid type comment or annotation”.

That’s the whole idea of static type checking: in return for restricting yourself to a controlled subset of the language that isn’t “too dynamic”, you gain the ability to type-check your code without running it.

The result of this paper is that one can nonetheless abuse subtyping to perform Turing-complete computation in the type checker. The computation doesn’t happen when your code is executed (annotations have no effect on the Python interpreter at runtime unless your program chooses to inspect them), it happens when you run mypy on it.

This is generally considered undesirable, because we want type checking to be decidable, and Turing-complete computation is not.

> That’s the whole idea of static type checking: in return for restricting yourself to a controlled subset of the language that isn’t “too dynamic”, you gain the ability to type-check your code without running it.

Eh, you are mixing up distinctions.

All that's required of _static_ typing is that your static type specification finishes running before you start your program.

Whether that static type specification language is Turing complete or not is a completely separate issue. And, yes, you are right that it's generally useful for that language to be decidable. But see eg Haskell's UndecidableInstances.

That's why my comment litterally starts with "even before talking about the typing system itself".

I’m sure that you understood the important differences, but there are likely many readers who didn’t (many more programmers are familiar with Python than with mypy). When you write “even if we don’t have P, we’d have Q anyway”, the reader is left with the impression that P and Q are somehow interchangeable or equivalent or comparable—that P is unimportant in the presence of Q. I think it’s instructive to point out that this impression would be mistaken, and the paper’s result of P is interesting and consequential.

What other use cases did the community come up with?

Dependency injections, foreign-language bridges, data validation and parsing, etc.

But thanks to how we can introspect annotations, we could have a specialized syntax for types that would be also usable for those things, while the other way around would be harder.

So it's nice we went the type hints road, because you can now define types, and yet have wonderful libs like pydantic, fastapi, typer, taichi and so on.

Pydantic typing differs from mypy typing. Optional[T] means something different under the two.

I designed typedload before pydantic was a thing and I made it with mypy in mind already, and with the idea of using the standard library, not defining my own dataclass.

Unfortunately mypy has a limitation so you can't make a function that takes Type[T] and returns T. You can define it but it won't work for all types. There is still discussion on how to provide this feature.

Yeah this ones is really annoying with default values.

Cool, any examples of libraries that use it for dependency injection or FFI or validation?

Dependency-injector uses it for exactly this purpose

    @inject
    def main(service: Service = Provide[Container.service]) -> None:
        ...

FastAPI also does a similar sort of trick, for DI and also request parsing.

https://python-dependency-injector.ets-labs.org/introduction...

django-ninja and fastapi does dep injection, parsing and validation based on annotations.

taichi does ffi with their own dsl, cython with c.

I you haven't tried pydantic or sister projects (django-ninja, fastapi and typer), this is a good place to start. They bring a breath of fresh air and are quite fun to use.

Specifically it's the underlying pydantic that does validation based on annotations

https://pydantic-docs.helpmanual.io/

Indeed.

The dep injection is fastapi and typer code though, and quite tied to it. So it's worth mentioning that somebody is attempting (quite successfully from the look of it) to create a generic lib of dep injection using annotation named DI, inspired by those libs: https://github.com/adriangb/di

I always find it very funny that fastapi has that name when it relies on pydantic that is one of the slowest libraries in that space :D

It's not one of those that you listed, but the @dataclass decorator uses type hints to auto-generate classes and class constructors (__init__ method) for you.

An example:

    @dataclass
    class HackerNewsPost:
        title: str
        author: str
        votes: int = 0

This will generate a constructor that looks like this:

    def __init__(title: str, author: str, votes: int = 0):
        self.title = title
        self.author = author
        self.votes = votes

This is pretty useful, writing __init__ boilerplate has been a common thing in my experience using Python, and it makes code more compact and readable.

It can do some other things too!

https://docs.python.org/3/library/dataclasses.html

For FFI, there is a pure python mode in cython:

  https://cython.readthedocs.io/en/latest/src/tutorial/external.html

I used them at runtime to define the types for my (absolutely stupid) library that takes a Python function and replaces its contents with a C call.

You define a Python function interface but populate it with a string of C code.

Wait, so like this?

    def add(x: int, y: int) -> int:
        """ // c code
            int x, y
            return x + y
        """

And it compiles it and turns it into a C extension?

That's...horrifying yet amazing.

That might be a worthwhile addition to the suggestion at https://github.com/faster-cpython/ideas/issues/453

Combined with the cog (https://nedbatchelder.com/code/cog) to have the c declaration in a proper separate c file, but included in the docstring, you could have the best of both worlds.

That's a neat hack!

There's a library for simulating the Excel calculation engine in Python. It uses annotations to hint functions, but according to Excel's type system, not Python's. Then at runtime, those hints are used to do conversion and promotion based on how Excel does it, so you can do e.g. the equivalent of "1" + 1 and get 2.

It's either xlcalculator or pycel, one of the two. I forgot which one does it that way.

Looks like it's xlcalculator:

https://github.com/bradbase/xlcalculator#addingregistering-e...

Things like pydantic and fastapi where the type annotations are used for run time argument validation and interface generation.

Does this still work with Python 3.10 changes removing runtime evaluation of type annotations? I seem to recall this screwed me when I was writing something for runtime usage of types.

They postponed that change.

It remained enabled with a future and while it kinda works, it doesn't work in all cases. Non top level definitions can never be retrieved basically.

There was a heated debate because it broke things at the time, but now it works, yes.

Oh it looks like they actually pushed back the PEP 563 changes..

From this thread [0] it does not look open and shut that it will work once these changes go into effect.

[0]: https://github.com/pydantic/pydantic/issues/2678

we still don't know what's going to happen with type hints in future though, but I'm sure things won't break as they might have broken if they changed the default behaviour

Creating argument parsers for CLIs: https://pypi.org/project/argtyped/

I've seen similar (in-house) to generate HTML forms also.

I've created strawberry graphql, it uses type hints to generate GraphQL schemas

Reply on news.ycombinator.com