Why if TYPE_CHECKING? | Hacker News Reader

Why if TYPE_CHECKING?

vickiboykis.com

28 pointsBerislavLopac72 comments

You should also add this:

    from __future__ import annotations

Then you will not get this error (NameError: name 'Sequence' is not defined) because it will not need the reference to `Sequence` for the annotation but the annotation is simply a string.

Using `if TYPE_CHECKING` is also useful when you want to speed up the module load time by not importing unnecessary modules (unnecessary at module load time).

And then, also to resolve import cycles, where you still want to annotate the function argument types.

One problem though: If this is a type/object which is not needed for module load time (there it's just used to annotate some function arg type), but which will be needed inside the function, type checkers and IDEs (e.g. PyCharm) will not show any problem, but then at runtime, you will the NameError once it tries to use the type/object.

Example module:

    from __future__ import annotations

    from typing import TYPE_CHECKING

    if TYPE_CHECKING:
        import torch

    def agi_model(query: torch.Tensor) -> torch.Tensor:
        torch.setup_secret_agi()
        ...

This module will load just fine, because `torch` is only needed for type checking at module load time. However, once you call this `agi_model` function, it will fail because `torch` is actually unknown at runtime. However, no IDE will show any problem with this code. (Edit: Eclipse/PyDev handles it correctly now, see answer below.)

Then you would add another `import torch` inside that `agi_model` function. However, then the IDE might even complain that this hides the outer scope `torch`.

> However, no IDE will show any problem with this code.

That's not entirely True... I just checked it here and Eclipse/PyDev does flag this correctly -- since version 11.0.0 ;)

See: https://www.pydev.org/history_pydev.html (Imports found inside a typing.TYPE_CHECKING will be considered undefined if the scope that uses it requires it to be available when not type-checking).

Adding that will break every single runtime typechecking module. Which is why in the end it was never included in python proper.

If you don't do runtime typechecking in the entire program, sure.

It kind of works, you just have to use `typing.get_annotations(func)` instead of `func.__annotations__`. Although if you aren't importing the types used in your annotations that isn't going to work... I would guess that's rare in practice.

If you define a class inside another class or a function, get_annotations() will never be able to resolve it and just return a string, which is not very useful.

I rarely butt into practices for languages that I don’t use.[1] But having a flag for “are we type-checking” is a, well, huge red flag.

[1] Except some light scripting

Yeah, the article explains in depth why we are this stage with Python ...

There is always a history. Which does not imply a good reason.

It's a work in progress, the article didn't say the current situation is ideal, nor am I.

Yeah you’re not saying anything.

doesn't Typescript have something similar?

``` import type {foo} from ... ```

Not really. `import type` just means that this import is guaranteed to be removed when compiling to JS. TS already fully erases types during compilation, and this is just a way to guarantee that the imports of types are guaranteed to be erased as well.

E.g. for `import { SomeType } from "backend-server"`, you don't want to include your entire server code in the frontend just because you imported the response type of some JSON API. `import type` neatly solves this issue, and it even enforces that you can't import anything but types from your backend.

I also want to state that TS already removes regular `import`s that only import types. `import type` is mostly used to help bundlers. https://www.typescriptlang.org/docs/handbook/release-notes/t...

PEP 649 “Deferred Evaluation Of Annotations Using Descriptors” should result in vastly reducing the need for TYPE_CHECKING.

https://peps.python.org/pep-0649/

I think in practice that increases the need for TYPE_CHECKING. If you use deferred annotations, and you don't import the types you use, type checkers and IDEs will complain, and go to definition won't work. But if you import the types outside of a TYPE_CHECKING block it's wasteful, since you don't actually need the import for your code to run. So you need TYPE_CHECKING to satisfy your linter without impacting runtime performance.

It will also break any runtime typecheck module that you might want to use.

> However, Python doesn’t have the compile-time check, because it’s an interpreted language that is dynamically-typed, which means its only real place to check is at runtime

Pet peeve: Python is compiled (into bytecode), so it could theoretically do checks at compile time. The "dynamically-typed" part is correct and is the real reason.

TypeScript is also interpreted, and does type checking without running code. Their argument is very weird.

That's an odd thing to say. The distinction between compiled and interpreted languages is very fuzzy, but TypeScript is clearly on the compiled side.

It is not so much that it is fuzzy, it is that the two definitions are not exclusive.

A compilation translates a language to another, whether it is native code, internal bytecode, portable bytecode, or another source language. Compilation might also type check and optimize depending on the language. Interpretation executes a language; normally bytecode is executed, but pure source level interpreters exist. Native code is "interpreted" by the cpu itself of course.

For example JAVA is certainly a compiled language, but it is typically compiled to bytecode and then interpreted or JITed (which is a combined interpreter/compiler).

Likewise, the Python typing annotation language could very well be on the compiled side...

Someone took the interpreted JavaScript language, added typing syntax on top, and typing can be checked statically.

The Python team took the interpreted Python language, added typing syntax on top, and here we are reading a claim that it can't be checked statically because it's interpreted. You understand my puzzlement.

TS is not interpreted. It compiles to JS, which is interpreted, but TS itself is not. There are no TS interpreters, they all compile to JS behind the scenes.

Or does Deno run TS natively? At a glance it looks like even its runtime is written in JS[0].

EDIT: looks like they have bindings for V8[1] so yeah, just a JS interpreter behind the scenes.

[0] https://github.com/denoland/deno/tree/main/runtime/js

[1] https://github.com/denoland/rusty_v8

The problem for Python with type checking is that the language has essentially no compile-time constructs at all, just a bunch of assignment operators with different syntax: `def` assigns a function object to a variable when it's executed, `import` assigns a module, `class` assigns a class object. This means that everything involving names in Python is just a variable lookup (incidentally, this is why a function call must follow the declaration in Python: the variable is unbound if you haven't executed the function definition yet). The reason `Sequence` fails in the example in the article is simply that the code is trying to read an unassigned variable, no different from writing `vairable` instead of `variable` in your code.

Small but important correction required in point 3 of the TL;DR:

> Python doesn’t care about types at runtime

This should say "Python doesn't care about type annotations at runtime."

Python _does_ care about types at runtime - but it doesn't use type annotations to compute them.

That doesn't detract from what's otherwise a really clear and helpful post. Python is living through a schizophrenic period in its evolution, and it's causing some problems. I hope it gets ironed out, though don't underestimate the difficulty. Python remains my go-to for many problems: a joy to use (for me) in many cases. But I definitely feel the need for static typing as codebase size increases. Having type annotations is good, but not being able to rely on them (i.e. static typing is not sound) adds to cognitive load and detracts from a confident development experience.

I agree with the gist of your comment.

I will add that, in my experience, having type annotations that are unreliable, cumbersome, with so many edge cases and requiring this kind of magic (e.g. "if TYPE_CHECKING", etc) which is beyond most "casual" Python users effectively means type annotations are a very hard sell for most non-hardcore team mates.

I've been in a position to try improving processes and code quality of the Python codebase, and introducing mypy and type annotations has always been a very hard sell. The team just won't accept the benefits, given the difficulties in using it. If only Python made it easier and more reliable, but the current state is a mess.

I come from the static typing world, so of course I see the benefits. But most/all of my team mates don't, and so they see this as some really odd hoops I want them to jump through, for minimal benefit (in their eyes).

I think type annotations are extremely valuable even if you don't use mypy. On my team we basically use them as comments that are understood by the IDE, but aren't enforced. Of course comments can lie, but appeasing mypy really is a lot of work, and I would generally rather have a slightly dishonest but mostly correct type hint than some TypeVar monstrosity or, worse, Any.

Hm. Type annotations as comments is the ultimate annoyance to my coworkers. They really do see them as "what's the point" in that case. Plus, like you say, they are misleading. The look like code, but they don't do anything.

I think if you're already documenting function argument types, there is no point. But in my experience it's unworkable to ask people to write full documentation for every function, but requiring type hints is not as hard and is nearly as useful. It is a little weird that they look like code, but I think you get used to it.

> However, Python doesn’t have the compile-time check

Yes it does. All Python source code is parsed and compiled into bytecode. SyntaxError is raised before runtime.

I think the author is obviously talking about compile time type checks here.

It's arguable whether a SyntaxError happens as a compile-time check. It's true that it is raised before runtime. But a document that can be parsed according to the syntax definition is a necessary precondition to all checks of any static program property. Syntactic correctness in itself is not a static property of a program. A program is, by definition, syntactically correct.

Without a correct syntax there is no way to assign meaning or execute or interpret, etc.

One question comes to mind: isn't cyclic dependencies something to be resolved at the root cause (remove the cycle)? It's an issue that causes various pains down the line, like type checking in this case.

Broadly speaking, yes, but for some situations like database models with bi-directional relationships that can cause unnecessary maintenance burden simply for the benefit of type checking which this allows you to work around since the cyclical nature is only caused by typing not runtime semantics

The issue is that, often, without type annotations there would be no cyclic dependency (because any type matching the implicit interface would work). So by introducing type annotations (which means extra imports), you end up having to refactor quite a lot of code and the end game is a one-type-per-module-with-separate-interface-definition system of the sort encountered in, well, statically-typed languages. If you don't do this you end up with the 'Type in string' annotations mentioned in the article and TYPE_CHECKING-gated imports. Both of which feel kind of hackish.

It's fine to have implicit circular dependencies in Python because duck typing means that every function is effectively generic. Type annotations eliminate this and turn Python into a fundamentally different language.

Type annotations aren't just extra boilerplate in function signatures, they also have these sorts of knock-on effects.

I read 'Type in string' like a forward declaration. I find uses for them within a single module. They're not always hacks.

Aside from type checking being a clusterfuck in python, wouldn't checking against a protocol be the right thing?

In python it's very hard to avoid cyclic dependencies. Something as simple as a parent-child link between two classes is a cyclic dependency, and if the classes are in different files you have a car like in the blog post. This comes up especially hard when typing your codebase, because you have to name every type, and in python implementation is the interface, i.e. you can't just include a type definition file.

> parent-child link between two classes is a cyclic dependency

What do you mean? Where is the cycle?

    class Parent:
      children : List[Child]

    class Child:
      __init__(self, parent: Parent):

Ah yes, it might be what they mean. I assumed inheritance. Thanks.

I also wonder. If a child class knows its concrete parent (apart from where inheritance is declared), and not just references it via super(), things have already gone badly wrong.

The parent holds a reference to the child. The child holds a reference to the parent.

Why does the parent hold a reference to the child?

Are you talking about a specific pattern or implementation detail in the interpreter? Because I would not call this "dependency"...

Parametrize the parent with the child and the child with the parent. Tie the knot after both are defined.

Removing cycles may require extensive refactoring of the code, sometimes resulting in unintuitive layouts. All, just to solve the cyclic-import errors.

At the core of the problem is the fact that there's no thing as "partial import" in Python. When you do "from M import A" in Python, all the contents of M are evaluated and a reference to A is added to the current namespace. So, cyclic dependencies arise sooner or later, unless you adopt some very un-pythonic style for your codebase (e.g. one file per class).

most of the Python code I've seen with type annotations appears to be significantly worse than the code without them.

You're in Python. Either embrace it or use another language.

Could you expand on it? It is truly surprising for me that anyone would find code with type annotations to be significantly worse, for any reasons whatsoever.

On the contrary, I joyfully read and write code with type annotations. It is obviously very useful knowing which object types a function expects and which it will return.

verbosity tends to make code worse, not better.

Professionals should stop putting up with unprofessional tools and languages. This? This whole mess is not professional. If you want types, do not use python. Python is not a typed language. You can't just bolt on a type system, as is being live demonstrated here.

TypeScript is a good example of perfectly "bolted on" type system.

But it is its own thing that compiles down to javascript, and is otherwise syntactically and semantically "very very similar" to javascript. It IS always statically typechecked and has good, or at least well defined, semantics for typing. Typescript does not change base javascript. It really is a well separated layer on top of javascript.

The way python does it is different. There's no "typethon". The python grammar was simply extended to allow any random expression as a "type".

In fact:

    x: print("hi") = 4

is valid python, and it does print "hi" without even so much as a warning. Personally, I don't think "print(hi)" should be a valid type.

This feels like splitting hairs -- python and typescript are equivalent, as they are both dynamically typed languages with type annotations.

The fact that the typescript implementation does not run typescript directly, and cPython does not type check before running the code, are both implementation details.

Typescript is just a better type system on top of javascript than python's type system is on top of untyped python, because typescript is a cohesive design rather than a collection or random stuff accumulated over time.

There's no reason that couldn't have happened for python, it just didn't.

> Personally, I don't think "print(hi)" should be a valid type.

And it isn't:

  a.py:1: error: Invalid type comment or annotation  [valid-type]
  a.py:1: note: Suggestion: use print[...] instead of print(...)
  Found 1 error in 1 file (checked 1 source file)

It is not. As the function `print("hi")` is executed, the argument type in this case becomes `Any`, as it would be for any other executed function.

As stated in the typing documentation [0]: "the only legal parameters for type are classes, Any, type variables, and unions of any of these types".

[0] https://docs.python.org/3/library/typing.html

That's fine and all. But it runs without errors or warnings.

I can put that in main.py and do `python3 main.py` and it will simply run fine.

What is the point of this whole system if it's not enforced?

Amen.

If you want type checking, run mypy.

If you don't want type checking, don't run mypy.

I don't see what's so difficult. If it was technically possible to run programs that have type errors in other languages, they would have the option to ignore type errors too, because it's convenient.

Typescript also ignores errors (generating js output for you to run) by default, so what's the point if it's not enforced?

Typescript does not ignore type errors by default. Type error = fail to compile

Try it, it produces output by default. You have to add a flag/option to change this https://www.typescriptlang.org/tsconfig/#noEmitOnError

Typescript "compilation" is just stripping out types, mostly. JavaScript is a subset of TypeScript (mostly). There was even a proposal to extend JavaScript comments syntax, so all typescript types would become comments (so you could just run typescript code with JS interpreter).

It’s a different language. Different enough to warrant a different name, at the very least.

PHP has been able to add in more typing over the same period of time and has seemed to avoid these Python problems.

I'm pretty sure that php has also explicitly broken backwards compatibility to achieve this. In general php has become far more strict over the years, which I think is a good thing. And they definitely at least sat down and thought about how to do types before adding grammar rules that allowed any random expression as the type, which is now the case in python.

The backwards compatibility has broken some - I think mostly visibly noticeable between 5.6 and 7.0. And every release there's some deprecations introduced that will eventually break in future. Many complex applications in 5.6 won't run 100% error free in 8.2, for sure. I'm working with a company to move from 5.6 to 7.4 and there's very little that doesn't work in the main code. Their earlier version of Doctrine was using a class called 'Null', and internally (unrelated to doctrine) they had a class named 'Null' - those had to be renamed. That took all of about 2 minutes. Upgrading Doctrine version got us to a version that removed the Null class name there too.

Python also broke backwards compatibility with their v3 release years ago. Unsure why breaking backward compatibility is being called out here(?).

But yeah, overall, it's struck me that PHP has introduced more and stricter typing rules with surprisingly little impact (compared to what could have been, of course).

I've had to work on a PHP project recently and it's horrible. Mypy has many pitfalls and it's not great, but at least it's entirely optional, so you can work around the issues or ignore some types until you can fix it or do some refactoring. Typescript might be a different language but it's the best implementation of adding types to a language I've seen so far.

> I've had to work on a PHP project recently and it's horrible. Mypy has many pitfalls and it's not great, but at least it's entirely optional...

Do you mean mypy is optional?

At the risk of opening a can of worms here, what's "horrible"?

In your own PHP code, you can get by without any typing at all. If/when you start to use 3rd party libraries, that may become a factor, though off the top of my head I can't think of a show stopper.

I disagree with this opinion.

I doubt many people picked Python "because it has types". It was picked because it was the right tool for the job.

Type hints were bolted on after the fact, and even in their limited form provide TONS of value for some people (myself included).

If you don't want them don't use them, but I doubt there is a project which uses Python specificially because it has type hints.

> It was picked because it was the right tool for the job.

I think in a lot of cases only the more general form of this statement is true, i.e.:

- it is a tool

- it was picked

- there was a job

It's reasonable and "professional" to want dynamically typed scripting language, for things where real types are a burden not a benefit, which also has libraries that work properly.

For me, that's the benefit of typing in python (as shonky as it is) -- you can use it to help write better library code, then ignore the types (except as accurate documentation) when using those libraries.

Maybe there are other languages that can do this, but the ones i know all have their own downsides.

There is, in my opinion, very little value in a type system that allows types to just be ignored. Or even a negative value, because it instills a false sense of security, because it does not actually prevent any of the mistakes that a type system is supposed to prevent.

If it catches bugs before run-time, that's a benefit to me, in the same way that testing doesn't catch all bugs (and yes, can introduce a false sense of security), but most people still think it's a good idea.

You obviously don't get the haskell experience, where if your program type checks it's probably correct, but the alternative (if, remember, you want a dynamically typed scripting language) is no type checking, which is definitely worse for some people.

python types can be ignored if you choose to ignore them.

Running pyright in CI/CD and in vscode has considerably improved the quality of my code, especially when refactoring.

We're not professionals. We don't profess anything. Whether we want software development to become a profession is a huge topic on its own. Professions come with a ton of strings attached and oversight. Something I would personally welcome, but know most of us wouldn't.

I think a type system might be successfully bolted on on Python. I think gradual typing might be made to work. I have no opinion on whether the current type system is good or not.

I do strongly believe that leaving the type checking to a third party program is a terrible idea. I also believe that it should at least be possible to opt-in to type-checking the annotations at runtime during execution (at the very least for tests).

Reply on news.ycombinator.com