Mehdi2277

That aligns pretty well with a past job. Those two areas were very popular user interests. Third one was cosmetics like skincare routine.

I'm very happy with pyright. Most bug reports are fixed within a week and new peps/features are added very rapidly usually before pep is accepted (under experimental flag). Enough that I ended up dropping pylint and consider pyright enough for lint purposes as well. The most valuable lints for my work require good multi-file/semantic analysis and pylint had various false positives.

Main tradeoff is this only works if your codebase/core dependencies are typed. For a while that was not true and we used pylint + pyright. Eventually most of our code is typed and we added type stubs for our main untyped dependencies.

edit: Also on pylint, it did work well mostly. tensorflow was main library that created most false positives. Other thing I found awkward was occasionally pylint produces non-deterministic lints on my codebase.

Uv is installer not a build backend. It’s similar to pip. If you install library with uv it will call backend like setuptools as needed. It is not a replacement for setuptools.

If I assume you mean LLM like models similar to chatgpt that is pretty debated in the community. Several years ago many people in ML community believed we were at plateau and that throwing more compute/money would not give significant improvements. Well then LLMs did much better than expected as they scaled up and continue to iterate now on various benchmarks.

So are we now at performance plateau? I know people at openai like places that think AGI is likely in next 3-5 years and is mostly scaling up context/performance/a few other key bets away. I know others who think that is unlikely in next few decades.

My personal view is I would expect 100x speed up to make ML used even more broadly and to allow more companies outside big players to have there own foundation models tuned for their use cases or other specialized domain models outside language modeling. Even now I still see tabular datasets (recommender systems, pricing models, etc) as most common to work in industry jobs. As for impact 100x compute will have for leading models like openai/anthropic I honestly have little confidence what will happen.

The rest of this is very speculative and not sure of, but my personal gut is we still need other algorithmic improvements like better ways to represent storing memory that models can later query/search for, but honestly part of that is just math/cs background in me not wanting everything to end up being hardware problem. Other part is I’m doubtful human like intelligence is so compute expensive and we can’t find more cost efficient ways for models to learn but maybe our nervous system is just much faster at parallel computation?

My experience working on ml at couple faang like companies is gpus actually tend to be too fast compute wise and often models are unable to come close to theoretical nvidia flops numbers. In that very frequently bottlenecks from profiling are elsewhere. It is very easy to have your data reading code be bottleneck. I have seen some models where our networking was bottleneck and could not keep up with the compute and we had adjust model architecture in ways to reduce amount of data transferred in training steps across the cluster. Or maybe you have gpu memory bandwidth as bottleneck. Key idea in flash attention work is optimizing attention kernels to lower amount of vram usage and stick to smaller/faster sram. This is valuable work, but is also kind of work that is pretty rare engineer I have worked with would have cuda kernel experience to create custom efficient kernels. Some of the models I train use a lot of sparse tensors as features and tensorflow’s sparse gpu kernel is rather bad with many operations either falling back to cpu or sometimes I have had gpu sparse kernel that was slower than cpu equivalent kernel. Several times densifying and padding tensors with large fraction of 0’s was faster than using sparse kernel.

I’m sure a few companies/models are optimized enough to fit ideal case but it’s rare.

Edit: Another aspect of this is nature of model architecture that are good today is very hardware driven. Major advantage of transformers over recurrent lstm models is training efficiency on gpu. The gap in training efficiency is much more dramatic with gpu than cpu for these two architectures. Similarly other architectures with sequential components like tree structured/recursive dynamic models tend to fit badly for gpu performance wise.

I’ve worked at companies with async training. Async training does help on fault tolerance and also can assist with training thoroughput by being less reliant on slowest machine. It does add meaningful training noise and when we did experiments against sync training we got much more stable results with sync training and some of our less stable models would even sometimes have loss explosions/divergence issues with async training but be fine with sync training.

Although even for async training generally I see dataset just sharded and if worker goes down then shard of data may be loss/skipped not some kind of smarter dynamic file assignment factoring when workers go down. Even basic things like job fails continue from last checkpoint with same dataset state for large epoch is messy when major libraries like tensorflow lack a good dataset checkpointing mechanism.

This is fair guess on intuition but working in recommender space on both content/ad recommendation, content understanding signals have pretty consistently across two companies and many projects tended to underwhelm and key signals are generally engagement signals (including event sequences) and many embeddings (user embedding, creator embedding, ad embedding, etc).

The main place I’ve seen content understanding help is coldstart specially for new items by new creators.

I’ve occasionally worked with more dynamic models (tree structured decoding). They are generally not a good fit for trying to max gpu thoroughput. A lot of magic of transformers and large language models is about pushing gpu as much we can and simpler static model architecture that trains faster can train on much more data.

So until the hardware allows for comparable (say with 2-4x) thoroughput of samples per second I expect model architecture to mostly be static for most effective models and dynamic architectures to be an interesting side area.

Having used it heavily it is nowhere near painless. Where can you get a TPU? To train models you basically need to use GCP services. There are multiple services that offer TPU support, Cloud AI Platform, GKE, and Vertex AI. For GPU you can have a machine and run any tf version you like. For tpu you need different nodes depending on tf version. Which tf versions are supported per GCP service is inconsistent. Some versions are supported on Cloud AI Platform but not Vertex AI and vice versa. I have had a lot of difficulty trying to upgrade to recent tf versions and discovering the inconsistent service support.

Additionally many operations that run on GPU but are just unsupported for TPU. Sparse tensors have pretty limited support and there's bunch of models that will crash on TPU and require refactoring. Sometimes pretty heavy thousands of lines refactoring.

edit: Pytorch is even worse. Pytorch does not implement efficient tpu device data loading and generally has poor performance no where comparable to tensorflow/jax numbers. I'm unaware of any pytorch benchmarks where tpu actually wins. For tensorflow/jax if you can get it running and your model suits tpu assumptions (so basic CNN) then yes it can be cost effective. For pytorch even simple cases tend to lose.

23 comes from 2023. Pip uses calendar based versioning. The 23.3.1 means 2023, 3rd quarter, second release. The last number is for bug fix release. The first two numbers are purely date based and do not follow semvar.

PEP 646 Variadic generics, https://peps.python.org/pep-0646/, was made for this specific use case but mypy is still working on implementing it. And even with it, it's expected several more peps are needed to make operations on variadic types powerful enough to handle common array operations. numpy/tensorflow/etc do broadcasting a lot and that probably would need a type level operator Broadcast just to encode that. I also expect the type definitions for numpy will go fairly complex similar to template heavy C++ code after they add shape types.

Having worked at similar companies on similar systems usually A/B experiments and smaller probability of an action bigger weight it must have to matter much overall. The constants are generally done through some ab tests to get them into reasonable overall behavior but they are a pain to tune and very unlikely optimal in any real sense as it’s often too difficult to do extensive search of them. Like often I’ll see new target have a couple different weights tried on an ab and then maybe second set of experiments after rough magnitude is determined.

Python documentation defines standard library and includes venv, https://docs.python.org/3/library/venv.html, as part of it. There is also python steering council/core developer group that has had discussions with debian/other distributions about this issue and considered those versions of python incomplete. The docs.python.org is pretty much formal document comparable to how c++ has cppreference for defining standard library.

It matters in sense that packaging/python tooling may in future treat your distribution that's not included as not supported use case and just reject it. Distribution made by vendor can do whatever it wants, but I do consider calling it python implementation problematic at that point. If vendor can change implementation of python however they like and remove large subsets of it, then name has little meaning.

It’s opposite. Venv is built into python and part of standard library. It is part of cpython repo and developed by python maintainers. Some Linux distributions remove it from standard library but windows/Mac always has it. The Linux distributions that have python without venv are considered incomplete python install and packaging community has considered not supporting that case in future. At moment pip tries to allow that case but in future may just require venv and consider Linux distributions that remove it as shipping a problematic python.

The issue is most of my commits have little meaning and often include in progress commits where code is not even functional/tests may crash. Especially as I have tooling that builds artifacts for me based on commits/runs ci based on commits. Often I’ll commit something just to see the result from ci running it or to build artifact for a small test deployment fully expecting it to not work yet. After I’m done I’ll collapse all commits for a pr into one for the actual thing the pr aimed to do and aim to keep collapsed commit around <300 lines.

Pyright's vendoring of stubs or pylance's? Pylance vendors extra stubs (microsoft type stubs), but pyright itself does not. Pyright does vendor typeshed, but all type checkers including mypy vendor typeshed.

Narrowing on object truthiness unsure what you mean. That's intended to be supported. The out example is real difference where mypy does do better. It should at least type check on basic pyright but won't type check on strict without doing out: list[int]. That one comes from mypy has special handling for lists to allow inferring type of elements later, while pyright always infers type of a variable only based on it's declarations and never it's method calls like append.

Pyright is pretty recent and started development years after mypy. But some of pyright's design goals (laziness for LSP) would have made it very difficult to do in mypy without a very large refactor/rewrite. Pyright is implemented in typescript for better LSP/vscode integration and performance. pytype (google's type checker) is only one I know that predates mypy for when it started development.

Pyre's reason is interesting, https://github.com/facebook/pyre-check/issues/38. Pyre is based off of Hack, facebook's php compiler. Facebook made pyre to re-use tooling/techniques from Hack on python. Secondary reason is performance. Pyre is done in OCaml and having same performance in type checker implemented within python is hard.

Typescript similarity is expected as maintainer of pyright does talk with typescript maintainers and sometimes pyright adds features inspired by typescript's inference.

I think the major reasons for mypy is if you are working on a library to be shared with many others (especially open source one) you'd like library to be compatible with mypy as your users are likely to use mypy as it's still most popular type checker. So for open source libraries it makes sense to use both mypy/pyright. For projects that you do not expect main users (company internal one you define standards) it's more fine to pick type checker you prefer. The other main place mypy can be helpful is plugins if you need some behavior outside of type system for a library you work with a lot.

Also pyright is implemented in typescript which does mean you'll need to have a CI dependency on javascript environment that python project's often don't need. Lastly for typical python developer it's a bit easier to make pull request to mypy vs pyright just as they normally have more python knowledge then javascript knowledge. This is pretty minor given most people don't debug there own type checker and pyright's bug list is very short.

Django is one library that has some apis that are beyond type system and likely difficult to ever fully describe in type system. Some of this it handles with a custom mypy plugin to overrule/extend some type rules. No other python type checker I'm aware of supports plugins so this is one place mypy will have an advantage. Usually when behavior is too dynamic to be described well the stubs will fallback to Any leaving some typing holes for other type checkers.

At same time this only affects a subset of Django behavior and I have used pyright with a Django codebase and it mostly worked well. The more you use very dynamic features of Django the more this matters.

One note for both is you do want to tune configuration settings. Pyright basic is fairly easy to satisfy, pyright strict is too hard for most codebases. Similarly mypy defaults are too lenient, but mypy strict is pretty hard (even mypy doesn't use strict to check itself). I roughly go for strict for both and then remove ~5 hardest rules and call that good enough. Strictest rules pretty much require all of your dependencies to be well typed/stubbed.

I’ve heavily used both mypy/pyright.

The semantic differences are mainly in areas where type system is not specific enough on exact behavior expected. As an example if you have a function that has overloads how do you decide which overload to pick? Sometimes that can become ambiguous especially when type variables are involved or multiple overloads match. Cases like that are where you see most intentional differences and as a user of both type checkers I consider depending on specific behavior there to be like using c++ compiler’s undefined behavior.

The bigger area I see differences is feature development/bug fix velocity. Mypy is maintained well. Pyright is maintained magically and I’ve reported bugs that maintainer fixes the same day. Most bug reports to pyright get fixed in a week. Only a couple tend to stay open for a while. New type system features (peps) get implemented really fast in pyright and usually if you try to use very recent features you will need to wait a while for mypy to catch up. A good example is release time for paramspecs or typevartuple. Most of the time new type system peps are added in pyright while still in draft stage.

That trivial way only lazily shallow imports. I don’t see a good way to do a lazy deep import. A lot of libraries I import, then transitively import hundreds or more of other files. The file I import I may only need a small subset of those transitive imports. The lazy import pep would have meant that whenever the import was finally executed, the imports in that file are also lazy and only done if needed.

Numpy has a large amount of python code. C is definitely used in places, but majority of numpy is python. Not all operations need C implementations especially as many can be built on top of other python functions which do wrap C. Tensorflow/pytorch similarly have a large amount of python code. I mostly work on developing a library that builds on tensorflow and is like 50K lines of code. Dividing that library into services makes little sense. One of the recent typing new features is mostly devoted to numerical ecosystem in python (variadic generics to do template like types for matrices). I don't want to use dynamically typed language, but python is clear leader in ML ecosystem. A lot of my department does work that builds on ML research/libraries and those are mostly found in python.

If other languages had comparable ML ecosystems then a different language may have been chosen. But at moment today there's no competitor anywhere close. One lazy metric, I would estimate 90%+ of research papers to ML conferences are in python.

For a while I used both mypy and pyright for my team’s codebase. After about half a year I eventually dropped mypy . I think type checking is valuable just that most of errors mypy detected pyright also caught and using newer type features often led to mypy false positives. I had trouble justifying using both when I could require my teammates to install pyright. Advanced type features tend to run into more bugs and while both are well maintained, pyright’s maintenance is magical. I do not know any other open source library that fixes bugs as fast (most bugs are fixed in under a week). The main thing that eventually forced decision was a flaky (depends on cache) mypy crash using paramspecs half a year ago. At time paramspec support was still in progress and there’s a good chance that specific issue is fixed.

The main awkwardness of pyright is it’s node library and most python devs I work with don’t interact much with node. But my team has a bash script that installs all our dependencies including node as needed (nvm) which mostly works. One benefit is you can use pyright as an LSP and it works very convenient in vscode.

Edit: 3rd party library lacking types is probably biggest issue. As my codebase is mostly typed by itself I’ve started gradually writing type stubs for library apis we use. Only writing stubs for small percent of what we use helps but there’s still a ton to add given codebase was started without types.

That's not been my experience much at all work or research wise. Tensorflow, pytorch, jax are still very dominant. I've worked at several companies and interviewed at several dozen for ML roles. They have 100% been python/c++ for ml. I'd be impressed if even 2% of ML engineers used Julia.

Python is also the dominant language for machine learning which does care for performance. The person who made recent nogil work is one of the core maintainers of key ML library. The standard workaround is ML libraries, the performance sensitive stuff is written in C/C++ (either manually or with cython) and then uses python bindings. But it would be much friendlier if we could just use python directly.

It's also commonly used language for numerical work in general. Most of the time numpy is enough and then occasionally you'll need something not already implemented and then have to do your own bindings.

It's implemented in 3.11. Documentation still hasn't been written for it yet though and that's why the issue is still open, https://github.com/python/cpython/issues/90908.

Partly motivated? The primary focus was ML space. The creators of PEP 646 work at Facebook with goal of supporting pytorch tensor dimension tracking. That was main motivation and many of the features/design of that pep were based on what's needed to type hint tensor dimensions.

Pep was originally even longer and there's planned follow up peps for other tensor related type features like literal arithmetic to allow type hinting function like np.concatenate. I expect 2/3 more peps in that area in the next year or two.

I quite like monitoring dashboards and use them frequently. Mainly for examining system health and debugging operational issues.

My previous team had some business metric charts/dashboards related to system we worked on that we reviewed weekly. Most weeks review was it looks normal and was pretty fast review. Occasionally we'd see an unexpected deviation in some top level metric and that could trigger follow up work. Ideally all interesting deviations an alert would catch, but alerts are never complete.

Most of dashboards I've cared for my team/department was primary user of. I have not made dashboards much for other stakeholders.

Arm CPUs including M1 macs do not support avx. AVX is an x86 specific thing.

I don’t understand this as distinction with file systems. Networked file systems exist and I have a lot of production code that shares data as files in either some networked file/blob system. Often databases even just store a reference to a file path for stuff like images/other large artifacts that are shared from some file system.