1,434 karma · joined July 17, 2010
To be clear - I personally don't have an answer to this.
So, it feels to me ideally within the same classroom there should be a natural way to work on your own pace at your own level. Is it possible? Have no idea - seems not, again primarily because it requires a completely different skillset and attention from teachers.
Machine -> Asm -> C -> Python -> LLM (Human language)
It compiles human prompt into some intermediate code (in this case Python). Probably initial version of CPython was not perfect at all, and engineers were also terrified. If we are lucky this new "compiler" will be becoming better and better, more efficient. Never perfect, but people will be paying the same price they are already paying for not dealing directly with ASM.
(that's is different from let's say DVC - that does copy files into a local cache, always)
Is it possible to mount this shared filesystem on the root directory of the container? Then this slurm plugin isn’t needed.
Since it's developed in UCLA - do they have resources to do the next phases of trials (humans, etc)? I wonder if in such cases pharma (those who obviously have enough money) care about these molecules at all? Since they probably can't put a patent on it and heavily monetize it later. Are there examples of drugs that were developed like this successfully?
It allowed us to have:
- same engine in CLI (can generate HTML and open in browser), VSCode extensions, SaaS
- have a way to describe plot visualization / representation as a declarative spec that can be then used in all those products (plot spec). We were exploring plotly and AFAIU there was no easy way to do the same
- it's quite comprehensive and community is responsive, the project is maintained
To name a few downsides from our experience:
- DSL is quite complicated. It requires some time to master it. It hurts the adoption. In our case I don't see that many users doing custom plots / templates - majority is using pre-baked built-in stuff or use Python and export as SVG.
- In our case some features were missing (and are still missing) - exponential average - that is most commonly used to smooth ML training curves.
[1] https://vega.github.io/vega-lite/
[2] https://dvc.org/doc/user-guide/experiment-management/visuali...
ML teams have a lot of files- texts, images, video, pdfs, etc. Those objects have information (metadata) about them (e.g. labels, embeddings, captions). Last 5-6 years building DVC, we’ve observed a need for data versioning at scale but also a very strong need to store, enrich this metadata and also slice and dice those files based on this metadata (create datasets). We’d seen teams keep building and rebuilding the same infra or glue ETLs and scripts again and again and decided that it’s better to solve this in a more systematic way using our knowledge and experience.
DataChain is a Python library. Think about DataChain as a “data frame” with a “chain” of operations that can be applied to it - filter, merge, map, etc. We don’t store (or require moving or converting) files, rather DataChain is storing references to the origin (paths + version id). It is using a database underneath (SQLite) to preserve results (datasets) and do out of memory computation. It can do parallel computations, data caching, and many more things that make it better suited for unstructured data, ML, and larger scale.
Saved datasets (Data chains) can be passed to a data loader (e.g. Pytorch) to access the original raw files + metadata from the DB.
Please let us know your thoughts and questions in the comments!
Now, with my second kid going to school (public) I have to drive (it's literally 1 mile). It has changed the whole schedule. That single drive! Annoys me as hell. It's hard to imagine how convenient it is when you don't have to drive anywhere, even school, day care, etc. This makes a huge difference.
Iterative.ai (Series A) | REMOTE, WORLDWIDE | FULL-TIME | OPEN-SOURCE
Developer tools for ML engineers. We need people who are passionate about building infrastructure that help ML teams manage data (large scale), make sane reproducible workflow, track models. We are building infrastructure to deal with datasets of the 5N LAION scale. Our flagship open source tool- DVC.org (12K+ stars), SaaS product - studio.iterative.ai.
We are looking for senior (SaaS and open source) engineers:
https://iterative.notion.site/Iterative-ai-is-Hiring-852cb97...ML/DS always seemed to me closer to science in its nature vs software engineering. Can be because of its nature as well - in a lot of cases it's a process of incremental improvements (vs - simplifying this a lot - let's say in SE we do a button that just works or not).
I have a different hypothesis on this (never checked it though). I think as those languages - Python, JS were getting more and more popular we started using them more and more to build and maintain more complex projects. What worked for a few glue-code scripts doesn't work for a Dropbox size and scale project. Thus there was a need to evolve and augment the language.
Iterative.ai (Series A) | REMOTE, WORLDWIDE | FULL-TIME | OPEN-SOURCE
Developer tools for ML engineers. We need people who are passionate about building infrastructure that help ML teams manage data (large scale), make sane reproducible workflow, track models. We are building infrastructure to deal with datasets of the 5N LAION scale. Our flagship open source tool- DVC.org (12K+ stars), SaaS product - studio.iterative.ai.
We are looking for senior (SaaS and open source) engineers:
https://iterative.notion.site/Iterative-ai-is-Hiring-852cb97...