Unfortunately legal teams were very discouraging to do so and publishing that type of thing is hard, so it didn't happen.
1,725 karma · joined August 29, 2020
Unfortunately legal teams were very discouraging to do so and publishing that type of thing is hard, so it didn't happen.
Everything about it is open source except for a little web app thing (which is a really nice one, but you don't have to use it).
I am using Typst to write my dissertation along with git and it's great.
They have several pieces to the project at https://github.com/typst .
The comemo package actually seems quite useful for everyone that writes Rust, regardless of if it's related to Typst or typesetting at all. Though, I don't write much Rust, so take that with a bag of salt.
It has a far nicer syntax and is quite a lot more capable.
Now tbe modern tech stack is to build a bash+python app in an env, add a touch of R, some js, and bundle it into docker container, then make that docker container into wasm with container2wasm, and give that out as the executable.
It's wonderful honestly. You can get just about anything working with stitching together stuff, and then serve your 10GB executable to anyone :).
So MuCh BeTtEr ThIs MoDeRn WaY!
Shmidhuber beat both of these guys easily with his paper from decades before these amateurs even claimed to start working on it.
Additionally capabilities can be added in a Makefile form to use IPFS as the cache to ensure that a script that takes 3 months on a HPC/supercomputer to make a 3MB file only has to grab the file in a few seconds, even if not yet computed locally (e.g. you just computed it at work and do `git pull && ipfsmake`).
And the wonderful thing is, every person that used git clone on this repo and ran it would be serving the NN weights.
But alas, this unfortunately hasn't been done yet.
So if everything on the entirety of github.com was deleted, you could still do a git clone somehow and it would pull it from everyone who has ever cloned that repo?
Because that's what I'm referring to here - p2p not hub-spoke architectures.
I'm fairly certain git doesn't inherently have this feature, unless the backend remote could automatically start an iOS or torrent daemon and deal with torrent or ipfs for pulling and pushing.
Rinse and repeat.
¯\_(ツ)_/¯ That's python!
Well, actually on second thought it does: If ipfs or torrent can be used as a data version control backend, people no longer have to pay $600 to get the very popular and basic dataset of arxiv.org from AWS - under torrent or IPFS it's free. So basically decentralization here means running away from money hungry scams like AWS.
If "ipfs" can be added as a remote, and it automatically pulls from thousands of different devices without having to specify them, that's a pretty big win for redundancy right?
The package management and build tools for python have been so atrociously bad (environments add far too much complexity to the ecosystem) that it turns many developers away from the language altogether. A system like Rust's package management, build tools, and cross compilation capability is an enormous draw, even without the memory safety. The fact that it actually works (because of the package management and build tools) is the main reason to use the language really. Python used to do that ~10 years ago. Now absolutely nothing works. It takes weeks to get simple packages working, only can do anything under extremely brittle conditions that nullify the project you're trying to use this other package for, etc.
If python could ever get it's act together and make better package management, and allow for cross-compiling, it could make a big difference. (I am aware of the very basic fact that it's interpreted rather than compiled yada yada - there are still ways to make executables, they are just awful). Since python is data science centric, it would be good to have decent data management capabilities too, but perhaps that could be after fundamental problem are dealt with.
I tried looking at mojo, but it's not open source, so I'm quite certain that kills any hope of it ever being useful at all to anyone. The fact that I couldn't even install it without making an account made me run away as fast as possible.
Ipfs remote for git should be the default.
This way your git repo could be located on: - GitHub - Your Closet (...) - UCLA's supercomputer - JBOD in Max Planck Institute (...) - GitLab
Doing this with a simple file with "[ipfs, github, gitlab]" on it would be revolutionary, especially for data version control, like nn weights or databases that are too large for git and cost too much on other services, as they would be free on ipf/torrent.
Then no one is phased by the inevitable failure of various companies.
If you want to host the data on Dropbox, Dropbox becomes a part of the network and is hosted by the community and Dropbox or whatever service you like. The problem is that it is very prohibitive to use services like AWS. For instance, downloading all of Arxiv.org from AWS is around $600 last time I checked. Not every student trying to run an experiment is going to have $600 laying around to run the experiment. But with torrent/ipfs it would be free to download.
It only adds. I don't understand how it subtracts.
There are a few that have started suggesting quantum mechanics playing a large role in cognition, but very few take them seriously (obviously it has an effect, but likely much can be understood more classically, etc).
The fact that few are moving toward that style of thinking seems to give a bit more credibility to NNs being closer to the correct model. If spiking NNs take off more, we'll probably see more arguments around that, and if Blue Brain's full in-silico modeling takes off we may see the succinct description given by those studies used to describe ideas. However, to first approximation, NNs and spiking NNs aren't really a bad way to reason about large descriptions of brain dynamics, in many circumstances.
It's an analogy for a reason. It bothers me when people combat analogies so incredibly hard. Of course, it is not really fitting or the same thing - it's an analogy.
For example, Snakemake (os-independent make) with data version control based off of torrent (removing complication of having to pay for AWS, etc) for the caching of build steps, etc would be a HUGE win in the field. *No one has done it yet* (some have danced around the idea), but if done well and correctly, it could reduce the amount of code and pain in reproducing work by thousands of lines of code in some projects.
It's important for the default of a data version control to be either ipfs or torrent, because it's prohibitive to make everyone set up all these accounts and pay these storage companies to run some package. Ipfs, torrent, or some other centralized solution is the only real solution.
Also, I'm not sure what your issue with ipfs is; If it's 'I saw something something crypto one time' - it's a really poor argument. IPFS works completely independently of any crypto - it has nothing really to do with it. The solution can also be torrent - I don't care too much - it's just possible that IPFS can run with far less resource usage on lower power, etc because it's more modern (likely uses better algorithms in the protocol, deals with modern filesystems better, with better performance, hopefully have better security, etc) and it's likely easier to implement. But it doesn't matter if it's torrent because it would work essentially the same way.
> * Not understanding dependencies, public/private, SCM or versioning, making their own code uninstallable after a few months
This is definitely true, but I've searched *far and wide* , and unfortunately it's not a simple task to get this right.Ultimately, if there were a simple way to get data in the correct state in an os-independent, machine independent (from raspberry pi to HPC the code should always work), concise, and idempotent way - people would use it. There isn't. But the certainly could be.
The solution we desperately need is a basically a pull request to a simple build tool (make, Snakemake, just, task, etc) that makes this idempotent and os-independent setup simple. Snakemake works on windows and Unix, so that's a decent start.
One big point is matching data outputs to source code and input state. *Allowing ipfs or torrent backends to Snakemake can solve this problem.*
The idea would be to simply wrap `input/output: "/my/file/here"` in `ipfs()`, wherein this would silently check if the file is locally cached to return, but if not go to IPFS as a secondary location to check for the file, then if the file isn't at either place, calculate it with the run command specified in Snakemake. It's useful to have this type of decentralized cache, because it's extremely common to run commands that may take several months on a supercomputer that give files that may only be a few MBs (exchange correlation functional) or only a few GBs (NN weights) so downloading the file is *immensely* cheaper to do than re-running the code - and the output is specified by the input source code (hence git commit hash maps to data hash).
The reason IPFS or torrent is the answer here is for several reasons: 1) The data location is specied by the hash of the content - which can be used to make a hash map of git commit hashes of source code state that map to data outputs (the code uniquely specifies the data in almost all cases, and input data can be included for the very rare cases it doesn't) 2) The availability and speed of download scales with popularity. Right now, were at the mercy of centralized storage systems, wherein the download rate can be however low they want it to be. However, LLM NN weights on IPFS can be downloaded very fast when millions of people *and* many centralized storage providers have the file hosted. 3) The data is far more robust to disappearing. Almost all scientific data output links point to nothing (MAG, sra/geomdb - the examples are endless). This is for many reasons such as academics moving and the storage location no longer being funded, accounts being moved, or simply the don't have enough storage space for emails on their personal Google drive and they delete the database files from their research. However, these are often downloaded many times by others in the field - the data exists somewhere - so it just needs to be accessible by decentralizing the data and allowing the community to download the file from the entire community which has it.
One of the important aspects to include in this buildtool would be to ensure that, every time someone downloads a certain file (specified by the git commit hash - data hash map) or uploads a file after computing it, they host the file as well. This way the community grows automatically by having a very low resource and extremely secure IPFS daemon host all of the important data files for different projects.
Having this all achieved by the addition of just 6 characters in a Snakemake file might actually solve this problem for the scientific / data science community, as it would be the standard and hard to mess up.
The next issue to solve would be popularize a standard way to get a package to work on all available cores/gpu/resources, etc on a raspberry pi to HPC without any changes or special considerations. Pyspark almost does this, but there's still more config than desirable for the community, and the requirement of installing OS-level dependencies (Java stuff) to work on python can often halt it's use completely (if the package using pyspark is a dependency of a dependency of a dependency, wet lab biologists [the real target users] *will not* figure out how to fix that problem if it doesn't "just work"[TM])
You should check the source code for every project you're using as much as possible to really ensure that there isn't anything obviously nefarious like telemetry included, because in some cases it's still there.
See Sunxi's Linux SBC backdoors on git for one of many examples of undocumented insecure or privacy-breaking code. In other cases it may be more documented, such as Golangs baked-in telemetry.
There should be better ways to check these problems. The best I have found so far is Crev https://github.com/crev-dev/crev/. It's most used implementation is Cargo-crev https://github.com/crev-dev/cargo-crev, but hopefully it will become more required to use these types of tools. Certainty and metrics about how many eyes have been on a particular script, and what expertise they have would be a huge win for software.