orbifold

Science shouldn’t be a zero sum game. As a physicist I love that we get a huge chunk of science funding for all kinds of stuff. Generally speaking developing a new accelerator leads to excellent technical advances at all levels of the stack. Web browsers and HTML were invented at CERN. Super conducting magnets at scale were pioneered for LHC. The detectors with custom radiation hard ASICs for particle detection and the data processing pipeline they feed into are at the pinacle of human engineering. The software infrastructure supports analysis of petabytes of data distributed across 100s of clusters around the world. This is because physicists working on colliders and particle physics are extremely passionate and intelligent people. Isn’t it nice that a fraction of the smartest people on the world are not working on getting you to click Ads and are using machine learning to advance our understanding of the fundamental forces of nature?

Tremendous value has been created by their work and an institution like CERN is hard to replicate because it is hard to get brilliant people to a agree what to do for the next 30 years of their lives. I’ve been taught by professors that did precisely that. Whatever theorists predict is not the only reason for them to do it, higher energies are a technical challenge, a data processing challenge, something to focus your attention and creativity on. There is no question we should try to build better telescopes, a 100 TeV collider is a better microscope for High Energy Physics.

This is personal to me, some of my friends work at CERN. I can’t help but feel it is personal to her. I actually met her once, she seemed nice enough, I just don’t understand why she focuses her time on work that is adversarial as opposed to constructive.

This is a bit of an oversimplification, because I don't actually know the details, but it works roughly like this: GPUs are devices on the PCI Express bus, without an operating system getting in the way everything on the PCI Express bus is organised into something called ECAM (https://en.wikipedia.org/wiki/PCI_configuration_space). Each device there specifies memory regions it supports. These memory regions map to a large number of configuration registers and on-device memory the GPU supports. Since GPUs are really complex most of its behaviour is actually not directly controlled but instead predetermined by its firmware (basically an operating system in its own right). Finally GPUs really execute multiple instruction streams with proprietary instruction formats and can independently load data from main memory with their build in DMA engines. These operations are typically kicked off by pointing various registers on the GPU to memory regions in main memory to tell it where to fetch instructions and data from. Everything else happens for the most part as consequences of instructions executing on the GPU.

A good physicist has the ability to grasp a large fraction of all physics that has ever been done. And there is a pretty good directed knowledge graph. The action principle and some ideas from quantum mechanics are enough to anchor almost all other results.

It is not ideal but for the subset of libraries that use CMake it can be as easy as dumping the library in a subdirectory and you are done. Things only start to go sideways when you have to integrate with google libraries since they have typically insane build tooling (skia).

Lattice QCD is something legitimate that people have been working on since the 80s. Some of the earliest massively parallel machines like the Connection Machine were thought to be useful for it. It’s totally plausible that physicists were crazy enough to build a custom Supercomputer for it. They routinely build complex custom ASICs for particle detectors. Note also the time (2004) nowadays they use GPUs for lattice QCD.

Wouldn’t be the end of the world. Most of those are really toxic and in the pocket of the NSA. Content licensing could lead to local competitors of Netflix easily.

It seems very unlikely that there can be vaccine that is effective long term. Just as there is no long term Flu vaccine.

Doing new drivers is not a huge deal. At least you are programming them in C with sane tooling against a sane API. Getting up a simple setup with a bit of DMA and a few registers is not that hard. It pales compared to the amount of pain you will go through working with the FPGA tooling. The insanity begins with the fact that they are very GUI centric with comparatively little guidance on batch processing capabilities. This is despite the fact that you usually run these tools on ~80 core servers, so you have nicely lagging X11 windows all day

time to precompile Plots: 138s

julia> @time Plots.scatter(data.eq_site_limit, data.hu_site_limit) 5.461595 seconds

subsequent calls report significant speed ups compared to matplotlib, but render about as quickly

julia> @time Plots.histogram(data.eq_site_limit) 2.143983 seconds

So I want to plot the data I just loaded. Since I wrote the comment I had closed the Julia window (not to make a point, btw.)

julia> import CSV (takes ~1-2 seconds)

julia> data = CSV.File("/User/orbifold/Downloads/FL_insurance_sample.csv) (takes seconds again...)

I want to explore the data. I vaguely remember there is Gadfly to do that. For some reason it depends on FFTW and a whole bunch of other packages, but the dependency story in Python is equally insane, so whatever. As I'm typing this, I'm waiting for Gadfly to precompile.

julia> @time import Gadfly (104.460307 seconds)

python> import matplotlib.pyplot as plt (~10s if you are unlucky and it has to generate the font cache)

Maybe it's a good idea to do a scatter plot of two of the fields? Let's find out

julia> @time Gadfly.plot(x=data.eq_site_limit, y=data.hu_site_limit) 0.320751 seconds

but that number is not accurate, in fact it took so long for the browser to open the result, that I had time to check the documentation on backends and see whether I'm missing something.

python> %time plt.scatter(data['eq_site_limit'], data['hu_site_limit']) 41.9 ms, a bit more for the window to open

Turns out that is not a useful plot... maybe a histogram?

julia> @time plot(x=data.eq_site_limit, Geom.histogram(bincount=10)) 0.042802 seconds

It in fact again takes several seconds until the browser shows a plot.

python> %time plt.hist(data["eq_site_limit"]) ~100ms

and the window with the plot opens instantly.

I installed CSV.jl and tried it out on the insurance dataset from here "https://support.spatialkey.com/spatialkey-sample-csv-data/".

First import took >10s just to precompile and then CSV.File took 5.21s to read the file. A fresh pandas install takes ~10s to import and then reads this file in 81.2 ms ± 1.25 ms. CSV.jl takes 38 ms on a second run so it is twice as fast, however especially to someone that just starts using Julia, these >100x slowdowns constantly happen. Especially when using libraries that heavily rely on multiple dispatch specialisation for performance.

You don't have to mandate something that people have done voluntarily. Mobile phones are cameras and microphones in every house, school and workplace. All the mobile standards have "legal intercept" provisions, which basically allow law enforcement to listen to your conversations / read your text-messages. This is just a demand to go back to the status quo ante, where Government had unrestricted access to all your phone conversations and mail if they chose to.

I would also wager that there is a non-zero chance that the baseband processor of all phones has a backdoor, which is why the US can't allow phones to predominantly use Chinese made baseband processors. The baseband has access to your microphone and camera and is mostly out of control of the application processor.

It should be "Scholze" of course.

I might misremember it slightly, I got told this by a professor (he was one of his examiners) and people that knew him at Bonn in ~2010. He could also have been awarded the majority of the credit points towards the Bachelor at entrance.

The starkest example to me of breadth first vs depth first learning are math geniuses. They typically ignore everything except mathematics to begin with and then hyper focus on a specific area of mathematics intensely. The most striking I know of is Peter Schulze who basically never attended an undergraduate class in mathematics and picked up Linear Algebra while he worked out the requirements to understand the Langlands Conjecture. He got awarded his bachelor by doing a 6 hour exam when he entered University.

Or they have a secretary that does ;)

Here is my main gripe with Julia: They blatantly copy Matlab. The same is also true to a lesser degree of Python's numpy and matplotlib, but Julia goes the whole nine yards, by basically replicating the syntax and copying most of the numerical APIs including the indexing convention. I understand why they do it and I get that Matlab is hugely popular, I just wish that there was more creative energy in the Open Source world and not just rehashing of 20-40 year old designs.

As a scientist: The amount I struggled with getting distributed training on a HPC cluster to work vs. how easy it was with Lightning was eye opening. Almost no code change and finally I can run across 20 nodes with 4 V-100 each :). Plus the automatic SLURM checkpoints and restarts <3.

Might not be your target audience, but high energy physics has been operating an infrastructure like that for years: https://www.etp.physik.uni-muenchen.de/research/grid-computi... You can basically use these frameworks to run your analysis jobs on any of the connected HPC centers and interactively move workloads around. The data never has to touch your hard drive either but gets moved to the compute on demand. This is how thousands of physicists do statistical analysis on petabytes of data.

It doesn’t apply to the universe because of GR at least that is the conventional view I think.

Exactly, it encourages mafia like structures. It matters a ton who you know and who your PI was as a Postdoc and PhD.

They don't address the fundamental issue of BPTT in an RNN model. To my mind this is a much larger block to biological plausibility than how to track the error locally. Besides this has been addressed in various forms in earlier work:

- Hochreiter and Schmidhuber present a truncated version of BPTT in the original LSTM paper that tracks part of the error locally and forward in time (https://www.bioinf.jku.at/publications/older/2604.pdf)

- Bellec et al. https://www.nature.com/articles/s41467-020-17236-y present a similar algorithm that tracks the error with something they call eligibility traces. Despite the title of the paper the method is also applicable to non-spiking RNNs.

- Bohnstigl et al. present another solution: https://arxiv.org/pdf/2007.12723.pdf

All the major toolchains work on Linux. I personally liked Pynq http://www.pynq.io/, it works fairly well and is beefy enough to do non-toy projects with it. For something totally barebones you can try https://www.latticesemi.com/Products/DevelopmentBoardsAndKit..., it has the advantage that it has a very nice open source tool flow available for it: http://www.clifford.at/icestorm/. I own both boards and I work professionally with EDA tools, honestly the biggest cliff is setting up tooling and processes around a given project. If you don't want to do everything from scratch you need to either rely on proprietary libraries, which will cost you a fortune and/or lock you into a very specific tool flow (Synopsis Designware libraries). The other alternative is to use one of the open source ecosystems like https://github.com/m-labs/migen or tooling around Chisel or various scattered libraries of open-source components which are often hard to get to work together, but also this requires a massive amount of time investment to just get the synthesis / simulation flow working.

The situation is slowly improving and there are various university research groups and corporations that have released larger amounts of open source code that one can use as a starting point:

- https://github.com/openhwgroup/cva6 (mostly done by ETHZ people initially, also see https://github.com/pulp-platform/pulpino)

- https://github.com/chipsalliance/Cores-SweRV-EL2 (one of several high quality open-source RISC-V cores by Western Digital)

- https://github.com/chipsalliance/rocket-chip (UCB by initial creators of RISC-V)

- https://github.com/electronicvisions/nux (POWER based embedded processor, that we have used in several neuromorphic chip tapeouts (silicon proven))

On the tooling side one great thing that happened is that the verilator open source simulator has gained a lot of traction.

- https://github.com/verilator/verilator

On the hardware synthesis side some exciting developments are also happening, mostly driven by the desire of companies like Google to be vendor independent for their Deep learning accelerator designs and through funding by DARPA.

You need at least 10x over an established approach to have any hope of surviving in the mid-term. The results are for batch-size 1 as well. GPUs are throughput optimized. So I wouldn't be surprised if an A100 chip would be able to outperform this for batch size >4. I still think this is a really cool architecture though.

Research in this area has always been cyclical. The Neural ODE paper that won a best paper award at Neurips two years ago just implements a technique that is common knowledge in the optimal control theory community since the 60s.

I wouldn’t dismiss an approach because it isn’t popular right now. Biological neurons force you to think about constraints you normally wouldn’t and crucially have solved temporal credit assignment, memory etc. in interesting ways that already have inspired some breakthroughs.

As an example the replay memory idea was directly inspired by the rough understanding neuroscientist developed of the role of the hypocampus in learning.

Ironically SPJ doesn’t have a PhD. Basically this is more a question of interest and time investment and not whether you got a title.

Not to be too mean about it, but I feel like this is an instance of brilliant marketing more than anything. The founder of Numenta knows how to communicate to Engineers in a convincing way. Neuroscientists (and Science in general) has a way of politely ignoring outsiders and in any case Computational neuroscience doesn’t have terribly high standards of rigor and quality anyways.

Bangkok also feels like a real life Gotham city. They have this elevated train line running between fancy shopping centers, high rise luxury apartments and office buildings. But just below it there is miles and miles of street prostitution and poverty.

I can highly recommend Shadow aswell, a friend of mine uses it and for online multiplayer it is a really nice option. It I didn’t own a decent PC I probably would choose that to play my Steam games.

Those seem like fun people to be around.