rkp8000

I would also highly recommend Gandhi the Man, by Eknath Easwaran.

Another very useful trick I learned for handling criticism, especially when you're standing nervously on stage and the critic is being unfriendly, unreasonable, and unlikely to change their mind about anything:

If you don't feel you can give a quick response in a way that will bring them to your side right away, instead of arguing simply keep a notepad handy and (visibly) take a quick note about what they said and thank them. Most folks that raise unreasonable criticisms at talks just want to be heard, and in my experience they may even suddenly become very friendly to you for visibly acknowledging their comments, opening the door for a productive and constructive conversation.

I love Markdown. I'm a bit surprised, though, that you still can't open a .md file by default in most web browsers. It seems like it should be quite trivial to have the browser automatically convert it to html and display it.

Computing entropies of high-dimensional random vectors for a theoretical neuroscience study. The journey is mostly a repetition of (1) almost giving up because it's completely hopeless, (2) taking a hot shower, (3) realizing there might actually be a path forward, (4) almost giving up because it's completely hopeless.

A notable and interesting point of this article is that convolutions and correlations (convolutions without flipping the filter) are quite a bit more subtle on the sphere than on Cartesian spaces. For a convolution between a function and a filter on R^N you just "slide" the filter around, integrating at each shift, which produces another function on R^N. On a sphere, however, there is not a clear cut way to slide a filter around a sphere. For instance, there are multiple ways to slide a filter centered at the north pole to the south pole, which will result in different filter orientations.

More generally, the space of rotations, which is the argument of the convolution (analogous to the shift amount being the argument of a standard convolution), is 3D (3 Euler angles), whereas the space of points on the sphere is 2D (polar and azimuthal angles). Thus, whereas convolution over R^N returns a function over R^N, convolution over the sphere actually returns a function over the 3D rotation group SO(3). This has interesting consequences for e.g. the convolution theorem on the sphere, which is not as clear cut as simply rewriting the standard convolution theorem in spherical terms.

A (non-exhaustive) list of some notable papers:

Maass 2002, Real-time computing without stable states: https://pubmed.ncbi.nlm.nih.gov/12433288/

Sussillo & Abbott 2009, Generating Coherent Patterns of Activity from Chaotic Neural Networks https://pmc.ncbi.nlm.nih.gov/articles/PMC2756108/

Abbott et al 2016, Building functional networks of spiking model neurons https://pubmed.ncbi.nlm.nih.gov/26906501/

Zenke & Ganguli 2018, SuperSpike: Supervised Learning in Multilayer Spiking Neural Networks https://ganguli-gang.stanford.edu/pdf/17.superspike.pdf

Bellec et al 2020, A solution to the learning dilemma for recurrent networks of spiking neurons https://www.nature.com/articles/s41467-020-17236-y

Payeur et al 2021, Burst-dependent synaptic plasticity can coordinate learning in hierarchical circuits https://www.nature.com/articles/s41593-021-00857-x

Cimesa et al 2023, Geometry of population activity in spiking networks with low-rank structure https://journals.plos.org/ploscompbiol/article?id=10.1371/jo...

Ororbia 2024, Contrastive signal–dependent plasticity: Self-supervised learning in spiking neural circuits https://www.science.org/doi/10.1126/sciadv.adn6076 Kudithipudi et al 2025, Neuromorphic computing at scale (review) https://www.nature.com/articles/s41586-024-08253-8

Thanks for the link to your article!

If you don't mind my asking, how much does the role of mutual information in linking logical and thermodynamic reversibility depend on considering quantum systems? I.e. does your footnote 37, which discusses independent systems vs "subsystems of correlated systems" hold for classical systems as well?

A great pedagogical article on thermodynamic vs logical reversibility, for those interested: https://arxiv.org/abs/1311.1886 (Sagawa 2014).

There are a few folks working on this in neuroscience, e.g. training transformers to "decode" neural activity (https://arxiv.org/abs/2310.16046). It's still pretty new and a bit unclear what the most promising path forward is, but will be interesting to see where things go. One challenge that gets brought up a lot is that neuroscience data is often high-dimensional and with limited samples (since it's traditionally been quite expensive to record neurons for extended periods), which is a fairly different regime from the very large data sets typically used to train LLMs, etc.

This is just to say: the poem at the end is a play on William Carlos William's poem, entitled "This is just to say".

This reminds me of a project I worked on during my PhD, where you create a network of scientific documents and notes/threads via Markdown, with a similarly structured "rabbit-hole" linking system: https://github.com/rkp8000/hypothesize . I'm not really a software engineer so never made it ready for general use, but I'm very happy to see a similar idea turned into something real! Kudos to the authors.

MSR has a very clear and accessible tutorial on quantum computing for anyone interested in getting up to speed with the fundamentals: https://www.youtube.com/watch?v=F_Riqjdh2oM .

Yep! This relationship is well known in statistical mechanics. I was just surprised that in many years of intersecting with information theory in other fields (computational neuroscience in particular) I'd never come across it before, even though IMO it provides an insightful perspective.

Here is a link to the press conference with Hopfield at Princeton University: https://www.youtube.com/watch?v=Uk3Ok1D1xR8

Connectome-adjacent neuroscientist here. Definitely not a dead end! But also definitely not the whole picture.

One of the main open questions in neuroscience right now is how network structure, dynamics, and function are related in the brain. Connectomes provide tremendous insight into structure, but as mentioned this does not generically solve either the dynamics or function problem. For example, for many of these neurons we don't have a good understanding of their input-output relationship, and the nature of this relationship can strongly affect the dynamics that emerge in a highly connected network. Individual variability across connectomes, and how connectomes change over development are also a significant issue, but at least for the fly it's thought that many of the basic structures are pretty conserved across adult animals, even if many of the details could differ.

Modulo these caveats, knowing the physical network structure of the brain does still impose huge constraints on what kinds of models we should be using for gaining insight into dynamics and function. For example, there are well known areas (the "mushroom bodies") with specific feed-forward connectivity patterns that are very different from a random recurrent network. Further, there are at least some areas in the fly brain where we think there are indeed quite clean structure-function relationships, e.g. in the central complex of the fly brain, which contains a physical ring of neurons and is thought to support a "bump" of activity that acts as a sort of compass that helps flies navigate via a ring-attractor-like dynamical system. Thus, even though it has many missing pieces, a wiring diagram like this can be tremendously useful for generating hypotheses to guide more targeted experiments and theoretical studies.

I love that Ridge Regression is introduced in the context of multicollinearity. It seems almost everyone these days learns about it as a regularization technique to prevent overfitting, but one of its fundamental use cases (and indeed its origin I believe) is in balancing weights among highly correlated (or nearly linearly dependent) predictors, which can cause huge problems even if you plenty of data.

I had the same thought. If a were constant I believe the solution would just be a convolution of b_t with an exponential filter. I think this is different because a_t depends on time, so it's equivalent to convolving b_t with a filter whose timescale changes at every timestep, which doesn't seem like something that's obviously parallelizable with FT.

The recent peer-reviewed version is here: https://www.jneurosci.org/content/43/7/1074 but unfortunately not open access, as far as I can tell.

For those confused about how the hyperdimensional computing (HDC) approach (also known as "vector symbolic architectures"/VSA or "holographic reduced representations"/HRR) described in the article differs from the use of vector embeddings in more mainstream artificial neural networks:

1. HDC is largely organized around the observation that certain symbolic-like operations become vastly simplified in high-dimensional spaces and can be performed with simple algebraic manipulations.

For instance, if you have a dictionary of words, each represented by a random (but fixed) high-dimensional vector, you can store subsets of these words just by summing their vector representations together. This works because random high-dim vectors are nearly orthogonal with very high probability. This implies that the sum of several of such vectors will have essentially a zero dot product with words (ie their embeddings) not included in the subset and a much larger dot product (~1 if the vectors are all normalized) with words included in the subset (as long as the number of words in the subset is sufficiently smaller than the total dictionary size). Hence the sum encodes the subset since subset membership can be checked with a dot product.

Notably, this also works when the dictionary size is exponentially large relative to the vector dimension, since it is possible to sample an exponentially large number of near-orthogonal vectors in a high-dimensional vector space (unlike in a low-dimensional vector space). This is also very mathematically similar to how a [Bloom filter](https://en.wikipedia.org/wiki/Bloom_filter) works, which is a probablistic data structure commonly used for set membership queries when the set is extremely large (e.g. all possible URLs). (Note also, though, that this is at the expense of being able to decode/recover the set elements directly from the summed representation.)

HDC/VSAs capitalize on other nice properties of random high-dimensional vectors as well, e.g. via the ability to "bind" two words together using a circular convolution. See [Kanerva 2009](http://rctn.org/vs265/kanerva09-hyperdimensional.pdf) for a nice review of various examples like this. There are also various similar ways to represent sequences of words, tree structures, graphs, etc, all within a fixed- but high-dim vector space, so they provide a nice way to represent "syntactic" structure in data objects, if you will.

2. None of the above emerges as a property of network training. In fact, a network isn't even necessarily required, which could be a feature or a bug, depending on your perspective.

As a feature, this is useful in the sense that it provides an immediate no-training-required way of representing objects with fairly complex relational structure (e.g. a sequence of words or a tree-structure) as a fixed- but high-dimensional vector. In a traditional neural network (although this may be less true for very modern LLMs) trained to do something with, say, sentences of a max of 20 words, the network would likely fail to "figure out how to represent" sentences composed of 100 words. With the HDC/VSA approach, however, you get a representation of such a sentence right off the bat, and don't have to worry about it being inside/outside your training dataset. The "utility" of such a representation is not necessarily obvious, and will depend on exactly how it is created, but IMO it is nice to know that there is a systematic way of constructing one that does not interfere with others (with very high probability).

On the other hand, one of the main drawbacks of this approach (at least so far) is that it has generally not been obvious how to make these systems learn robustly, so that e.g. vector representations could change through learning to capture more semantic relationships between words and objects. Nonetheless, given the above one can imagine how the HDC/VSA approach may provide something akin to a useful inductive bias or initialization for representation learning in more trainable systems.

It will certainly be interesting to see if and how these might get incorporated into modern AI systems in the coming years.

Just in case anyone hasn't tried it before, the "tight_layout" keyword in matplotlib is a game changer. Instantiating a figure via e.g. "fig, axs = plt.subplots(1, 2, figsize=(6, 3), tight_layout=True)" automatically arranges the labels and subplot boundaries etc to not bump into each other. It doesn't work 100% of the time, but it usually does a pretty good job and has honestly saved me a ton of plotting headaches.

There is a fairly unknown (in my experience) but absolutely amazing podcast about storytelling called The Narrative Breakdown (https://www.stitcher.com/show/the-narrative-breakdown-story-...) . They cover many different aspects of what makes stories work or not work and apply their analyses to a variety of well known films. I highly recommend The Power of Irony episode if you're looking for a place to start.

I haven't touched it in years, but during grad school I created a fully operational browser-based app for integrated note-taking and reference management for academic projects, based on locally stored markdown files, and designed to minimize attentional breaks: https://github.com/rkp8000/hypothesize .

I ended up using it for the remainder of my PhD; however, it was unfortunately a bit too easy to accidentally delete an entire note document, and I never got around to fixing it (although at least one other person ended up using it as the primary tool for their grad studies also :) ).

It definitely earned me no more than $0/month.

This reminds me of the quote: "If I had more time I would have written you a shorter letter" (attributed to various people, e.g. Twain and Churchill).

In my domain (developing research software for small-scale scientific projects) more lines of code rarely correlates with more functional/readable/trustworthy software. Instead I frequently find myself thinking: If I had more time I would have written less code!

This isn't a typical academic research paper but is from a University of Chicago researcher, and I think gives a great overview of failure modes of complex systems in general (software included): https://how.complexsystems.fail .

As someone who writes a lot of complex/evolving data analysis software that needs to work correctly, I find some of the considerations listed in the above to be immensely helpful.

Always cool to see unexpected applications of information theory, especially outside of probabilistic contexts.

A related toy example that comes to mind is the puzzle where you're given 12 metal balls and told that one of them is either heavier or lighter than the rest. You're then given a balance/scale and instructed to figure out which ball is different and whether it is heavier or lighter using the balance a maximum of three times to weigh different combinations of balls.

Part of the reason this is solvable is because there are 12x2=24 different possibilities (or log2(24) = 4.58 bits) for the ball ID/weight and 27=3x3x3 possible outcomes (or log2(27) = 4.75 bits) for the weighing process (left heavier, right heavier, or equal in each use). Thus, there's a chance the weighing sequence could convey the ball ID/weight. If there were 14 balls, however, then you would require log(28) = 4.81 bits of information to specify the ball ID/weight, so the puzzle would be unsolvable, since there wouldn't be enough information available in the weighing sequence to specify the ball ID/weight.

A big one for me was realizing that a matrix times a vector returns a weighted sum of the matrix columns, with the vector's elements as weights. It's not particularly profound but it has definitely clarified my intuition around a number of matrix problems.

An audio recording of D.T. Suzuki's "What is Zen?"

In the era of large language models and the ethical and existential questions they raise, it seems useful to explore a variety of philosophical stances on language. Zen (as well as many other eastern ways of thinking) has a rich and beautiful relationship to language yet which starkly contrasts with the way language is typically studied and integrated into western thinking, and which I think could provide useful perspective into how we relate to LLMs going forward.

For anyone interested in these ideas, I'd highly recommend reading Shannon's original 1948 paper (linked in the article). While technical and precise it is actually quite digestible for someone with a bit of a quantitative background.

I also think this is a quite reasonable. For some work on this beyond anecdotal evidence, see https://journals.plos.org/ploscompbiol/article?id=10.1371/jo... . The authors' main finding (in rats and humans) is that the perception of the intensity of a stimulus relates to the perception of its duration, i.e. more intense stimuli are perceived as longer lasting. A speculation beyond the specific study would be that more intense experience generally (perhaps due to the saliency of more unique life events) could lead to a slowed down perception of time.