HNHacker News
TopNewBestAskShowJobs

smaddox

1,728 karma · joined January 15, 2015

submissionscomments
smaddox··on Reproducing GPT-2 in llm.c
Transformers have quadratic computational complexity in sequence length, i.e. O(N^2) where N is the sequence length. RNNs, Linformer, Mamba, etc. have linear or quasi-linear computational complexity in sequence length, which often bottlenecks information movement across tokens.

In theory, if you grew the RNN's state quadratically vs sequence length, you could likely achieve comparable performance to transformers, but it would likely be less efficient than transformers.

smaddox··on Ask HN: Hacker News-like forum, specific to game development?
There's https://hero.handmade.network/forums , which is sort of a part of https://handmade.network/forums, which has a jobs board. And there's an associated discord chat, and loosely associated Handmade Cities conferences: https://handmadecities.com/
smaddox··on Dragonfly: An optical telescope built from an array of off-the-shelf Canon lens
> with unprecedented nano-fabricated coatings with sub-wavelength structure on optical glasses.

Also known as an anti-reflection coating. Definitely not unprecedented.

Cool project, though.

smaddox··on Full-colour 3D holographic augmented-reality display with metasurface waveguides
The image quality looks quite good, and the variable depth of focus is something that I've been eagerly awaiting. I personally find the 3D effect wears off after a bit with existing stereographic systems, likely due to the fixed depth of focus.
smaddox··on Atomic nucleus excited with laser: A breakthrough after decades
Very interesting! Two-photon pumping of traditional lasers with only 2 states would require absurd energy densities, but I guess the combination of a strong quadrupole coupling and long state decay times makes it feasible for a nuclear laser.

Still not clear what it would be useful for outside of clocks, but definitely a very cool physics project, and I'm sure someone would find other uses eventually.

smaddox··on AI leaderboards are no longer useful. It's time to switch to Pareto curves
> Retrying makes sense because LLMs aren’t deterministic even at temperature zero.

This is news to me. I'm trying to think where non-determinism would come in at temperature zero, but coming up with nothing. What am I missing?

smaddox··on Electromigration Concerns Grow in Advanced Packages
It was a major issue with aluminum interconnects. It was primarily addressed by switching to a different metal that exhibits less electromigration, namely copper. But you have to be careful putting copper in a silicon fab, because if it gets into the silicon channel, it kills device performance (copper doping induces a deep trap in silicon).
smaddox··on Atomic nucleus excited with laser: A breakthrough after decades
To build a laser, you would need at least three energy levels, and ideally four, with particular constraints in the transition probabilties so that you can create population inversion. And you would need to pump it with a higher-energy (shorter wavelength) laser.

Perhaps doable with a free electron laser, but probably not with traditional lasers, due to the energies involved.

But, yeah, not sure what the use would be. Maybe a form of lidar that allowed measuring speed of objects to extreme precision, by measuring the dopler shift of reflected light? (Assuming light at such a short wavelength is reflected sufficiently, which it probably wouldn't be).

If there happened to be one atom/matrix that could be tuned to the transition energy of another atom's fission transition energy, then you could use it to burn nuclear material / waste. But you could just do that with the pump laser directly.

I suppose you could maybe pump such a laser with a very high temperature plasma (like fusion temperatures hot), rather than with a free electron laser. Then maybe it might make more sense.

smaddox··on Atomic nucleus excited with laser: A breakthrough after decades
For free atoms, yes. For atoms in a crystal lattice (or other solid), it's quite common for electrons to decay through phonon interactions, i.e. by emitting vibrations (i.e. heat) to the lattice.
smaddox··on LLMs use a surprisingly simple mechanism to retrieve some stored knowledge
Bag of words models use a context that is a "bag" (i.e. an unorder map from elements to their counts) of words/tokens. GPT's use a context that is a sequence (i.e. an ordered list) of words/tokens.
smaddox··on The baffling intelligence of a single cell: The story of E. coli chemotaxis
AFAIK, the part that's been debunked is that there's a complete separation of concerns between the two hemispheres. From studies on split-brain patients, there does appear to be some specialization, but it's much fuzzier than "right brain does art, left brain does analysis" or anything like that.
smaddox··on Shattered Pixel Dungeon is an open-source traditional roguelike dungeon crawler
I feel your pain. I've spent more time playing Dungeon Crawl Stone Soup than I care to admit. The fact that this can be played on Android makes it even more dangerous.
smaddox··on It's official: Europe turns to the Falcon 9 to launch its navigation satellites
Anyone care to comment on why this concise response to the parent's question, supported with a link to the relevant data, was down voted? I'm confused.
smaddox··on It's official: Europe turns to the Falcon 9 to launch its navigation satellites
It doesn't help that most of Europe is facing significant population decline: https://zeihan.com/demographics-part-4-the-european-breakdow...
smaddox··on Stealing Part of a Production Language Model
They don't disclose the embedding dimension for gpt-3.5, but based on table 4, comparing the Size and # Queries columns, gpt-3.5-turbo presumably has an embedding dimension of roughly 20,000? Interesting...
smaddox··on Avoiding fusion plasma tearing instability with deep reinforcement learning
Agree, that will be the proof. But tokomaks have yet to do the same despite decades of investment.

We know that helion is able to recover 95% of the energy of every pulse. And they've measured the scaling laws. There doesn't seem to be anything that will prevent net positive energy in the reactor they're building right now.

smaddox··on Avoiding fusion plasma tearing instability with deep reinforcement learning
It's relevant when it doesn't work after all that time. As I argue further down the thread, the trajectory of FRCs looks much more promising.
smaddox··on Avoiding fusion plasma tearing instability with deep reinforcement learning
If tokomaks worked as well as Rockets, then I would agree. But the age of a technology does become relevant when you're discussing the history and development trajectory. The trajectory for FRC's looks far more promising. Direct electric conversion reduces the required efficiency by ~2x and the use of a low-neutron emissions fuel dramatically reduces shielding complexity and maintenance costs. And FRC requires much smaller reactors to reach net positive energy. Even considering high-temperature superconducting tokomaks, FRC's appear far more promising.
smaddox··on Avoiding fusion plasma tearing instability with deep reinforcement learning
Tokamaks are 1960's technology. The future of economical fusion appears much more likely to be based on the field-reversed configuration (FRC). Helion expects to produce net positive energy production from a reactor designed primarily for He3 production in the next few years: https://www.helionenergy.com/
smaddox··on The Era of 1-bit LLMs: ternary parameters for cost-effective computing
Damn. Well, I guess I better hurry up and write and publish a paper on the Ternary Neural Network research that I've been doing (part-time) for the last several months, before it all gets scooped.
smaddox··on Why flying insects gather at artificial light
I read about this explanation years ago. It's not novel. I don't remember where, though... It might have been from Feynman.
smaddox··on I don't always use LaTeX, but when I do, I compile to HTML (2013)
That's just HTML that looks like a PDF, though. Incredible feat, but not really what I want from PDF turned to HTML. I want something mobile friendly.
smaddox··on Maestro: A Linux-compatible kernel in Rust
CPU-local could work, but would give up the possibility of static type verification. Agree passing around context can be tedious, but Rust currently doesn't have a way to implicitly pass context (although there have been some proposals along those lines).
smaddox··on Understand how transformers work by demystifying the math behind them
That isn't a very good example. The vectors for each token are randomly initialized with each element taken from the normal distribution. After training, similar words will have some cosine similarity, but almost never as much cosine similarity as [1,2,3,4] and [2,3,4,5].
smaddox··on Maestro: A Linux-compatible kernel in Rust
You would probably need to pass around a context type that encodes information about the current context and which interrupts are possible. You would then acquire the lock via that context, which would handle disabling those interrupts.
smaddox··on Show HN: Rewriting our engine on Rust and going open source
I don't understand what this has to do with AI. It seems like Heroku for Fast API?
smaddox··on Mamba: Linear-Time Sequence Modeling with Selective State Spaces
> If replicated, it's arguably the first linear-time architecture with Transformer-quality performance!!

RetNet was published in August (although I only learned of it last week): https://arxiv.org/abs/2307.08621

smaddox··on Low current around roots boosts plant growth
Yeah, seems like they would need to use gold or platinum electrodes to rule out copper (or whatever) leaching. But even if it's copper leaching that boosts growth, that seems interesting in and of itself---just not really due to electric current.
smaddox··on Kolmogorov Neural Networks can represent discontinuous functions
Well, the approach currently doesn't apply to architectures being used in practice, and it's not clear, at least to me, if it can scale to large datasets.
smaddox··on Kolmogorov Neural Networks can represent discontinuous functions
It might also be possible to solve a corresponding convex optimization problem, similar to https://arxiv.org/pdf/2303.03382.pdf
← PreviousPage 2 of 29Next →