HNHacker News
TopNewBestAskShowJobs

smaddox

1,728 karma · joined January 15, 2015

submissionscomments
smaddox··on Unexpected downsides of UUID keys in PostgreSQL
KSUID's are have temporal-lexicographical order plus 128 bits of entropy, which is more than UUIDv4.

https://github.com/segmentio/ksuid

smaddox··on Follow up to “I booted Linux 293k times”
Based on my reading, under some circumstances the scheduler wasn't dedicating large enough chunks of time to a particular boot process for it actually make progress. The fix appears to have been to change how the time chunk size was calculated.
smaddox··on Carrington event of 1859 disrupted telegraph lines. Miyake event would be worse
Even if it didn't damage any equipment (which seems unlikely), it would still likely shut down most of the power grid, which would shut down most of the power plants, since they would have nowhere to sink they're output energy. Cold booting the energy grid could take months, during which most of the country would be without power.
smaddox··on Morris Tanenbaum, inventor of the silicon transistor, has died
Basically, yes. Or at least it's related. The game is based on (arguably, a misinterpretation of) the writings of Elizer Yudkowski.
smaddox··on Astmaker – A DSL in Rust for programming language designers
How so?
smaddox··on Astmaker – A DSL in Rust for programming language designers
Easier for quick prototypes, for sure (although I switched to Rust from Haskell due to even worse compile times in Haskell and my disdain for monad transformers). Also, Rust can't currently do nested pattern matching through Boxes, which can be annoying, coming from Haskell.

But for a high performance compiler, manually managed arena allocation will beat out a generational GC.

smaddox··on Morris Tanenbaum, inventor of the silicon transistor, has died
Interesting hypothetical. If semiconductors were never discovered, I suspect we would have kept shrinking vacuum tubes, eventually down to the point (which has now been demonstrated) that atmospheric pressure is sufficiently low pressure to operate at.
smaddox··on Morris Tanenbaum, inventor of the silicon transistor, has died
Flip flops existed before transistors.
smaddox··on Morris Tanenbaum, inventor of the silicon transistor, has died
I'm not so sure. Last time I followed the field, modern HDDs were nanostructured, such that each magnetic domain was literally a physically distinguishable object, in a sense. I'm not sure if that's still the case, and I'm also not 100% sure that ever made it in to production, but I'm pretty sure it did.
smaddox··on Morris Tanenbaum, inventor of the silicon transistor, has died
Paperclips?
smaddox··on The Neon God
Quite the deluded and hypocritical take. The author focuses exclusively on the negative aspects of technology while ignoring all of its positive benefits. Even the fact that he's using technology in both writing and publishing this essay is framed as valiantly attacking the machine from within.

No doubt our world is far from perfect, but does he honestly believe destruction of technology would somehow magically improve the situation, when we know from history that our technological past was full of just as much, if not more, unnecessary suffering than the present?

Vigilance is necessary, yes. Reducing our reliance on fickle technology is valuable, yes. Practicing mindfullness, in our use of technology as in other aspects of everyday life, is beneficial, yes. But, no, destruction is not merited.

smaddox··on OpenAI ignored the ‘have a problem to solve’ rule, says president Greg Brockman
Technology is just the tip of the iceberg that is knowledge and understanding. The deepest advancements to knowledge and understanding come primarily from self-driven and self-motivated investigation. The closer you get to "technology", the more development is driven by the necessity to solve a problem. But technological advancement would stagnate and plateau without advancements in knowledge and understanding.
smaddox··on Writing a SQLite Clone from Scratch in C (2022)
Even just watching the first ~20 episodes dramatically altered how I approach coding, for the better. At least, when combined with his blog posts, e.g. https://caseymuratori.com/blog_0015 (you can skip the snarky intro)
smaddox··on Why did Google Brain exist?
I don't think that's really true, but you don't need the full context in ML in order to apply it to your field of interest. Just like you don't need the full context in Physics to apply it to an engineering problem.

Some of the most successful ML researchers have read several decades of research papers. It's not uncommon to see references to papers from the 70s or 90s.

Edit: No doubt that the relevant parts of stats, random matrix theory, and ML is a newer field than Physics or Chemistry, though.

smaddox··on SpaceX Starship rocket explodes minutes after launch from Texas
The larger the rocket, the less overhead per kg of payload.

Also, yes: larger space telescopes. And eventually,manned missions to Mars.

smaddox··on Most of my skills are now worth nothing, but 10% are worth 1000x
Well, it was trained on Stack Overflow responses.
smaddox··on From deep to long learning?
RWKV is random chance on the PathX problem. Both of those models are > 95% on it, if I remember correctly.

I don't think Hyena can be run in RNN mode, though. But it _might_ scale better than the LRU model (second paper).

smaddox··on Emergent abilities of large language models
How would a non-concious entity intelligently plan and act in response to stimuli in situations that require self reflection, such as interaction in a social environment? Consciousness seems necessary for effective goal pursuit in such an environment.
smaddox··on From deep to long learning?
This EfficientVectorRouting paper (and related papers) is fascinating. It seems to have endless potential applications. I'm surprised I had not heard of it before. I'm excited to experiment with novel model structures incorporating it. What's the catch? I'm guessing they're fairly expensive layers, since they're based on EM? Although it looks like it's using incremental EM (iEM), which perhaps makes it less expensive?

May I ask what you are using it for?

smaddox··on From deep to long learning?
Yeah, just saw that. Sorry for the noise.
smaddox··on From deep to long learning?
Other approaches for long sequence lengths:

- "Hyena Hierarchy: Towards Larger Convolutional Language Models" https://arxiv.org/pdf/2302.10866.pdf

- "Resurrecting Recurrent Neural Networks for Long Sequences" https://arxiv.org/pdf/2303.06349.pdf

It's unclear (at least to me) if any of these scale as well as transformers at this point.

smaddox··on OpenAI Tokenizer
Yes, one distribution per position. This was a key innovation that allowed training over the entire sequence in parallel, rather than on one token prediction at a time, thereby massively speeding up training.

More recently, there are models like RWKV that can run in both parallel (GPT-like) mode for training and serial (RNN-like) mode for inference.

But transformers always output a probability distribution at each position in the context.

smaddox··on OpenAI Tokenizer
> An LLM is a function that takes an array of integers and returns a new array of integers.

To refine this a bit more, a LLM is a function that takes an array of integers (or really, a batch of arrays of integers), and returns a probability distribution for each possible integer, with the array shifted left by one place to enable prediction.

smaddox··on The Future of the Transistor
> 2D materials are clearly the future of the industry, with a lot of momentum pushing the field forward.

I've been out of the semiconductor materials and devices game for a while, but I wouldn't be so sure. Just because these materials have a lot of papers about them, doesn't mean they're the future. They get a lot of papers written about them because they're relatively easy to study and because governments are funding research on 2D materials (though I can't really say why they receive so much funding. It's completely disproportional to their promise.)

Even if you could manufacture wafer scale 2D material transistors with high yield (which is a huge if), what does it buy you? Based on my, admittedly out of date, knowledge of 2D semiconductor devices, they're no panacea. Contact resistance is even more of a problem than it is for silicon at leading edge node scales. But even if it could match, or maybe even slightly beat the performance of silicon (another huge if), again, what does it buy you? One extra node, if you're lucky?

No, if my money had to be on a CMOS successor, it would be on nanoscale vacuum switches. Basically a vacuum tube scaled down to modern VLSI dimensions, at which atmospheric pressure is sufficiently low pressure to operate. Their biggest problems are 1) the lack of a complementary switch (so no CMOS equivalent, at least not without MEMS which doesn't scale), and 2) device longevity, which is likely solvable. These devices are far more manufacurable than 2D semiconductor material transistors. All of the process steps already exist, they can be stacked, unlike semiconductors, which must be single crystalline for good performance and therefore cannot be grown on insulators, device characteristics have a much higher ceiling thanks to the full vacuum barrier, and contact resistance is far less of a problem, because all of the junctions are between metals.

My second runner up would be room temperature superconductor switches, but we don't have that technology yet.

smaddox··on Llama.cpp 30B runs with only 6GB of RAM now
Based on that discussion, it definitely sounds like some sort of bug is hiding. Perhaps run some evaluations to compare perplexity to the standard implementation?

Edit: looks like there's now confirmation that running it on a 10GB VM slows inference down massively, so looks like the only thing strange is the memory usage reading on some systems.

smaddox··on Cerebras-GPT: A Family of Open, Compute-Efficient, Large Language Models
Cerebras is "training compute optimal". Llama appears to be trained far beyond "training compute optimal". The tradeoff is that inference is closer to optimal for Llama, i.e. better performance with a smaller model.
smaddox··on RWKV RNN: Better than ChatGPT?
There's already at least one company with a product: Mythic AMP https://mythic.ai/products/m1076-analog-matrix-processor/

And there's several research papers about other architectures. For example: https://arxiv.org/abs/2205.05853

smaddox··on LoRA: Low-Rank Adaptation of Large Language Models
There's already RWKV, if you want a decent performing pre-trained model that's Apache 2.0 licensed: https://twitter.com/BlinkDL_AI/status/1638555109373378560?s=...
smaddox··on Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models
Cool. Stereoscopic diffusion images coming soon.
smaddox··on CoLT5: Faster Long-Range Transformers With Conditional Computation
Sounds interesting.

Another recent paper on efficient architecture for long context lengths: https://arxiv.org/abs/2302.10866

← PreviousPage 4 of 29Next →