HNHacker News
TopNewBestAskShowJobs

heyitsguay

1,081 karma · joined January 5, 2017

submissionscomments
heyitsguay··on Do simpler machine learning models exist and how can we find them?
People have tried (and continue to try) that human-in-the-loop data growth. Basically any applied AI company is doing something like that every day, if they're getting their own training data in the course of business. It helps but it won't turn your bag-of-words model into GPT3.

Companies like Google have even spent huge amounts of time and money on enormous labeled datasets -- JFT-300M or something like that for computer vision tasks, as you might guess, ~300M labeled images. It creates value, but it creates more value for larger models with higher capacity.

heyitsguay··on Do simpler machine learning models exist and how can we find them?
How often are closed-form equations actually useful for real world problem domains? When i did my PhD in applied math, they mostly came up in abstracted toy problems. Then you get into the real world data or a need for realistic modeling and it's numerical methods everywhere.
heyitsguay··on Do simpler machine learning models exist and how can we find them?
I think this is one of those issues where it's easy to observe from the sidelines that models "should" be smaller (it'd make my life a whole lot easier), but it's not so clear how to actually create small models that work as well as these larger models, without having the larger models first (as in distillation).

If you have any ideas to do better and aren't idly wealthy, I'd suggest pursuing them. Create a model that's within a percentage point or two of GPT3 on big NLP benchmarks, and fame and fortune will be yours.

[Edit] this of course only applies for domains like NLP or computer vision where neural networks have proven very hard to beat. If you're working on a problem that doesn't need deep learning to achieve adequate performance, don't use them!

heyitsguay··on Ask HN: How will the content on Hacker News change with AI tooling?
Like I said, rough analogy. The "lossy compression" here is in the implicit world models these networks learn, not necessarily in the generation space (though we certainly see deviations from reality there too). However, the bigger difference between learning from SEO-generated blogspam and human teaching is the lack of feedback mechanisms to correct errors in generated output that then get taken as ground truth.
heyitsguay··on Ask HN: How will the content on Hacker News change with AI tooling?
This is only a very rough analogy, but consider these text AIs like lossy JPEG compression for images -- using an efficient statistical representation of the content, you can store a lot of information in a little space and reconstruct something aaaalmost like the original. If I (and a few hundred million other people) have gone through the trouble to create a nice PNG picture or whatever, and then someone comes along and saves it as a JPEG at.. say.. 95% quality, that JPEG will look pretty great. Hard to distinguish from its source.

But what happens when our JPEG saver only has other JPEGs to start with? And soon, is saving copies of copies of... you get the idea. The images lose more and more of their original real-world structure, and become dominated instead by artifacts of the compression and reconstruction processes.

That's basically the (a) challenge for this AI training paradigm of scraping web-scale content. The more that content is generated by the previous generations of AI, the more we're asking the next generation to learn from copies of copies of the original human intelligence that powers them

heyitsguay··on 'M2 Max' Geekbench scores leak online, revealing rumored specs and performance
For tech work computers, I'd say 32GB is the minimum one should have these days. My partner got a 16gb RAM laptop from her new company as a data analyst and she runs into trouble with it at least a couple times a week when she's got a lot of processes running.
heyitsguay··on Twitter to employees: all office buildings closed, badge access suspended
Is there precedent anyone can point to where these sorts of wild dramatic moves under new leadership led to turning a large company around into success?
heyitsguay··on Musk’s first email to Twitter staff ends remote work
Yeah but at least from the outside, to continue the metaphor, this seems like a CO taking over a struggling ship and deciding to just blow up all the ammunition in place. The ship needs to do something new, and this is something new, but it seems like it's just sinking faster now.
heyitsguay··on DarkForestSim: A Netlogo Simulation of the Dark Forest Hypothesis
I appreciated the write-up but i think it runs into one of the issues I've encountered myself messing around with netlogo, where for complex agent-based stuff, you sort of end up baking the result into your modeling choices. For the three civ variables, there are no constraints or trade-offs between them, and all comparisons are monotonic (more power + detection always better, less visibility always better). It's cool to see the visualizations and quantify the progression, but I don't think you need to run the sim to predict the result.
heyitsguay··on RTX 4090 review: Spend at least $1,599 for Nvidia’s biggest bargain in years
Probably similar to the 3090Ti - it comes with a little leash that connects to the card and ties off on the top of your case!
heyitsguay··on Transformers seem to mimic parts of the brain
What do you mean? That's an incredibly important distinction in understanding mathematics for ML in the neural net age. Perhaps a bit of a sensitive spot for me personally, coming from a harmonic analysis group for my math PhD, but the short version basically goes like: Up until the 2010s or so, a huge portion of applied math was driven by results from "continuous math": functions mostly take values in some unbounded continuous vector space, they're infinite-dimensional objects supported on some continuous subset of R^n or C^n or whatever, and we reason about signal processing by proving results about existence, uniqueness, and optimality of solutions to certain optimization problems. The infinite-dimensional function spaces provide intellectual substance and challenge to the approach, while also being limited in applicability to circumstances amenable to the many assumptions one must typically make about a signal or sensing mechanism in order for the math model to apply.

This is all well and good, but it's a heavy price to pay for what is, essentially, an abstraction. There are no infinities, computed (not computable) functions are really just finite-dimensional vectors taking values in a bounded range, any relationships between domain elements are discrete.

In this circumstance, most of the traditional heavy-duty mathematical machinery for signal processing is irrelevant -- equation solutions trivially exist (or don't) and the actual meat of the problem is efficiently computing solutions (or approximate solutions). It's still quite technical and relies on advanced math, but a different sort from what is classically the "aesthetic" higher math approach. Famously, it also means far fewer proofs! At least as apply to real-world applications. The parameter spaces are so large, the optimization landscapes so complicated, that traditional methods don't offer much insight, though people continue to work on it. So now we're just concerned with entirely different problems requiring different tools.

Without any further context, that's what I would assume your colleague was referring to, as this is a well-understood mathematical dichotomy.

heyitsguay··on Transformers seem to mimic parts of the brain
The professionals I've met actually working in ML R&D have basically all been very technically competent, including in mathematics. It's more the people who talk a lot about AI in grandiose and anthropomorphized terms that I was referring to.
heyitsguay··on Transformers seem to mimic parts of the brain
There are no actual ethical issues with contemporary ML architectures (including transformers) being too conscious or brainlike, it's just laypeople and demagogue chatter. It could be an issue in the future but only with computational systems that are utterly unlike the ones being created and used today.

Actual practitioners are tired of the debate because getting to an informed answer requires grokking some undergraduate-level math and statistics, and nobody seems particularly willing to do that

heyitsguay··on If You Want This Job, We Must Interview You Forever
My partner just got off the tech job hunt roller coaster last week - signed a solid offer at a great company with a team she's excited to join. It took months, she encountered some similar disappointments, felt the same doubts and fears, and I know how awfully hard it can be. Keep rolling the dice though! It's hard but it will pay off.
heyitsguay··on Gaudi: A Neural Architect for Immersive 3D Scene Generation
Cool! Any demos you can link to?
heyitsguay··on Detecting Deepfake Video Calls Through Monitor Illumination
The model here is about using a challenge window with an illumination sequence specifically designed to change faster than can be incorporated into live deepfake models on the fly without creating telltale artifacts from improper approximation of the illumination effects on the subject. Correctly faking the effects of monitor illumination is a non trivial challenge for well-chosen illumination patterns.
heyitsguay··on At the Bored Ape restaurant, your ApeCoin is no good now
Yeah but the hippie movement was about realizing the untapped potential for social freedoms locked away behind a repressive and outdated society. (Before it burned out and dissipated)

Crypto is about shallowly co-opting the patterns of discourse of genuine social movements to try and get rich quickly.

heyitsguay··on Returning to the office is costing you untapped talent
Alright, well if we have to take someone's personal work preference and make everyone do it even if they don't want to, let's pick WFH!
heyitsguay··on Returning to the office is costing you untapped talent
Just make it optional! If you wanna go into the office, go for it. If you wanna work from home, that's fine too!
heyitsguay··on A Google Employee Fell for the Eliza Effect
Agreed. One distinction that I've found people have trouble in debates regarding lamda is separating the idea that any billion parameter ML algorithm could be conscious vs lamda in particular being conscious. I think the jury's out on the former (I'd bet you could create a reactive embodied agent with learned environmental goals in <1B parameters given the right setup, which is at least a stepping stone), but for the reasons you describe here, lamda could only be considered conscious by a definition so broad as to declare all digital computation as conscious, and the reasons for that come down to understanding the network architecture and prediction task.
heyitsguay··on Show HN: Into the Futureverse
Another attempt at shoehorning artificial scarcity into VR platforms to make a quick buck. I can't tell if there's just a generation of engineers who grew up never realizing that web3's promise of "decentralized computing" already exists and is called "hosting whatever you want on your own servers". If you have the hardware to run a VR server (still a stretch), there's no need to strap an inefficient ledger to it on the off chance people want to pay for their ones and zeros; just run whatever cool VR thing you were planning to run already!
heyitsguay··on LaMDA is not sentient
You're still making generic arguments, nothing specific to the models in question. I'm a materialist, I believe consciousness is an emergent process that we definitely don't fully understand, but arises from connections of simpler material things. The reason why current-gen large language models aren't conscious (or at least no more conscious than a calculator, in which case you have a definition of consciousness that is not productive for discussing real world phenomena) comes from an understanding of how they work in particular. You don't need to know what each of a billion parameters is doing in an LLM to understand the computational envelope in which it operates, any more than I'd need to understand the ten billion parameters in a 100000x100000 convolution kennel to know that it's not conscious because they just get multiplied with some image region and added up.
heyitsguay··on LaMDA is not sentient
In these conversations there are a lot of people making assertions like the one you just made. What's the common trait y'all have? You don't understand how the system works. This isn't a matter of narrow-minded computer scientists incapable of conceiving of emergent phenomena, it's the uninformed filling in the blank spaces in their mind with "here be dragons".

It's not even a question of whether a computer program in general could become consciousness. It's a question of, if a key-value store with connection patterns derived from a large natural language corpus is conscious, then so is every other program run on that circuitry. Then my phone's keyboard's next word predictor is conscious.

In large language models today, words are turned into continuous token embeddings, embeddings are selected and recombined a few times (or many), and training data shapes the embedding space and recombination process to create a system that's good at predicting plausible next tokens in a sequence. The reason people who understand their operation can confidently assert they're not conscious is because they understand that that's all that's happening. There's no "there" there to give rise to anything deeper, unless you're also willing to say your computer's calculator app is also conscious.

Also as someone that's also studied a bunch of neuroscience, this is nothing at all like how the brain works. And yes, once again, the people most prone to philosophize like this on intelligence and consciousness don't understand how any brain science works either

heyitsguay··on Do Kwon in more trouble
For context, OP is CTO of an NFT company. Doesn't determine which way intent goes in this post, but you've gotta believe this to take such a strong financial stake in the space.
heyitsguay··on Artificial neural networks today are not conscious (Douglas Hofstadter)
The difference between GPT-3 and a Markov text generator is that GPT-3 is much better at handling long-term semantic correlations between text elements, but they're performing the same task.

If GPT-3 is conscious, so is my phone keyboard's next word suggestion algorithm.

heyitsguay··on Addressing AI’s energy cost
I don't think their scientific claims are all that inaccurate, maybe some stuff is simplified but that's inevitable. To my understanding, the issue is more converting that advanced scientific knowledge into something tangible.

Can I build a Google photos competitor using spiking neural nets for content recognition? Can i train GPT-4 with 50% of the power bill of GPT-3? Can i dare hope for better accuracy or robustness for some production task? Or better compressed nets for edge computing? I'm not seeing anything like that. We've known a long time that ANNs as implemented today are very crude compared to biological neurons; they really just share a name. What's supposed to differentiate a company from a lab is taking that insight and doing something concrete with it.

heyitsguay··on Addressing AI’s energy cost
You're right on the face of it, the added context that's missing is Numenta has been pushing bio-inspired AI for like 17 years and nothing practical has come of it. If using some energy to achieve a practical goal is wasteful, using less energy to accomplish nothing is more so.

I got into AI after a grad school career split between mathematical signal processing and computational neuroscience. I knew folks back in the early 2010's looking at joining Numenta. The ideas are absolutely good to explore, it's the execution that's lacking. Maybe their big breakthrough is just around the corner, but how long do you wait for product 1 alpha build 1 before calling out vaporware?

heyitsguay··on Addressing AI’s energy cost
Lol, this is a good angle for Numenta to take, but for all the time and brainpower they've spent so far you'd think they could come up with one thing that moves the needle in practice!
heyitsguay··on Embodiment is indispensable for AGI
"qualia" is the bikeshedding for any discussion on intelligence (biological or artificial). It's an unfalsifiable concept that's much easier to reach to than engaging in the concepts that are actually relevant to modern work in those fields.

Posting this is a bit of trying to hold back the tide, because the dominant threads in HN AI/BI discussions are almost always horribly uninformed, but I'm posting this so that if anyone else who actually does work in those fields is reading this and wondering what on earth is going on, now you know you're not alone!

heyitsguay··on Robocode Tank Royale
You say it's an ugly hack, but our +0.8% profit margins speak for themselves!
← PreviousPage 6 of 15Next →