samvher

Haha fair question. I didn't make any special changes, I just left the lid open and put the laptop in a ventilated spot. I'm actually in the tropics, so I guess Lenovo scores some points here (the laptop is a Legion 5 Pro).

It's all behind the submission link! I've set it up so that you can run it start to end, if you want. The only thing I'm not 100% sure about is resource requirements - I have an 8GB GPU and 32GB of RAM, it could be that if you have less than that you'd run into out of memory errors. Those would be fairly straightforward to fix, though (honestly I'd be happy to help if someone runs into this).

I built and trained a BERT on my gaming laptop (3070 RTX) to ~94% of BERT-base's performance in ~17 hours* (BERT-base was trained on 4 TPUs for 4 days). This notebook goes over the whole process, from implementing and training a tokenizer, to pretraining, to finetuning. One feature that makes this BERT different from most (though not unique) is the use of relative position embeddings.

Edit - for anyone unsure about what "BERT" is or its relevance, it's a transformer based natural language model just like GPT. However, where GPT is used to generate text, BERT is used to generate embeddings for input text that you can then use for predictive models (e.g. sentiment prediction), and that process is also demonstrated in the notebook.

*Edit 2 - The 17 hours are pretraining only, not including the time to train the tokenizer, or finetuning.

Can you say more? It sounds like you're referring to more than just the contamination of the word "open".

Interesting idea. I'd argue that while salary correlates with value, it's not the same though. It seems most likely to me that this mechanism would just reserve the privilege of employing immigrants to a few high income sectors, rather than filter the most skilled immigrants evenly across sectors.

To make it more concrete, it would be nice if NGOs could hire foreign talent just as effectively as Google can.

I don't like the exclusive network of countries either... I guess we have different perceptions of what's fair/acceptable.

I'm from a non US country, so the concept of a Harvard is foreign to me as well. And yes, I've also worked for selective employers where maths degrees mattered a lot. It never bothered me.

Not sure what you're trying to say. Hiring people based on individual merit and skills is not the same thing as hiring people based on an exclusive network. The Harvard employer favored Harvard folks over others who otherwise had the same (or better) skills.

I have worked in an environment where it really did seem to matter. An organization set up by Harvard folks where other Harvard folks seemed to have advantages in (a) getting in and (b) getting ahead. It wasn't ever really addressed explicitly (though "credentialism" was occasionally mentioned by others as well, I wasn't the only one who noticed). It definitely played a role in my leaving that organization.

Edit - I do feel like I should mention that the organization was improving on this front when I left.

If you use vim every day, the common operations become muscle memory, I don't even think about jumping through the file, find and replace, block edits etc, they just happen.

For me personally, the context in which I edit files changes constantly - sometimes it's inside a Docker container, sometimes it's on a server, sometimes it's on my personal Linux machine, sometimes on my work Mac... Vim is easily available on all of those, and I just use the default configuration.

I do see my colleagues benefit from VSCode. If my work environment was very consistent (same computer, same language) I think you're right that I would possibly gain in productivity from using VSCode (with Github Copilot, perhaps). But my work environment is not consistent, so Vim it is.

Thanks a lot for the recommendation, read this over the weekend and learned a bunch. If you have other papers to recommend in similar categories I'm all ears :)

I found this lecture and the one following it very helpful as well: https://www.youtube.com/watch?v=ptuGllU5SQQ&list=PLoROMvodv4...

It's not so much that a different setpoint is worse per se "at steady state", it's that the world has been in a fairly steady state for quite a while now, with little variation, and our structures have grown to depend on that particular state. Think of the locations of cities, the allocation of agricultural crops in given regions, the whole economic system built up around that, etc.

Essentially we've been able to rely on things being a particular way for ~10,000 years or so, and if things keep going the way they are everything will (a) be different from what we've grown to expect and (b) be changing unpredictably for a while to come, meaning we can't just set ourselves up for a new configuration.

This is the way I understand it, anyway.

What would be the set of skills that would put you in the category of LLM talent that is in extreme short supply?

Just curious what the current bar is here and which of the LLM-related skills might be worth building.

Care to expand on your thinking here? In what way do you think these recipients would be cherry picked, and what do you think would be the political goals?

This Wikipedia page is worth a read: https://en.wikipedia.org/wiki/Interlocking_directorate

I recently had the same questions and here is how I understand it:

1. You could concatenate the positional embedding and the semantic embedding and that way isolate them from each other. But if that separation is necessary, the model can learn the separation itself as well (it can make positional embeddings and semantic embeddings orthogonal to each other), so using addition is strictly more general.

2. My sense is that you could merge the Q and K matrices and everything would work mostly the same, but with multi-headed attention this will typically result in a larger matrix than the combined sizes of Q and K. It's basically a more efficient matrix factorization.

Curious to see if I got this right and if there is more to it.

Just to check if I follow - is this a red flag because these founders are then not properly incentivized, because their ownership is a sure thing? (Although I guess you could then still argue that the incentive is growing the value of their share.)

1) Is everything really being mined already? My sense is that another round of GPT-like training on YouTube, EdX and Coursera data + some other large video archives (BBC and the like) could still make quite a bit of difference. Text and images independently is one thing, having them together in context might be something else.

2) The available power seems to be growing pretty rapidly and dropping in prices. I think there are still quite some gains to be had from architectural optimizations (both in hardware and in models).

4) They were defined decades ago but I think they did actually seem to move us closer to AGI only recently.

You might be right, and there are definitely interesting times ahead! But I kind of doubt that we will have decades to sort out what we have and figure out its impact on society (which is a bit scary).

Why is that, that they are key to performance with RISC-V? And why are they the reason not to have conditional moves? Would love to know more!

Similarly there's an open Yale course "Listening to music": https://oyc.yale.edu/music/musi-112/lecture-1

I enjoyed it, it goes into quite a bit of music theory and many different musical styles, with a lot of listening exercises.

As a sibling comment said, of course you don't need this to enjoy music, but I did feel like it deepened the experience in some way (and maybe made it a more analytical experience simultaneously - which may or may not be what you want).

> An important component of credit courses is the time limitness

Huh? Why would that be an important component?

I mean I know that it's typical for courses that come with credit, but in my mind it's just a side effect of the administrative side of providing courses, not something that's at all central to accreditation. I.e. if you build up the knowledge over a longer period (which you can still do with e.g. resits) that's completely fine as well.

What's /g/?

How do you show that? A large portfolio of successful projects/many years of experience?

Interesting - if you do manage to formulate it I'd be curious to read that.

I like how they're taking a proactive approach in trying out what role ChatGPT and the like can play in generation and exploration of knowledge. I like the breakdown of the different ways of interacting with the tool, and the callout of how collaborative editing seems like the best out of the approaches they've witnessed. I also feel somewhat comforted by the remarks about how students engage with the output of the tools critically and fact checking can actually be a valuable way to interact with the material.

Personally I think I would enjoy this class - I like the openness to experimentation. I can imagine that _getting graded for it_ could be a frustrating experience though, I think with experimental classes that's a kind of thing for which there tends to be little standardization and it can be unpredictable how you'll be judged.

I'm Dutch so I don't know to what extent my experience applies here and what it will do to your expectation on outcomes (teacher is a decently paid job with us, though definitely not high paying). The people I know in education are generally highly caring and work way more hours than their contract asks them to. When I was in school, I've definitely also had teachers that were not so good - in some cases that may have been because they were in a field where teaching was one of the few feasible careers (e.g. Latin/Greek) and it was not a fit for them, in other cases maybe there was also a lack of competition or accountability to an extent. Teaching is definitely viewed as a safe/fallback career option, but there are also lots of people who choose it specifically because they are passionate about it.

> as attested by actual professionals usually taking the top comment to correct the majority of

Got a bit confused here, what is this sentence?

I agree that the band on HN is somewhat narrow, though I feel it is a distinctly different band from most other places which makes it worth visiting.

Any subreddits in particular you recommend for high quality discourse?

It's not obvious to me why it would be very beneficial to have a data center close to an Intel plant, but it sounds a bit like it might not be a coincidence that they're close together. If someone knows why this pairing makes sense, care to explain?

(The closest thing I can think of is low delivery costs on electronics, but that seems really tiny compared to basically anything else that matters here, and it's not like they would typically get delivered straight from a plant, so that doesn't seem like it would be it.)

I wonder when we will get to a point that network effects start to dominate in this space, rather than tech/research. It does seem like (a) ChatGPT being publicly available, (b) it learning from user interactions and (c) it having generated a significant level of hype could put it in a position where it pulls away from competition because there is a flow of user information coming in that might allow it to learn faster and better than competitors. Maybe this can't be compensated for just by scraping more data from the web.

I'm not familiar enough with how they've implemented part (b) to be able to judge how effectively they're doing this though.

No offense but this approach worries me - it seems like a novel mechanism to (perhaps inadvertently) generate and spread false information. It takes a lot of fact checking to make sure everything is right, and if you do the research yourself that's a natural part of the process. It seems way too easy to minimize that effort in a process like this.

I was already worried about ChatGPT-like systems generating mass-produced nonsense and polluting the internet, but if people are also going to edit ChatGPT output just enough to make it seem right (a mechanism I hadn't thought of so far), that might make the nonsense a lot harder to detect.

I totally understand the reasoning though, it sounds like a productive workflow.

Software Foundations is a series of interactive books where you work through proofs in Coq, gradually increasing in complexity: https://softwarefoundations.cis.upenn.edu/

This is a bit on the edge of the domain that you're asking about (not really for beginners, and proofs are perhaps a somewhat niche type of programming) but I learned a lot from this and many people don't seem to know it, so I think it belongs in the list.