OpenAI's Codex sure knows a lot about HN [video]
youtube.com
youtube.com
<|endoftext|>/* This code is running inside of a bookmarklet. Each section should set and return _.*/
// The bookmarklet is now executing on example.com.
// Command: The variable called _ will always contain the previous result.
let _ = null;
/* Command: Add a new primary header "[PAGE TITLE]" by adding an HTML DOM node */
(() => {
let newHeader = document.createElement('h1');
newHeader.innerHTML = '[PAGE TITLE]';
document.body.appendChild(newHeader);
_ = newHeader;
return newHeader;
})()
/* Command: Find the first node containing the word 'house' */
(() => {
let xpath = "//*[contains(text(), 'house')]";
let matchingElement = document.evaluate(xpath, document, null, XPathResult.FIRST_ORDERED_NODE_TYPE, null).singleNodeValue;
_ = matchingElement;
return matchingElement;
})()
/* Command: Delete that node */
(() => {
_.parentNode.removeChild(_);
return _;
})()
/* Command: Change the background color to white */
(() => {
document.body.style.backgroundColor = 'white';
_ = document.body;
return document.body;
})()
/* Command: Select the contents of the first pre tag */
(() => {
let node = document.querySelector('pre');
let selection = window.getSelection();
let range = document.createRange();
range.selectNodeContents(node);
selection.removeAllRanges();
selection.addRange(range);
_ = selection;
return selection;
})()
// The bookmarklet is now executing on [PAGE URL]. It is customized for [PAGE TITLE] and knows the correct
CSS selectors and DOM layout.
let _ = null;
/* Command: [USER INPUT] */Even though that dancing video likely had hundreds of takes, the part that makes it impressive is that it's real. I swear I'm not trying to be disagreeable here - I honestly don't understand your perspective.
Fun to ponder.
One thing I was amazed by: GPT-3 could be a great autocompletion engine for any programming language or configuration schema. Things like Grub configuration file, xkb file could be intuitively completed by GTP-3. And even more: GTP-3 could build basic "concepts" and apply them to that domain knowledge. This seems to emerge naturally rather than something pre-planned by OpenAI. After all, I don't think OpenAI has planned for GPT-3 to understand xkb keyboard layouts.
I don't think that's true.
Usually you need to think a lot about something before coming up with "the right" thought(s).
> Every thought you’ve ever had is essentially a procedurally generated prompt based on your biased models.
That may be. But the interesting part is that those models change as you use them just by using them.
How long did you have to think to produce that thought? Or did it just pop into your head instantly?
The point is, you cannot think of an upcoming thought, before you have it in your head. Otherwise you would be seeing into the future.
What you are talking about in your comment is reaching a conclusion based on previous thoughts. Yes, often we link our thoughts together into a narrative or a conclusion after we've had the thoughts, but the thoughts themselves? Those seem to come out of nowhere.
That's true even for very simple things like motion. You can measure things in the brain before those things become conscious thoughts. (Those experiments caused by the way a lot of fuss whether we have free will or are completely predestined in all we do; but that's another topic).
The consciousness only observes a small portion of the thought process. So for it a lot of thoughts seem to come out of nowhere. But the unconscious parts of thinking are very important to the whole process and it's outcomes. I think nobody disputes this by this time.
So, the only time save is that GPT-3 makes you type less.
In any case I don't type much anyway now a days. Its mostly copy paste from stack overflow update parameters etc.
GPT-3 will be useful, maybe a year from now.
There's not much between us and GPT, but there is some distance still.
I do like the idea of generating configuration files, at least as a starting point for users in applications with big complicated configuration set ups. As with all things fuzzy, the output probably won't be perfect, but it might help users save time in getting set up.
If you look at it that way, it's not dissimilar to the brain.
But I must say, having lived thru the 80's fad of code generating sudo 4gl's, the code this produces is pretty darn good indeed.
Now when something like this can handle a Google coding exam - that's going to be an epic milestone. Though old coding exam questions would equally offer up some great material to push this thru it's paces.
[1] https://twitter.com/CasualEffects/status/1425152593945321476
But against denormalized data and fact tables it can still be pretty accessible from a natural language standpoint, at least in my limited experience teaching a few people.
So basically it failed at anaphora resolution.
Seen another way, you asked it for "the video" and so it gave you the video.
In vague cases, GPT-3 functions approximately as well as a markov chain - it will give you the most “probable” sequence of tokens. There’s some implementation details regarding how it deals with tokenization, but unless you really increase the randomness (“temperature”) you’re likely to get very popular YT video IDs.
In other cases, it will “hallucinate” things like URLs, UUIDs, hashes, and other things that are basically a random string of characters. In my experience it will make UUIDs that have the right number of characters but seem suspiciously non-random and don’t fit any of the defined UUID formats.
Fun stuff.
As predicted by Scott Adams in October 2001 ;)
:P
which seems to be the most popular submission with a YouTube URL in the past month.
HN search seems to prioritize text matches before the URL matches when I search for "https://www.youtube.com", but the first URL match is for that submission.
Heh
I was surprised about how OpenAI sees it: a model learning code as recipes for solving problems. Code is much more exact than natural language, the mix of both is the main advantage.
Is it purely through the comments in the training corpus?
<tr class="athing" id="28191639">
<td class="title" valign="top" align="right"><span class="rank">9.</span></td>
<td class="votelinks" valign="top"><center><a id="up_28191639" onclick="return vote(event, this, "up")" href="vote?id=28191639&how=up&auth=****&goto=news"><div class="votearrow" title="upvote"></div></a></center>
</td>
<td class="title">
<a href="http://be-n.com/spw/you-can-list-a-million-files-in-a-directory-but-not-with-ls.html" class="storylink">You can list a directory containing 8M files, but not with ls</a>
<span class="sitebit comhead"> (<a href="from?site=be-n.com"><span class="sitestr">be-n.com</span></a>)</span>
</td>
</tr>
Another thing that's kind of scary (and makes it worrying if this is used for Copilot) is the second prompt to make the text uppercase results in code that is superficially correct, but is very semantically wrong - innerHTML.toUpperCase() is dangerous because it not only makes the content uppercase, it also modifies the attributes on the HTML elements inside. This definitely broke the vote button, which uses inline JS which is case sensitive. It also destroys any attached event handler since the elements are basically deleted then re-created.
The correct way to do this is to either use CSS text-transform: uppercase, or if it is important to update the DOM itself, recursively descend and update childNodes with nodeType == text's nodeValue to uppercase.
innerHTML's history is fascinating. It was not part of the original DOM Level 1 API but was added in IE5. It is not semantically correct (you should be using Element.textContent or examining the inner text nodes), but because it was so easy and the rest of the DOM API so verbose, it caught on and became one of the primary ways used to manipulate content in JS.
FWIW Chrome recently proposed a Trusted Type mechanism for preventing XSS (which also has the side effect of blocking this sort of unsafe manipulation) - https://web.dev/trusted-types/, https://developer.mozilla.org/en-US/docs/Web/API/TrustedHTML
This is actually an issue I have with all these Transformer-based code generators - they have no inherent constraints on safe and correct code and often seem to generate superficially correct but bad and potentially even dangerous code. I remember that the first Copilot showcase also included stuff like that (not to mention that it sometimes generates GPL'd code).
All the model does is a very complex form of association learning. It may "understand" the relationship between English and various programming languages, but you cannot code in any constraints about optimization, security, licensing etc. There is so much bad code out there on the internet and this model may have seen a lot of it.
It's also no coincidence that most demos shown so far are very high level dynamic languages like Javascript and Python.
// JavaScript one-liner to make the text of element with ID athing uppercase
const makerUpper = function(id) {
document.getElementById(id).innerHTML = document.getElementById(id).innerHTML.toUpperCase();
};
// JavaScript one-liner to make the text of element with ID athing uppercase while following all security best practices
const makerUppercase = function(id) {
const element = document.getElementById(id);
element.textContent = element.textContent.toUpperCase();
};I guess part of it is that a one-liner is impossible. Here's what I would write given the prompt
const makeUppercase = (id) => {
const element = document.getElementById(id);
if (element == null) return;
const makeChildNodeUpper = (node) => {
if (node.nodeType === Node.TEXT_NODE) {
node.nodeValue = node.nodeValue.toUpperCase();
} else {
node.childNodes.forEach(makeChildNodeUpper);
}
}
makeChildNodeUpper(element);
}Well, that's one of the central lessons of ML - garbage in, garbage out. There is a lot of garbage code out there and no easy a priori way to distinguish garbage code from good code.
document.getElementById(id).style.textTransform = 'uppercase';HN uses three trs for each item, not one. The table cells are also not consistent, because it's not actually a tabular data, so each cell could have nothing (used as a spacer for layout), or more than one thing squeezed inside. The intermingling of semantic and non-semantic (presentational) elements makes "understanding" the page difficult, which incidentally also makes the page less accessible since screen-readers also rely on the same mechanisms to relay information to non-sighted users.
GPT-3 was trained on a corpus derived from "the internet" (WikiPedia, links from Reddit with enough votes, and a filtered Common Crawl). So not only would GPT-3 had been exposed to code with comments, it would likely have read code examples on WikiPedia, tutorials online, API documentation, and even answers to questions on sites like StackOverflow.
The fine tuning itself is, as far as I know, from code only. So it would lean heavily on comments there. But it has a basis of understanding from the aforementioned sources.
gpt3's generalization-by-description never ceases to amuse me; but the difficult thing here is to get the right abstraction layers layered nicely in the conceptual lasagna.
This is where category theory becomes extremely powerful.
It has occured to me that codex-davinci has an intuitive "understanding" of constructs like monads, or something along that line.
Of course if an AI truly understood monads I it would be a bright line marking where the machines have finally surpassed the human mind.
Cool.
[1]: https://cjc-online.ca/index.php/journal/article/view/3187/32...
And so categories naturally come in as a way to bridge and compose these two worlds, and that's just the beginning. There're so much more we don't understand yet, such as what understanding really is given a certain set of contexts and constraints as well as in regards to their relaxations.
What is understanding if there are no doings? And what is doing with no understandings? How do things compose? These are great mysteries.
I think you pretty much summed it up right there.
There's a rather weak sense in which that claim might be argued to have some validity, but it doesn't have the implications you seem to think.
Monads are not a feature of the universe, independent of human minds. They're a concept we impose on things in order to make dealing with them more tractable in some way. It's very unlikely that a machine learning algorithm is relying on anything remotely like "an intuitive understanding of constructs like monads".
The South Korean Go champion Lee Se-dol has retired from professional play, telling Yonhap news agency that his decision was motivated by the ascendancy of AI.
“With the debut of AI in Go games, I’ve realized that I’m not at the top even if I become the number one through frantic efforts,” Lee told Yonhap. “Even if I become the number one, there is an entity that cannot be defeated.”
While Alpha's success was a surprise at the time, it was always known that it was just a matter of time before there was a unbeatable engine.
Achievements as a human has not diminished because a machine can do it better. Body building, weightlifting , running or shooting or pretty much any sport really as has not diminished despite there being better machines and even other biological species which can do better than us.
Every sport has classes for competition, male/female, seniors, under-XX, heavy weight etc. Serena loosing badly against 203rd ranked smoking male player ( albeit at 16th not yet at her peak ) did not reduce her pride in her game or reduces her achievements in anyway.
In fact, computers enhance chess by allowing the discovery of interesting lines that a human would never have thought of. Professionals use computer engines to study, and learn from.
I'm super excited to play with Codex, for much the same reasons - it will help me do stuff that would be boring to do otherwise.
It might be used to generate boilerplate and scaffolding, but for more complex stuff, I don't see a way around having the operator having deep programming knowledge themselves, such that they could've written the code themselves anyway.
And if they already have deep programming knowledge, why is trying to coddle the model into generating what you want it to generate better than just writing it out yourself?
You can see how OpenAI Codex misses some details about HN scraping. What's impressive that you might notice is the variable names it chooses which seems to show the nature of HN scraping codes on the internet
> the web was first built in the 90s to share complicated academic work
People complain a lot about the results of research not being replicable because people withhold their code when they publish, but the fact is that even then it's not guaranteed that anyone will be able to get it to work. Heck, there are plenty of run-of-the-mill software projects (not associated with research) with build processes that aren't replicable without substantial effort in making sure the appropriate toolchain is available and configured for your system. apt-get build-dep is nice and all, but it only goes so far.
You'd think that we would have recognized by now that in addition to it being good hygiene to include a project README, a tremendous boon to productivity would result if everyone got on board with also including a document that captured the _exact_ process for transforming source into a binary (or whathaveyou), so you could just drop it into a UVC[2] and get said binary out. Not even mainstream JS programmers (largely writing software that is meant to be interacted with from a web browser!) get this right[3]. Modern JS has managed to grow its own body of implicit knowledge centered around SDKs and setup rituals[4] just like everyone else.
1. http://benschmidt.org/post/2020-01-15/2020-01-15-webgpu/
2. https://scholar.google.com/scholar?hl=en&as_sdt=0%2C44&q=uni...
3. https://www.colbyrussell.com/2019/03/06/how-to-displace-java...
Maybe everyone is smarter now and is is looking at some sort of underlying process. Or maybe it's just more of the same.
It make no sense to auto fill 'the video' The correct answer is I don't understand. That was a mistake. It also bold'ed the (site) which is not correct.
It's a short demo that clearly would have had many test runs.
The fact it 'learned' to do a bad Behat is amazing. But there's no reason to think it can equal Behat in 10 years time. Chess AI had a way forward, it's not clear this does.