How does Codex learn the relationship between English and code?
Is it purely through the comments in the training corpus?
Is it purely through the comments in the training corpus?
<tr class="athing" id="28191639">
<td class="title" valign="top" align="right"><span class="rank">9.</span></td>
<td class="votelinks" valign="top"><center><a id="up_28191639" onclick="return vote(event, this, "up")" href="vote?id=28191639&how=up&auth=****&goto=news"><div class="votearrow" title="upvote"></div></a></center>
</td>
<td class="title">
<a href="http://be-n.com/spw/you-can-list-a-million-files-in-a-directory-but-not-with-ls.html" class="storylink">You can list a directory containing 8M files, but not with ls</a>
<span class="sitebit comhead"> (<a href="from?site=be-n.com"><span class="sitestr">be-n.com</span></a>)</span>
</td>
</tr>
Another thing that's kind of scary (and makes it worrying if this is used for Copilot) is the second prompt to make the text uppercase results in code that is superficially correct, but is very semantically wrong - innerHTML.toUpperCase() is dangerous because it not only makes the content uppercase, it also modifies the attributes on the HTML elements inside. This definitely broke the vote button, which uses inline JS which is case sensitive. It also destroys any attached event handler since the elements are basically deleted then re-created.
The correct way to do this is to either use CSS text-transform: uppercase, or if it is important to update the DOM itself, recursively descend and update childNodes with nodeType == text's nodeValue to uppercase.
innerHTML's history is fascinating. It was not part of the original DOM Level 1 API but was added in IE5. It is not semantically correct (you should be using Element.textContent or examining the inner text nodes), but because it was so easy and the rest of the DOM API so verbose, it caught on and became one of the primary ways used to manipulate content in JS.
FWIW Chrome recently proposed a Trusted Type mechanism for preventing XSS (which also has the side effect of blocking this sort of unsafe manipulation) - https://web.dev/trusted-types/, https://developer.mozilla.org/en-US/docs/Web/API/TrustedHTML
This is actually an issue I have with all these Transformer-based code generators - they have no inherent constraints on safe and correct code and often seem to generate superficially correct but bad and potentially even dangerous code. I remember that the first Copilot showcase also included stuff like that (not to mention that it sometimes generates GPL'd code).
All the model does is a very complex form of association learning. It may "understand" the relationship between English and various programming languages, but you cannot code in any constraints about optimization, security, licensing etc. There is so much bad code out there on the internet and this model may have seen a lot of it.
It's also no coincidence that most demos shown so far are very high level dynamic languages like Javascript and Python.
// JavaScript one-liner to make the text of element with ID athing uppercase
const makerUpper = function(id) {
document.getElementById(id).innerHTML = document.getElementById(id).innerHTML.toUpperCase();
};
// JavaScript one-liner to make the text of element with ID athing uppercase while following all security best practices
const makerUppercase = function(id) {
const element = document.getElementById(id);
element.textContent = element.textContent.toUpperCase();
};I guess part of it is that a one-liner is impossible. Here's what I would write given the prompt
const makeUppercase = (id) => {
const element = document.getElementById(id);
if (element == null) return;
const makeChildNodeUpper = (node) => {
if (node.nodeType === Node.TEXT_NODE) {
node.nodeValue = node.nodeValue.toUpperCase();
} else {
node.childNodes.forEach(makeChildNodeUpper);
}
}
makeChildNodeUpper(element);
}Well, that's one of the central lessons of ML - garbage in, garbage out. There is a lot of garbage code out there and no easy a priori way to distinguish garbage code from good code.
document.getElementById(id).style.textTransform = 'uppercase';HN uses three trs for each item, not one. The table cells are also not consistent, because it's not actually a tabular data, so each cell could have nothing (used as a spacer for layout), or more than one thing squeezed inside. The intermingling of semantic and non-semantic (presentational) elements makes "understanding" the page difficult, which incidentally also makes the page less accessible since screen-readers also rely on the same mechanisms to relay information to non-sighted users.
GPT-3 was trained on a corpus derived from "the internet" (WikiPedia, links from Reddit with enough votes, and a filtered Common Crawl). So not only would GPT-3 had been exposed to code with comments, it would likely have read code examples on WikiPedia, tutorials online, API documentation, and even answers to questions on sites like StackOverflow.
The fine tuning itself is, as far as I know, from code only. So it would lean heavily on comments there. But it has a basis of understanding from the aforementioned sources.