mloncode

Yes that would be really cool. The field of program synthesis has made strides in this area but it doesn't appear you can create anything more than trivial programs from human languages at the moment. I think that you are more likely to see technology that augments the human significantly -- for example better code completion, error detection etc. that allows you to work much faster. It will be exciting to see how machine learning shapes developer tools and workflows in the future.

What is exciting about the tools we provided in this competition -- especially the weights and biases leaderboard is the level of transparency you get that you don't always see in a Kaggle competition (unless it is shared voluntarily). You can see:

- All the system logging information including CPU/GPU utilization, with the runtime and type of GPU card used - Extensive logging of model training progression - All of the model artifacts and metadata - A link to the code on GitHub with the code that ran that data. - Anything emitted to stdout (for logging) - etc.

This allows for extreme reproducibility and insight that is very helpful. With these tools, the community can see if an "ensemble everything: method is used and how long the model takes to train and what resources are consumed, etc.

Good question!

The unit of observation for this data are functions or methods and their associated docstrings or top-level comments. So there are no code reviews in the data. However, we do include metadata including the SHA and owner/repo which would allow you to retrieve this information! What were you thinking of doing with code reviews?

Hello folks, this is Hamel from GitHub -- I’m one of the Machine Learning Engineers who worked on this project. The reason we are excited to host this data is that we believe the community will be able to innovate and advance the state of the art much faster if it is provided in a tractable format for machine learning researchers. This data is already public, however there is specialized knowledge required to acquire, parse, dedupe, and clean code from many programming languages at a massive scale on GitHub. We strived to reduce these barriers to encourage greater involvement.

While we present the task of information retrieval as one possible use case of this dataset, we know there could be other practical applications of this data (i.e. code summarization). While we went through great lengths to pre-process the data for the community, the data is still messy and often you will find that there might not be high-quality comments that are aligned with a code snippet we parsed. However, we believe this is part of the excitement of the dataset it poses challenges that machine learning practitioners will have to address.

Code is very different than natural language with regard to its structure and syntactic rules and may benefit from different approaches relative to natural language processing. Our baseline models and benchmarks mostly treat code as natural language, however we are aware that there could be an opportunity to innovate on this front. If anyone creates any interesting projects from this dataset, please do get in touch. Happy to answer any questions!

Oh and https://towardsdatascience.com/semantic-code-search-3cd6d244... is completely open source end-to-end, with code, data, and detailed explanations on how to reproduce step-by-step.

Hello there! Glad to see other people working on this area as well. Ho-Hsiang and I from GitHub have been prototyping this exact same approach and have published / open sourced our work about a month ago: https://towardsdatascience.com/semantic-code-search-3cd6d244...