fantispug

> Practical/applied stuff seems to require levels of compute, training data, etc. that make it unapproachable for a home hacker for everything beyond basic exercises like a handwritten digit classifier.

Compute is only an issue if you want to train large foundational models from scratch. You can easily train non-neural models on CPU, and fine-tune smaller models (say <4B) on a commodity GPU.

For training data there's lots of datasets online (like Kaggle), but you can build your own and can label things yourself or get a language model to label them for you. Getting good data is the hardest part of ML so this is a good exercise.

Then do some hobby projects: classify whether your photos contain a photo of you or a particular friend/family member, predict the expense category from bank transaction exports, classify your emails into categories, extract book recommendations from Hacker News, ...

Multi layer perceptron. A tiny neural net.

Boosted trees over embeddings work surprisingly well too.

Their strength was the ability to keep at the very fast moving AI frontier.

When a new model is released they have been able to have a working implementation days later, not months when the model was redundant. And often it's much more usable than the one-off, unmaintained, research code written in a very fragile environment.

BLOOM was a decent fully open and very large model for its time, and a huge coordination effort. It was around the same time as another "joke", OPT, but that gave Meta the knowledge that let them build Llama. Huggingface didn't have the money to compete in that race, but I'd bet it helped them stay at the frontier of AI software.

Google could have owned this space; they had TensorFlow hub, but it was too hard to use and too locked down. Huggingface made it easy to run and train transformers, and got the timing just right.

I'm glad someone is trying to build truly open models (along with the great work at Allen AI), because the likes of Meta and Alibaba are only going to release weights as long as it suits their business.

Yes, this seems to be a common capability - Anthropic and Mistral have something very similar as do resellers like AWS Bedrock.

I guess it lets them better utilise their hardware in quiet times throughout the day. It's interesting they all picked 50% discount.

Meta's primary business is capturing attention and selling some of that attention to advertisers. They do this by distributing content to users in a way that maximizes attention. Content is a complement to their content distribution system.

LLMs, along with image and video generation models, are generators of very dynamic, engaging and personalised content. If Open AI or anyone else wins a monopoly there it could be terrible for Meta's business. Commoditizing it with Llama, and at the same time building internal capability and a community for their LLMs, was solid strategy from Meta.

I find this style changes the way I think about and write code transformations. It's also in shell pipelines, R's magrittr, and Clojure's thread macros, and can be emulated in some OO languages with methods that return the transformed object itself.

I have seen it work better than LSH.

Each time you embed a document you search for approximate nearest neighbours before adding it, so it is O(N) like MinHash. Vector indexes like HNSW and PQ have better performance/quality tradeoffs than SimHash LSH which is the analogue of MinHash for cosine distance.

The quality depends on what you mean by near duplicate and the embedding model you use. Current models work well, and if you have labelled data you can fine tune them to be better.

The main drawback is the additional cost of embedding all the documents, especially for longer documents. But this cost has dropped really quickly with smaller models, better optimisations, and faster hardware.

It covers a lot of the fundamentals in some detail (attention and transformers, decoding, transfer learning) that are underneath current cutting edge NLP; this is still a very good foundation likely to be good for several more years.

What might be missing is in-context learning, prompt engineering, novel forms of attention, RLHF, and LoRA (though it covers adaptors), but this is still changing rapidly and the details may be irrelevant in another year. If you have a look at a recent course like Stanford CS224N 2023 there's a lot of overlap.

> Maybe my data structure has a hash table of items, as well as a direct link to the largest item. When I say: "write the function to insert a new item in the list, and remember to update the largest item if it is larger than the current one", would Copilot do the right thing? Each step is easy in itself (adding an element to a hash, comparing an item to another one).

In general it won't solve all your problems, but it's helpful for automating simple things like this (but you still need to test edge cases). With this prompt in Python (which I'm more familiar with):

  from dataclasses import dataclass
  from typing import TypeVar
  
  T = TypeVar('T')
  
  @dataclass
  class MaxDict:
      items: dict[T, float]
      max_value_item: T

      def add_item(

It completed:

    def add_item(self, item: T, value: float):
        if value > self.items[self.max_value_item]:
            self.max_value_item = item
        self.items[item] = value

This was my second attempt; first I called it `max_item` and the completion did something about comparing the key.

With the prompt

  // Print the fibonnaci sequence, except the number 5.
  int main() {

I got the result:

    int a = 0;
    int b = 1;
    int c = 0;
    while (c < 100) {
        c = a + b;
        a = b;
        b = c;
        if (c != 5) {
            cout << c << endl;
        }
    }
    return 0;

This happened in mathematics about a decade ago when Wolfram Alpha came out. Lecturers started complaining that Wolfram Alpha could solve assignment problems with worked steps.

In both cases I think this is a real opportunity; we can let students get more quickly to bigger problems and systems thinking by leveraging these tools. It requires professors to start thinking innovatively about how to teach and assess these subjects.

These issues are often interactions between packages. I use TRAMP daily without freezing, but only when I disable company mode in shell modes. VSCode certainly requires much less of this tweaking, even with a variety of extensions installed.

The killer feature of racket is it is very easy to make Domain Specific Languages (including the teaching language) and related tooling. However last time I looked the library ecosystem didn't seem great; there were many libraries but few that were actively maintained.

Kaluza-Klein theory is an attempt to exhibit a transformation that charged particles travel on "straight" lines in a 5-dimensional space. The fifth dimension is supposed to be a very small circle. There are some experimental implications, but the circle is so small that we can't detect them (it's similar with string theories).

String theory tries to do the same thing with other forces; so it's not clear whether there's such a transformation. Forces are technically simpler than embedding extra dimensions that there are no evidence for.

I'm the same; I use evil-mode in emacs and find the friction of switching between them small.

I've thought about a variant of this with Baking Pi [1]. That takes you through assembly to the point of having a Raspberry Pi with a minimal terminal (with just an external USB driver the author wrote).

I think it would be interesting to see how quickly you could get from there to a useful system, say one where you could download the USB driver from github and compile it. But I definitely don't know enough to do that.

[1] https://www.cl.cam.ac.uk/projects/raspberrypi/tutorials/os/

This may make sense if you want to do image processing and deep reinforcement learning. But there are lots of other domains.

For tabular data (which is probably most relevant in Pharma, and probably the best place to start) Introduction to Statistical Learning by Hastie et al and Max Kuhn's Applied Predictive modelling cover a lot of the classical techniques.

For univariate time series forecasting "Forecasting Principles and Practice" is great.

For natural language processing foundations Jurafsky's Speech and Language Processing is broadly recommended; for cutting edge natural language processing Stanford's CS224n is great: http://web.stanford.edu/class/cs224n/

If you like books and you want to deeply understand ML techniques I'd suggest jumping straight into "Introduction to Statistical Learning" and only learning calculus/stats/matrix methods (linear algebra) as you need them (you really don't need much from them in practice).

But it's ok to start using libraries and fitting models without understanding how they work deeply, and coming back to these books later (just make sure you come back; there's lots of useful ideas in them!) In which case I'd recommend some of the resources the parent doesn't recommend

Without experience in ML it's often hard to know what problems are solvable, how to frame the problem, and to tell a good solution in Github from a bad one, etc.

If you want to go an applied route I'd suggest starting somewhere like Kaggle and looking through the competitions for ones vaguely similar to yours. They've done all the hard work of choosing a challenging but solvable problem, sourcing and splitting the data, and choosing a metric. You then can see what techniques actually work really well, and benchmark different approaches. Academic challenges like Imagenet or Coco are also good for this, but you'll have to work harder to find relevant resources.

Once you've done this a couple of times, you can start framing your own problems, collecting and annotating your own datasets, deploying and maintaining models.

They do mention Penrose notation in the linked arxiv paper[1], as well as Cvitanović's Birdtracks[2] which let you do calculations on representations of groups.

They claim they're making it more accessible with pedagogical examples (and innovations in representing differential vector calculus).

I still prefer index notation. Their notation looks pretty intricate and I think it would be really hard to do long calculations in it. It's also not clear to me how you'd generalise it to non-commutative vector algebras like in quantum mechanics.

[1] https://arxiv.org/pdf/1911.00892.pdf

[2] http://birdtracks.eu/

Circa 2009 I worked on an application that would render maths tests for teachers using plain TeX. I'm sure there are a few other wild examples (my CV is in eplain TeX).

My experience is pip and virtualenv (or venv in 3.3+) works pretty well in both interactive and deployment contexts. My main gripe is Pip doesn't resolve conflicts between downstream dependencies (which I could only get a stable environment from by mapping dependencies and aggressive version locking). Conda has some advantages for certain use cases (handles libraries with installation dependencies well, curated repository), but I've never felt a need to use poetry or pyenv. Why do you have so many versions of python in production? I'd be surprised if now there are many packages that only work on specific versions (especially on 2.7 point releases).

I don't have much experience with packrat - but as opposed to pip it's another thing you need to discover and install. And so people don't do it by default when releasing code, and I've had to bisect versions of dependencies to get a working version of code. This can happen in Python too, but is rarer.

This is missing the most important difference - deployability. R was built as a language to use interactively and does things like raise warnings for things that should be errors, requires an external package (packrat) for reproducible package management, and in general is foreign to most developers running operations. Python has good error handling, scripting and logging out of the box and managable package management, and is familiar to most developers and operations. Python has much better libraries for building general purpose tools (but fewer libraries for complex statistics).

I disagree with the "learning curve"; if you've learned other programming languages Python has a pretty simple and familiar core, and Pandas (while the API is an inconsistent mess) is well documented. Base R is quirky compared with modern programming languages, and the API is pretty inconsistent.

I also strongly disagree with the Tidyverse bashing. I'd say it has the shortest learning curve (especially for someone familiar with SQL), and is one of the main reasons I still use R today outside of deep learning - I find it much more friendly to work with than any alternative.

I hope that you're wrong because it's solving the wrong problem.

A compatibility interval (at an agreed upon arbitrary level) communicates the magnitude of the difference as well as the uncertainty, which makes it much better for comparing options.

If my medication alleviates symptoms for 94-95% of patients above the current gold standard of 92-93% you could say it's "statistically significantly better", but the marginal improvement may not be worth the investment. Conversely if my medication alleviates symptoms for 50-80% of patients and the gold standard is 45-55% it would at least warrant future research (and if my medication has fewer severe adverse events it might be a better bet overall).

But this is just one small part of the whole picture: ideally we'd have preregistered experiments, experimental data published (or available to researchers where not possible due to confidentiality) and incentives for replication. Maybe this is too much for every field of science, but for ones where a wrong decision could have a severely detrimental impact they would create much more value than moving the P value.

The details are even hairier.

He wrote TeX82 (modern TeX) in WEB, a literate programming language he invented that could be tangled into the Pascal machine code or weaved into literate documentation in ... TeX itself.

In fact he wrote WEB itself in WEB and by hand compiled the output of the tangle program, which he ran on the tangle source to get the same output.

He also wrote the program METAFONT to produce the typefaces, which he used to design the Computer Modern font.

And then he used that to write instruction books for TeX and METAFONT as well as new editions of The Art of Computer Science and others.

TeX is still in regular use in the academic community (ported via Web2C) over 35 years later, despite the changes in technology in that time (with hacks to deal with modern fonts, outputting PDFs instead of DVI, including images and the like).

He taught the yak to shave itself, and it's still clean shaven 35 years later.

I wonder whether using computers more (whether numerical or symbolic calculations) could remove some of the burden of understanding mathematical techniques and make this process smoother. Or maybe it would just obscure the fundamental assumptions to software and increase the burden on learning computational techniques.

Does it matter whether mathematicians think it is handwavy? If any two trained practitioners can get to the same result of a calculation and it matches experiment to appropriate approximations that seems good enough to me.

Absolutely. There are some known anomalies in particle physics (e.g. neutrino oscillations) but for the most part it is insanely robust.

What's really hard is moving from these first principles to modelling real world phenomena; there are huge problems like protein folding, low temperature superconductors and even just predicting properties of new materials and chemicals.

It's still important for some people to work on fundamental physics, but I suspect there is a lot more opportunity in mere phenomenology.

Apache libcloud works well for more basic services (e.g. storand and compute) and is extending into container management services.

The web is dominated by a closed source browser built on an open source core, owned by the company that has the most private information and power to censor of any public company.

The "top level" problem is real in Python and in Lisps; but in practice it's not a huge issue (if you really don't like __name__ == "__main__" you can create your main app as a separate script to your library). main in Java is as magical as __name__ in Python.

Python now has type hints and good static checking libraries so you can add as much type checking as your want. Java's type system is atrocious (c.f. Haskell, OCaml); it doesn't check nulls (a common bug), making simple constructs (tuples, unions) often requires a new class file of boilerplate.

The HelloWorldApp is a bad fit for objects. All the functions are static. This confused me for a long time learning Java. You can get a very long way in Python without knowing @staticmethod.

The static nature of Java does make it easy to know where to look for things and introspect code; there tends to be less magic.