fultonn

I have some property that doesn't really get cell reception in the summer, and even in the winter the reception is at-best spotty. I use granite and qwen together with a manually curated set of books, articles, etc. to get work done while disconnected. I tend to update the set of resources before leaving town because I'm generally intentionally disconnecting to focus on deep work. Building a workflow that makes that possible has helped me build out a setup that replaces a ton of my day-to-day anthropic/openai use even when I'm near fiber.

It is interesting how, even pre-llms, offline computers change the nature of a lot of activities. E.g., about once a year I'll walk do a random walk in the woods for a few days without paying much attention to maintaining a bearing. With maps and compasses it's a whole skill set to find your way back. With a phone and downloaded maps you just kinda press the power button and check your heading every hour or two.

The "survival" use-case is interesting. I fear a lot of the practical applications of that knowledge are either time-sensitive (e.g., RAG on a "Stop The Bleed" course won't help because by the time you learn the patient is dead) or require tools+skills that aren't easily improvised completely from scratch in high-stress situations (e.g., building shelter or even really simple stuff like building a fire when it's very cold out). And for other stuff common sense prevails (e.g., power out -> keep fridge closed).

It's a very "The Year of Linux on Desktop" situation, in the sense that in any particular year it seems like Linux is close, but the OSS stuff requires a lot of configuration to get to feature parity, and the barrier keeps rising because the prop solutions are running as fast as they can to justify the cost premium.

Unlike Linux, the world is pretty literate about tech business models, so I think there's a way for the open weight world to avoid the situation Linux got into, where it only really caught up once the market was pretty much commoditization any-ways.

If anyone interested in PL theory (including adjacent fields) and/or inference engines is reading this wants to help make open weight models and associated ecosystems feature-competitive, we're hiring, drop me a line: nathan at nfulton org.

The Firm is confederated.

Consider:

When the police officer gave the human an order to disburse, and the person willfully refused, they then went to jail. Perhaps they are vindicated. Perhaps not. (even if vindicated the process is the punishment.)

However, if a police officer also gives the robot a verbal order to disburse, and the lidar just sits there are spins, who goes to jail? And if willfulness is a component to the crime, and the engineers did not design a system capable of understanding police orders, then did a crime even occur?

In PA statute, "pedestrian" confers mostly protection, not obligation. The corresponding obligations imposed on human pedestrians in PA code are mostly downstream of willfulnes and/or personhood. I.e., you need to be a person or act willfully to be bound by the typical obligations of a pedestrian.

The law worked well to-date because all pedestrians were human persons, so you didn't end up in a weird edge case of having a pedestrian's protections without a human person's obligations. But if the robot gets all the protections but few of the obligations of pedestrianhood, well, that violates my moral intuition.

In the spirit of intellectual curiosity regarding this fascinating legal situation, the next officer facing a similar situation should issue a disbursal order to the robot and the human, and the DA should pursue criminal charges against any party that willfully refuses that order. Which, to be fair, is probably what would happen if two humans were engaged in a similar standoff.

Mathematicians working at universities are paid to teach, to engage in the politics and business of government contracting, and only in very rare circumstances to play prestige games on famous problems.

It is unlikely that Congress will decide to slash budget for NSF's MPS directorate on account of AI getting good at proving theorems. That's just not how Congress works.

It is also unlikely that LLMs will be able to do, in the near future, do the sort of applied mathematics that happens in the context of occasional NIH or DoD funding that flows into Mathematics departments, for a variety of complex sociological reasons I'm happy to expound upon.

And it's unlikely Engineering degree programs will stop requiring calculus sequences, since being able to do the computation was has not been the actual point of teaching calc since he late 20th century (if ever).

So, the good news is that I doubt any of this will have much of an effect on the job market for Mathematicians who want to prove theorems for a living. The bad news, of course, is that the job market for Mathematicians who want to prove theorems for a living was already pretty horrendous.

Kempe's reduction technique, on which the eventual proof still depended (at least spiritually? I think?), did contain true insight with reach beyond the original problem statement (e.g., Chaitin/register allocation).

I submit that "clever way to soundly reduce to finite set and check" is itself probably one of the more important types of insight in mathematics. Actually, all of my favorite mathematics, from an aesthetic perspective, has exactly this flavor. That's down to taste, of course, but it is sort of a fundamental/powerful move in modern mathematics. The beauty of that insight can only be appreciated once you're willing to accept that boring case checking is tractable.

In that sense, the FCT gave us two really important insights -- first some profound insight from Kempe then an equally profound insight about the role of computers as a bicycle for human thought.

If you are ascribing intent ("Anthropic designed a system that intentionally does this") or emergent model intent ("OMG at AGI") then I mostly agree.

But there's a more pedestrian version of this, where an unwise person puts a model in an open loop without sufficient HiL checking and then the model does dumb things. There are even some recent stories about OpenAI and Anthropic systems "emailing the FBI", which is sufficiently close to "emailing opposing counsel" [1][2].

But even if you feel those stories are trolling/marketing/etc., I think it's clear that there's a plausible risk model here. If you hook up an LLM to a toolset that has access to Outlook and your file system and opposing counsel's email address, then of course the dumb thing could happen. The LLM doesn't need to be AGI/ASI; in fact, to me, the worse the model capability the more likely this sort of screw-up feels. There are even (non-sarcastic) instructions or CoT roll-outs I could imagine which might elicit a sequence of tool calls the net effect of which is an errant disclosure.

So this feels less "unforeseeable lightning strike" and more "car accident while driving at speed". I'd put the HF hack in that bucket as well.

A sufficiently careful engineer and a well-trained expert operator can together properly constrain the system to not do the dangerous thing. But, in general, that is not how agentic systems or their underlying models are currently being designed, and most users are not receiving sufficient training.

A lot of these LLM-backed products feel very poorly engineered, rushed, and optimized for brain-feel instead of surfacing and forcing the user to acknowledge the large quantities of actual essential complexity in play. It's all very "unsafe at any speed".

--

[1] https://tech.yahoo.com/ai/chatgpt/articles/chatgpt-allegedly...

[2] https://www.bgr.com/2055251/anthropic-ai-claude-contact-fbi/

[3] https://en.wikipedia.org/wiki/Unsafe_at_Any_Speed

If anyone here is looking for engineers to work on legal AI products, both of the students who worked on this project come highly recommended.

Returning to the substantive matter, I'm happy to try to re-state the topic at hand using your preferred symbols.

Suppose:

1. Someone has a type of task to be done.

2. Finding a systematic way to complete that type of task has immediate value.

3. That type of task can be solved at lower per-unit cost without paying for tokens from the latest Anthropic or OpenAI model release.

Then why would you tell that person that they need to "think of better problems"? Particularly when they've already creatied a solution to the type of task they need to complete with de minimis per unit cost.

There is nothing wrong with consumers who choose to become early adopters and ask what new things can be done with a new class of product. And there is certainly nothing wrong with creating a new technology or making a scientific discovery and spending some time looking around for applications of that discovery/invention. These are important things. The PC and Internet revolutions both relied on this sort of activity, so a culture of doing that type of work is deeply embedded in our field.

But, likewise, there's nothing wrong with driving an ordinary nail through two ordinary piece of lumber to solve a problem for a person who has money and needs two boards fastened together. That framer doesn't "need better problems".

I hope this makes the meaning of my position clear.

> well, think up some harder questions.

There's nothing wrong with this mindset, but your insistence that other people must have your values is off-putting.

To wit: at the end of my phd I was offered several tenure-track positions where "work on problems that are hard" would've been a big part of my job.

I turned those offers down due to my distaste with "because it's hard" as a justification for working on problems.

I'm much more interested in helping people and solving (def'n 1) problems. Those problems are almost always too large and messy to one-shot, but very often contain many components that can be solved with non-sota models, and make sense to solve with non-sota models for any number of economic/scaling reasons. E.g.: "can I serve this thing that costs $0.25 per run on even c.2024 sota models to 100,000 K12 students who will use it dozens of times a day on a non-profit's budget of ~$0.00."

P.S. The "problem" definition thing is very confusing to your readers because the person you responded to was clearly using the first definition! It would have been helpful for you to sign-post that you are using the same word as the person you are replying to but changing the meaning out from under your reader.

I have enough real problems in life. I don't need to invent new ones just because a new technology is available.

Many of my problems in life are fully solved far past my satiation point by a 3b model that costs me nothing to run.

Many others are not.

But in either case, when I am acting and living wisely, almost all of my problems exist prior to the existence of technological solutions to those problems.

This is also true for the customers and employers that I care to work with. This has changed in me over time, but I now try my best to avoid inventing new problems. The world has enough big, important problems already.

> This morning I elicited a microkernel operating system from Opus 5.5.

This is cool but also a good example. I don't need a personalized microkernel just because it's possible to have one.

Maybe I need one and I don't know it, but the problem statement definitely isn't "I have inherent desire for a personalized microkernel".

I know this because I spent a lot of social time around a Math dept in grad school and many of them are moonlighting working for subcontractors on production and evaluation of training data [1].

Anyways, mathematics is a bit of a funny discipline... some buckets:

There are lots of obvious cases where progress has obvious applications for incremental progress, and where you can ask for novel math from the perspective of the application or ask for applications from the perspective of novel math. That seems like the sort of thing that you could throw $10M at get something valuable without much human input. I do this, at much much much smaller dollar amounts, pretty regularly, and with open models, so nothing secret/sota/etc.

But there are also lots of cases where some progress gets made on something very pure and esoteric and the implications aren't really clear. Those are cases where just asking a bunch of nerds to marinate on it while interacting with a messy world in the full social generality of a modern university could make a lot of sense. The connection between the four color theorem and register allocation, for example, feels very... hard to do without humans free-associating in a diverse academic environment.

And, at last, there are cases where there are subfields or problems that are big and important in the way that driving a rare car or wearing a particularly fascinating rolex is important. Some of modern mathematics feels like it has its cultural roots in intellectual gamesmanship amongst wealthy gentleman and/or those under their patronage. A lot of the... less well-argued... objections come from this set.

Like I said, math is kind of a uniquely weird discipline. It's got aspects of practical gritty science, noblesse oblige/high-culture, fashion/taste, craftsmanship, tutelage, and so on.

--

[1] Some enterprising journalist should track down all the subcontractors used by OpenAI for mathematics data, then talk to all of those people and figure out how much breakthrough-adjacent data work was happening... pure conspiracy but I'd be curious for someone to explore the hypothesis that there were maybe people helping find fragments here and there and it wasn't all purely mechanical monkeying.

By verifying that all of its possible behaviors conform with the "it works" spec, regardless of which of those behaviors it chooses.

Monitoring with a known-safe fallback is the easiest case.

USA-specific: At ~$38K/year (including SS) you probably need to own a home without substantial deferred maintenance to make it work. So you can retire on $250K but only if you're over 62 and only if you have substantial wealth outside of those liquid assets (so probably a net worth at least 2x your liquid). If you have a spouse also on SS it much easier.

scala's repl is decent. It has its annoyances, but so does python's (white space sensitivity + repl + terminal emulators stuck in the late mid century don't mix).

If you've ever read through FORTRAN code from a mathematics department or MATLAB/C/C++ from (non-software) engineering disciplines, then you probably understand why productionizing a jupyter notebook is definitely not the worst of all possible worlds.

Also worth mentioning 2026 - 4 = 2022. Not a great year to be stick building from a cost perspective.

I would use a different adjective: exceptional.

There are not not times where lone geniuses produce amazing output. But most of humanity's progress over the last few thousand years (or at least certainly the last few hundred) resulting from a different type of work.

I'm deeply suspicious. I do not yet have a concise statement for why, but a lot of literature on the sociology of knowledge work sort of points at my thoughts.

Section 5 of the Thurston article cited by Tao touches the elephant. Raduchel's article on the economics of software [2] also touches it.

I've tried to put words to this for a few years. I think I'm just going to start writing versions of it as see if that helps me shape the thought into something more concise.

So, in the spirit of this article's style, here are some postulates:

1. There is a sociological process happening in the production function during knowledge work.

2. That production function and the associated sociological process spans years or even decades, and must outlast many of the artifacts that are produced during the early years of the function.

3. You cannot get the right lines of code or the right theorems proved without running that sociological process alongside the artifact production process.

4. It is impossible to completely separate the sociological process from the artifact construction process. If you just iterate on artifacts then too much of the required hidden state is lost to make progress in the right direction. This is true even if you include distilled artifacts capturing pieces of the sociological process (eg meeting notes, documentation, commit logs, prompts).

5. So you need that sociological process, or something like it, to still happen.

6. For a lot of knowledge work that process plays out in extremely high-fidelity social interactions [3] that we have not yet captured in the datasets that would be required to reproduce those dynamics.

7. And even if we do collect that data, our current architectures and training algorithms and hardware would be useless given the size of the datasets.

So: the technology today gives us the ability to iterate on the production of artifacts. But it does not sufficiently simulate the social process which gives rise to the Right artifacts.

This isn't exactly what I actually think, but it's a version of the thing that I intuit when I watch heavy use of AI in both software projects and formalization projects. And simulating that process feels way harder than people are currently assuming.

[1] https://arxiv.org/pdf/math/9404236 Section 5.

[2] https://www.nationalacademies.org/read/11587/chapter/11 pp 166-168.

[3] there is a reason we still gather in-person around white boards, and why doing so is more crucial for some types of work than others.

(And I only reached for the LLM after working through all the flowcharts and tables in the shop manual, reading relevant parts of two engine repair books, and watching a bunch of youtube videos.)

It's especially awesome in domains where:

1. you personally don't have the time to learn,

2. you can trivially evaluate the output ("does engine go brrrrrr or brr-splaugh-crunch"), and

3. there are no bad outcomes; or, there are bad outcomes (destroy engine, hurt myself) but you can augment with ground-truth ("do X as suggested by nlp machine but follow shop manual for doing X").

It was a 3B or 8B model too, so I probably burned more energy on unnecessary oil changes than on running the llm inference.

Say what you will about LLMs, and dear god can the claude-style harness outputs be annoying in the workplace. But they're great tools for accelerating the sorts of ultra-narrow-path learning curves that commonly pop up in tinkering/hacking/similar sorts of activities.

(Also, people without critical thinking skills believing everything they read pre-dates LLMs by thousands of years. This was a widely discussed problem with popular media consumption in the USA during the several years preceding the invention and popularization of LLMs (say 2015 to 2021). There's a different, more fundamental educational diagnosis for that problem, which LLMs may exacerbate, but which they did not cause and cannot cure.)

So, to be fair: most philosophy majors wouldn't have much luck with Rudin.

> Dive into the classics after you have gained the maturity from modern texts.

Diving into old texts is a skill unto itself. That's why a lot of institutions do the great books thing as a core curriculum (so, maybe 2-3 courses taught in this style, as an alternative to more conventional phil 101/history 101 style distribution requirements). Then a more conventional education from there onward. The theory is that this is a mid-point precisely because it provides lots of transferable skills for diving into the classics in your chosen field, while avoiding the "let's learn analysis from descarte" excesses.

Of the lot, Elements feels misplaced.

Lots of people actually do read Elements as part of their course of study. It's niche but there's a whole cottage industry within academia for that sort of thing. There are probably over a dozen institutions that have either a degree program or a core curriculum that is organized around original texts, with Euclid usually serving as the math distribution of that sequence. So running into people who have read (big chunks of) Elements is not that uncommon. That's true even IRL outside of online discussions forums on thread topics that likely select for such people.

My impression is that this is not really true of the other examples. Except maybe Godel's proofs; I do think a sufficiently motivated instructor could pull a decent chunk of college students through the original text in a semester. Probably better ways to spend everyone's time, though.

> IMO that's the same as going on the street door by door and checking if one is left open to steal everything inside the house...

From experience: this does happen regularly in some neighborhoods of some cities in the US, and even that isn't always an enforcement priority. So lack of enforcement on the internet, where most the perpetrators probably aren't even in a jurisdiction with an extradition treaty, isn't exactly surprising.

In a past life I spent a lot of time on that sort of thing for control systems and RL. Spec says what not to do, reward says what to do, implementation can be arbitrarily complex wrt the spec.

There are many opportunities for an analogous move in LLM-assisted software engineering.

The conceptual gap I'm referring to here has nothing to do with formal methods per se. It's just an analogous problem with the quanta of information required to state the spec vs the quanta of information required to state the implementation.

Namely: once your problem has enough of a certain type of essential complexity, there's not a huge delta between "a sufficiently specific description of the problem" and "the source code that solves the problem". The complexity of a sufficiently specific prompt approaches the complexity of the actual solution. At that point, the former does not have the purported benefit and the latter has a lot of huge benefits (determinism, modularity, etc).

When one is operating in that regime of problems, proposing that one can substantially automate the software engineering function has a real "I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question" feel to it.

This is deeply related to the conceptual error a lot of executives are currently making around automation in/of their software engineering orgs.

It has always been true that learning some formal methods probably makes you a better programmer in certain ways, even if you never use them. I think it's increasingly also true that learning some formal methods probably makes you a better manager of people/processes that product software.

edit: edited.

We already arrived at that destination a decade ago.

Probably even further back tbh.

There's just a lot more people playing the game now, without the social indoctrination that made it more tolerable in some circles.

It's not that bad, though. September is annoying but you kind of miss the eternal renewal once you're out.

You are not. A disproportionate amount of value in the computing industry was created by <strike>geniuses</strike> decently smart people who worked together and who decided to just tell people how to do things instead of trying to capture the value of being the first person to figure out how to do those things.

This observation pre-dates the current wave of AI hype by a half century or so.

> driven by greed

I can only speak for myself.

For me it's exactly the opposite. If I want to explain how something works, I can just... do that. If I want to share an artifact demonstrating how to solve a particular type of problem, I can just... do that. If I want to mentor/teach, I can just... do that.

Doing those things within the confines of Academia Approved Institutions is exhausting and distracting.

To wit, and the actual point of this post: the term "Publishing Research" in this article doesn't mean "post it on a .html page and share the source code". It means engaging in a very specific and peculiar and extremely political modality of communication.

And it really only makes sense to do that specific and peculiar and political thing you're at a stage in your professional/personal development where you need to play that particular prestige game. (Which there's nothing wrong with, but it is a deeply cargo culted version of the actual scientific process.)