Vetch

Poe's Sonnet is limited to 15 free messages per day. The best freely accessible LLM with a generous daily allotment (300 msgs/day) is Bing Green Precise Mode. It's at about GPT4 level.

This specific problem is certainly not one for all on-devices AI processing. As someone else mentioned, there are unique UX and browser constraints that come from serving large compute intensive binary blobs through the browser (that are almost identically shared by games).

Separately, having to rely on preinstallation very likely means stagnating on overly sanitized poorly done official instruction-tunes. With the exception of mixtral7x8, the trend has been the community overtime arrives at finetunes which far eclipse official ones.

I disagree with your characterization of Vinge's works as primarily about disasters but I agree they were all about an accelerating technological pace and its relation with intelligence.

I'm fairly certain the mysterious event in Marooned in Realtime was Ascension.

For Fire Upon Deep, it was sealed and there was a powerful countermeasure.

The rabbit of Rainbows End felt like a trickster to me. Child-like playfulness, fey-like chaotic neutral at worst. I do not interpret Rabbit's survival as hints of doom. The weapon was plain old human abuse of power for control.

> Are you sure? I think "Open"AI uses the chat transcripts to help the next training run?

> Fine-tuning.

The learning that occurs through SGD is proven to be less flexible and generalizing than what happens via context. This is due to the restricted way information flows through transformers and which is further worsened in autoregressive GPTs vs models with bidirectional encoders.

On top of that, SGD already requires a great many examples per concept and, the impact of any single example rapidly diminishes as learning rate tampers down as training ends. Finetuning a fully trained model is far less efficient, more crippled when compared to learning from context for introducing new knowledge. It's believed that instruction tuning helps reduce uncertainty in token selection more than it introduces new knowledge.

> Co-pilot gets to watch people figure stuff out

We don't actually know if that's true. It depends on how many intermediate steps Microsoft records as training data. If enough intermediate steps lead to bad results and needed backtracking, but that erasure is not captured, it will significantly harm model quality. It is not nearly as easy to do well as you make it seem.

All in all, getting online learning into models has proven very challenging. While some "infinite" context alternatives to self-attention are promising for LTM, it'd remain true that the majority of computational power and knowledge resides in the fixed FF weights. If context and weights conflict this can cause degradation during inference. You might have encountered this yourself with GPT4 worsening with search. Lots of research is required to match human learning flexibility and efficiency.

That doesn't seem to be the case here. Reading through the article and twitter thread, the impression I get is that between moyix and the author, a decent amount of time was spent on this. A valid criticism that could have been made is the use of Claude Sonnet but based on the twitter thread, it looks like opus was what @moyix leveraged.

This is a poor analogy, a better one would be nuclear physics. An expert in nuclear physics can develop positively impactful energy generation methods or very damaging nuclear weapons.

It's not because of arcane secrets that so few nations have nuclear weapons, all you need is a budget, time and brilliant physicists and engineers. The reason we don't have more is largely down to surveillance, economics, challenge of reliable payload delivery, security assurances, agreements and various logistical challenges.

Most countries are open and transparent about their nuclear efforts due to the diplomatic advantages. There are also methods to trace and detect secret nuclear tests and critical supply chains can be monitored. Countries who violate these norms can face anything from heavy economic sanctions and isolation to sabotage of research efforts. On the technical side, having safe and reliable launch capacity is arguably as much if not more of a challenge than the bomb itself. Logistical issues include mass manufacture (merely having capacity only paints a target on your back with no real gains) and safe storage. There are a great many reasons why it is simply not worth going forward with nuclear weapons. This calculus changes however, if a country has cause for fear for their continued existence, as is presently the case for some Eastern European countries.

Was it via gemma.cpp or some other library? I've seen a few people note that gemma performance via gemma.cpp is much better than llama.cpp, possible that the non-google implementations are still not quite right?

Which is why I say it is contextual and depends on the task. I'll note that it's not only programming ability that is empowered but learning math, electronics, history, physics and so on up to the university level. As long as you take small enough steps such that you are able to verify with external sources, you will move faster with than without.

Writing it as "feel empowered" made it come across as if you meant the empowerment was illusory. My argument was that it is not merely a feeling but a real measurable difference.

> thought should feel empowered?

This is a strange question since augmentation can be objectively measured even as its utility is contextual. With MidJourney I do not feel augmented because while it makes pretty images, it does not make precisely the pretty images I want. I find this useless, but for the odd person who is satisfied only with looking at pretty pictures, it might be enough. Their ability to produce pretty pictures to satisfaction is thus augmented.

With GPT4 and Copilot, I am augmented in a speed instead of capabilities sense. The set of problems I can solve is not meaningfully enhanced, but my ability to close knowledge gaps is. While LLMs are limited in their global ability to help design, architect or structure the approach to a novel problem or its breakdown, they can tell local tricks and implementation approaches I do not know but can verify as correct. And even when wrong, I can often work out how to fix their approach (this is still a speed up since I likely would not have arrived at this solution concept on my own). This is a significant augmentation even if not to the level I'd like.

The reason capabilities are not much enhanced is to get the most out of LLMs, you need to be able to verify solutions due to their unreliability. If a solution contains concepts you do not know, the effort to gain the knowledge required to verify the approach (which the LLM itself can help with) needs to be manageable in reasonable time.

In this specific instance, I think it is necessary to make a distinction because given the direction things are headed, spied upon is tautological in the context of AI. AI-as-a-service requires sending private details to gain utility and there is a real risk this is the only kind of AI that will be allowed to exist. GPT4 was most probably used given they stated this action was taken in collaboration with OpenAI and so the emphasis on Microsoft was likely used as engagement bait by the blog author. In truth, Microsoft probably put this out to advertise how they take steps to keep AI safe. Unfortunately, this will also be ammunition for those who seek to ban opensource AI, you can only do this kind of thing with Monitored AIs.

Even if you're not an adversarial government leaning on it as a hacking enhancement, it's only a matter of time till governments worldwide demand certain controversial conversational topics be reported to law enforcement. I suspect the Knight Paladins of Anthropic would be more than eager to further this greater cause for the Safety of mankind.

From the start, since all but a few services are using your data to improve their models there is already no notion of privacy. The ethical ones will eventually (if not already) report suspicious activity (as determined by law) to law enforcement and the less ethical ones will report all activity to advertising and insurance agencies.

Some already lobby to ban opensource AI because enhancing the learning rate and reducing the friction of gaining new information for humanity without controlling oversight will also enhance hackers or other bad actors ability to access sanctioned knowledge. They consider this a heresy and deem humanity at large incapable of responsibly handling such increases in cognitive ability. Only a few Adept can be trusted with administering AI. Truly, spying is among the more trivial concerns for the future of computing, given AI's compute heaviness and the amount of centralizing control it engenders by default.

Don't take tacit knowledge for granted.

I'd also wager that "I don't know Timmy" is more thematically related. I feel most of the discussion in this thread glosses over what is most unsettling about Permutation City. It isn't just a book about what it could be like to be a simulated mind, it's most deeply about exploring the disquieting metaphysical consequences of computable minds. I can't think of a story that has as thoroughly scattered my basic grasp of reality as this one. Only Blindsight even begins to comes close.

In "I don't know Timmy" there's a sequence that goes:

"Well, we can't exactly turn it off."

"Why not?" asked Tim, halfway to the door, then stopped mid-stride and stood still, realising.

"Oh."

But you can turn it off without consequence and Permutation City explores the disturbing implications of why thoroughly (with a deus ex machina ending to save causal physics, as expected of an Egan story, physics is what has plot armor).

If we're talking about open-source LLMs, among the best embedding, multimodal, pure and coding LLMs are Chinese (attested and not just benchmarks).

For a drone, an LLM derived solution is far too slow, unreliable, heavy and not fit for purpose. Developments in areas like optical flow, better small CNNs for vision, adaptive control and sensor fusion are what's needed. When neural networks are used, they are small, fast, specialized and cheap to train.

A multimodal or segmentation algorithm is not the solution for bee-level path planning, obstacle avoidance or autonomous navigation. Getting LLMs to power a robot for household tasks with low latency to action and in an energy efficient manner is challenging enough, before talking about high-speed, highly maneuverable drones.

An AI that would be like an Illustrated Primer or the AIs from Fire Upon Deep is the dream from which we are currently far, doubly so for open source models. I wouldn't trust one with a sauerkraut recipe, let alone the instructions for a doomsday device. For the forseeable future, models cannot be relied upon without external resources to augment them. Yet even augmented with references, it's still proving to be a bigger challenge than expected to get reliable results.

This issue is overstated. Given the material is in Chinese, it stands to reason the bulk (but of course not all) of it will not be in violation of whatever policies. Furthermore, there have been a series of open-weights Chinese models that give reasonable answers to sensitive (in China) questions. Unless you plan to release something customer facing in China, it's not something to stress about.

The idea that overly burdensome regulations around open source models in the US shifts the global center of mass of LLMs to China is not implausible.

One reason resonant among government officials against opensource LLMs is to keep them from China and Russia. But among the very best embeddings, ~13Bs and perhaps 34B open-weights models are the Chinese ones. The recent DeepSeekCoder also tops EvalPlus: https://evalplus.github.io/leaderboard.html For those concerned about over-training, ChatGPT and GPT4 are almost certainly also contaminated and there have been independent confirmations of DeepSeekCoder's strength.

Let's assume that these LLMs are very useful and provide boosts to their users. It follows that anyone not leveraging them where applicable would be at an economic disadvantage. Without opensource models, everyone would have to pay a Google or Microsoft tax for the pleasure of using a service which could not exist without our work. All code, writing, art, data would have to be continually sent to their servers for anyone wanting to leverage their tools. You might use a FOSS editor or shell but you are still sending all your data to Microsoft servers.

The ones in control of the models also control what sentences are sanctioned, this is a problem the more widely LLMs are used. To add insult to injury, while we are not allowed private use of the models, governments and ad-tech surveillance capabilities will skyrocket.

Do you see the problem here? The capabilities of opensource models are not anywhere near high enough to justify such a cost, now or anytime soon.

And it won't end there. As the march of progress continues, we will see the AI doom crowd agitate for tighter surveillance of money flows, limits on private compute, bandwidth limits to homes, tracking what programs we run on our computers, on who is allowed to read the latest in semiconductor research and on and on.

It makes sense to make a purchase decision based on others opinions yes, but what I don't understand is why sillysaurusx seems to have already decided based on one response, and without knowing how (dis)similar Drybones's taste in games.

Alan Wake is a great game but I consider Alan Wake 2 to be much better, but I also enjoyed Control more than Alan Wake and enjoyed all aspects of Quantum Break. Where you fall on those games will probably most affect how much you enjoy Remedy's latest.

How much you enjoy unorthodox multimedia narrative story telling, psychological and cosmic horror meta-narrative weirdness, how much you accept being confused from design jankiness in certain spots as part of the experience, stuff like that will decide how much you enjoy this game. Personally, its only real flaw is mystery solving can be a little too handholdy and bruteforce-able.

One of the best games ever made, Outer Wilds, was done in Unity. Dyson Sphere Program is an excellent factorio style game that's very well optimized. I do not share your same trepidation on seeing the logo (doesn't move me in any direction, both some of the best and some truly awful games have been made with it).

How does developing another engine help at all? It is very likely its graphics are so under-optimized because they spent most of their time on the simulation aspect, which is more than challenging enough. Dyson Sphere Program is a game with lots of stuff going on at a time, the engine isn't the issue. Cities Skylines 2 is just a very complex game.

As a city builder it has not just dynamic geometry (which makes graphics optimization more challenging) and lots going on on screen like DSP, but also a complex agent based simulation of a city, its transportation, its economy, and individual agent AIs. It is far more complex than most games despite not being graphically intensive. And do note that as far as cities builders go, it's also one of the most ambitious graphically. My personal opinion is they deserve some slack as long as the game continues to improve.

Ah you're right. That makes sense. The autocomplete, informal proofs, translation or autoformalization, reference, search, feedback interactive assistant use-cases do seem promising though.

I don't think the right way to think about its utility is as a replacement. I see it in terms of a NNUE for Stockfish type augmentation, where a small neural network supercharges search. Small neural network because no LLM, not even GPT4, is good enough that the time lost evaluating with them is gained in disproportionally less search done.

Other uses are: better autocomplete from comments for Coq and Lean VScode envs than generalist tools.

Translation from NL sketch to formal proof. This is different from autocomplete in that it should generate long attempts as an automated spitballer. Leave it running a a long time to see if it finds anything when you're stuck. This works for formal and informal proofs but the latter gets no feedback (this is imagining future tunes able to make better use of interactive prover feedback).

Translating formal proofs to natural language.

Combine it with RAG and have it pull in and summarize references from your personal paper collection and the internet. Particularly useful in place of search when you don't have the vocabulary for a concept. As a basis for non-code based autocomplete of mathematical work.

I see unbounded potential for this combination. And a natural setting where between cost of search and the symbolic prover doing the heavy lifting, it's one of those areas that naturally lends itself to specialist Open over API models.

Don't forget that Vannevar Bush also inspired Douglas Engelbart (it saddened him that Bush would not accept his and his team's digital computer innovations). The Mother of All Demos, by Engelbart's Augmentation Research Center (ARC) group was far ahead of its time and was a major influence on all of modern computing. It was a major and more direct influence on Xerox PARC than the also influential "Libraries of the future".

Engelbart was primarily motivated by augmenting the human intellect [1].

> Increased capability in this respect is taken to mean a mixture of the following: more-rapid comprehension, better comprehension, the possibility of gaining a useful degree of comprehension in a situation that previously was too complex, speedier solutions, better solutions, and the possibility of finding solutions to problems that before seemed insolvable.

Licklider also wrote:

> It is often said that programming for a computing machine forces one to think clearly...it disciplines the thought process. If the user can think his problem through in advance, symbiotic association with a computing machine is not necessary.

>

>...many problems that can be thought through in advance are very difficult to think through in advance. easier to solve...solved faster, through an intuitively guided trial-and-error procedure in which the computer...[revealed] unexpected turns in the solution

We have yet to realize the core vision laid forth in As We May Think and by other pioneers of modern computing. And if a certain group that thinks Tools for Augmenting Human Intellect are too dangerous to be spread widely has their way, we may never do so; at least not without first regressing for a substantially long time.

[1] https://en.wikipedia.org/wiki/Intelligence_amplification#Dou...

We are not anywhere near 160 IQ assistants, otherwise there'd have been a blooming of incredible 1-person projects by now.

By 160 IQ, there should have been people researching ultra-safe languages with novel reflection types enhanced by brilliant thermodynamics inspired SMT solvers. More contributors to TLA+ and TCS, number theoretic advancements and tools like TLA+ and reflection types would be better integrated into everyday software development.

There would be deeper, cleverer searches across possible reagents and combinations of them to add to watch lists, expanding and improving on already existing systems.

Sure, a world where the average IQ abruptly shifts upwards would mean a bump in brilliant offenders but it also results in a far larger bump in genius level defenders.

Knowledge is power true, but even more powerful and rare is tacit knowledge. A vast collection of minor steps that no one bothers to communicate, things locked in the head of the greybeards of every field that keep civilizations running.

It's why simply reading instructions and gaining knowledge is only the first step of what could be a long journey.

My opinion is that if you're already well into a project, it makes far more sense to donate to Godot's development and remain in unity while also developing as much as possible in an engine agnostic manner. Moving to Unreal makes about as much sense as moving from Microsoft Edge to Google Chrome. You're still beholden to a corporation's whims.

Godot seems to be at the state Blender was ~5-10 years ago. Many major features are ticked but on closer inspection, there are significant missing pieces or implementation bugs.

When I last looked, there were issues involving scriptable render pipelines, post-processing, lighting, maturity of 3D physics, feature completeness of AI components such as navmeshes, lots of QoL issues around shaders. Lots of individually small headaches around animations, transformation hierarchies, input handling, collisions, IK that add up. There are also of course the usually mentioned limitations around asset store, platform breadth, 3rd party support and tutorial/community size and documentation.

AFAIK, there is no Shader and vfx graph equivalent for non-programmers. A Jobs and burst equivalent for C# users would also be very useful.

As good as Godot is already, given it still needs work, you should seriously consider donating to its development even if you don't intend to use it. Also, if you're starting a new hobby game you should ideally choose godot over unity, it's the only way holes will get identified and filled.

The only way to make corporations accountable is for open-source alternatives to be a viable third option such that whenever cost of use exceeds a commercial offering's value proposition, switching to open-source is a no-brainer.

I hope this didn't come across as too critical of Godot. It is rather the opposite, I really want it to get to where Blender is today. If I'm out of date anywhere I'd actually be happy to hear that.

Something to realize is that different models require different prompting styles. You can't prompt non-gpt4 models with GPT4 tuned stylistic ticks and expect similar results.

I've gotten great performance from llama2 derivatives. Out of the box performance is not near GPT4 but it is still very strong in its own right. And, if you are able to break down your problem so precise logit control coupled with guidance from forward or backwards chaining on knowledge graphs is applicable, you can easily exceed gpt4's reasoning ability for your domain. No fine-tuning necessary either.

I've been getting useful things out of LLMs since the days of roberta and raw T5, when Large stood for hundreds of millions of parameters. I am flabbergasted when people say a 7B parameter model is no good for them.

That is stretching arguably too far. If you are taking 1 sample path, you are not in any meaningful sense searching a tree. In the context of sampling a probability distribution, which is what LLMs do in effect, there is extra depth to this. Any random response need not be representative of what the model "thinks". And maybe counter-intuitive to some but the most likely generation might actually be unrepresentative as well.

Drawing lots of samples and then marginalizing (as a kind of vote) is methodologically more principled where appropriate. Constraining generation according to some gating function, continually redrawing samples, can be used to significantly reduce error rates at the cost of longer generation times.

LLMs are not being used to their full potential because it is too costly to do so.

Whether or not backtracking is needed is really down to the grammar's ambiguity.

The auto-regressive nature of LLMs is actually something that counts against them, at least as some tell it. Although, really, the root problem is generating autoregressively from LLMs precludes planning ahead while also lacking any iterative refinement stage.

Backtracking, look-ahead, early failure pruning and staged generation are all very useful for fitting both concepts (refinement and planning ahead) in an auto-regressive generation framework.

idbfs has posted this link already but did not explain that Shalizi provides a deep theoretical explanation for why universal source coding (does not require information about symbol distribution or statistics) such as Lempel ziv derived compression algorithms can serve as powerful language models if practical restrictions on them such as dictionary and input corpus size are lifted.

This is a good example of how old methods can be pushed quite far if similar resources were devoted to them. Who knows, they might even posses advantages hitherto unmet due to a lack of exploring at larger scales.

That said, Transformers have a number of practical advantages. The learned projection matrices in attention lend Transformers a dynamic adaptability with respect to learned patterns that help make them programmable by their context, able to work out patterns present in context zero shot and on the fly. gzip based language models will be limited to their dictionary of patterns. The underlying vector space of neural language models also makes semantics more readily learnable (driving novel synthesis such as neologisms and more) while feed forward layers can learn a large range of computations.

http://bactra.org/notebooks/nn-attention-and-transformers.ht...