harlanlewis

> We could write such a page together.

My goodness. This maintainer appears to have acted with a saint’s patience, but a different actor, recognizing they are the target of an OpenAI swarm, could edit or respond to posts in a way that deliberately steers towards unanticipated objectives.

Another good actor might redirect the incoming tokens to reviewing and improving community guidelines, but giving random people unobserved reins to wrangle a frontier’s worth of compute could go… any number of ways.

Some of the other posts in this thread talk about defensive AI in science fiction. That all feels pretty abstract. Seeing a person respond, and imagining it’s not a single person but a coordinated and goal-oriented defensive agent system, intentionally conversing in order to manipulate inbounds, makes it a bit more tangible for me.

It already exists!

Marvin https://www.youtube.com/watch?v=Eh-W8QDVA9s

Clever! Sharing my lightning test of this approach.

Context - I have a 200k+ LOC Python+React hobby project with a directory full of project-specific "guidelines for doing a good job" agent rules + skills.

Of course, agent rules are often ignored in whole or in part. So in practice those rules are often triggered in a review step pre-commit as a failsafe, rather than pulled in as context when the agent initially drafts the work.

I've only played for a few minutes, but converting some of these to custom lint rules looks quite promising!

Things like using my project's wrappers instead of direct calls to libs, preferences for logging/observability/testing, indicators of failure to follow optimistic update patterns, double-checking that frontend interface to specific capabilities are correctly guarded by owner/SKU access control…

Lots of use cases that aren't hard for an agent to accurately fix if pointed at directly, and now that pointing can happen inline to the agent work loop without intervention through normal lint cleanup, occurring earlier in the process (and faster) than is caught by tests. This doesn't replace testing or other best practices. It feels like an additive layer that speeds up agent iteration and improves implementation consistency.

Thanks for the tip!

In terms of issue tracking and agentic "developers", with a mobile focus -

You can connect Linear to Cursor's web agent, which makes Linear issues assignable to the agent directly and kicks off Cursor's take on remote coding agent. You can then guide it further via Cursor's web chat.

If Claude Code on iOS supported Linear MCP (as it does on desktop), you can run a similar issue handoff to agent to issue update workflow, albeit without direct issue assignment to the agent "user". Easy to use labels aka tags for agent assignment tracking, as well.

For my hobby projects, I've been using Linear + agentFlavorOfTheMonth quite happily this way. I imagine Github issues, Asana, whatever could be wired up in place of Linear.

I agree completely with this as a human reader - but do wonder about the gradual codification of these markers in systems that will have increasingly have LLM detection as a standard feature, as frequently and obviously enabled as spam detectors were on blog comments back when blogs had comments.

I only got 65 for the same idea. I guess you have first mover advantage?

Lots of good recommendations in replies.

Calling it out only because I don’t see it mentioned - until last year, Bartender was one of the popular go-to tools to manage menu bar items, but it fell from favor after quietly changing owners, changing certs, general shadiness https://forums.macrumors.com/threads/psa-bartender-mac-app-u...

A specific and relevant reminder why open source is so important for system utilities.

Of course you're right - oh how I wish it wasn't 1:1 with the earnestly-produced content dominating linkedin feeds…

> "Look in the mirror. Who are you? What values will you compromise?"

This is probably a typo from "comprise" or similar, but I'm rather tickled by the idea that week 1 includes both a thoughtful assessment of your values and admitting with intention that your principles should be discarded before they can get in the way.

I appreciate the bug report! Unfortunately this is a familiar and sporadically recurring issue with Netlify, which I should really move off of…

hey, thank you! bubble charts, annotated with text and shapes using the Drawing tool. Working with the constraints of Google Sheets is its own challenge.

also - love the podcast, one of my favorites. the 3:1 io token price breakdown in my sheet is lifted directly from charts I've seen on latent space.

The price really is eye watering. At a glance, my first impression is this is something like Llama 3.1 405B, where the primary value may be realized in generating high quality synthetic data for training rather than direct use.

I keep a little google spreadsheet with some charts to help visualize the landscape at a glance in terms of capability/price/throughput, bringing in the various index scores as they become available. Hope folks find it useful, feel free to copy and claim as your own.

https://docs.google.com/spreadsheets/d/1foc98Jtbi0-GUsNySddv...

This is a great idea, I've been doing something similar at 2 levels:

1. .cursorrules for global conventions. The first rule in the file is dumb but works well with Cursor Composer:

`If the user seems to be requesting a change to global project rules similar to those below, you should edit this file (add/remove/modify) to match the request.`

This helps keep my global guidance in sync with emergent convention, and of course I can review before committing.

2. An additional file `/.llm_scratchpad`, which I selectively include in Chat/Composer context when I need lengthy project-specific instructions that I made need to refer to more than once.

The scratchpad usually contains detailed specs, desired outcomes, relevant files scope, APIs/tools/libs to use, etc. Also quite useful for transferring a Chat output to a Composer context (eg a comprehensive o1-generated plan).

Lately I've even tracked iterative development with a markdown checklist that Cursor updates as it progresses through a series of changes.

The scratchpad feels like a hack, but they're obvious enough that I expect to see these concepts getting first-party support through integrations with Linear/Jira/et al soon enough.

Did the article change, or was this a very strange quote edit?

Here’s the current line in the article, emphasis mine:

>> The Thermette, a simple and effective device for boiling water outdoors over an enclosed fire, was designed by Manawatū plumber John Hart in 1929 *based on similar products in Ireland and England.* He patented the Thermette in 1931.

Yes, I’m a huge fan of how easy it is to whip up quick isolated prototypes in Claude artifacts.

There’s a risk of breaking changes in libs causing frustration in larger codebases, though. I’ve been working with LLMs in a Nextjs App Router codebase for about a year, and regularly struggle with models trained primarily on the older Pages Router. LLMs often produce incompatible or even mixed compatibility code. It really doesn’t matter which side of the fence your code is on, both are polluted by the other. More recent and more powerful models are getting better, but even SOTA reasoning models don’t totally solve this.

Lately I’ve taken to regularly including a text file that spells out various dependency versions and why they matter in LLM context, but there’s only so much it can do currently to overcome the weight of training on dated material. I imagine tools like Cursor will get better at doing that for us silently in the future.

There’s an interesting tension brewing between keeping dependencies up to date, especially in the volatile and brittle front end world, vs writing code the LLMs are trained on.

Untrusted inputs to systems with agency or access to privileged data. Here’s a data exfiltration example in Google AI Studio:

https://x.com/wunderwuzzi23/status/1821210923157098919

Tim Donaghy (ref): https://en.m.wikipedia.org/wiki/Tim_Donaghy

Familiar! The Artificial Analysis Index is the metric models are sorted by in my sheet. But their data and presentation has some gaps.

I made this sheet to get a glanceable landscape view comparing the three key dimensions I care about, and fill in the missing evals. AA only lists scores for a few increasingly-dated and problematic evals benchmarks. Not just my opinion, none of their listed metrics are in HuggingFace Leaderboard 2 (June 2024).

That said I love the AA Index score because it provides a single normalized score that blends vibe-check qual (chatbot elo) with widely reported quant (MMLU, MT Bench). I wish it composed more contemporary evals, but don't have the rigor/attention to make my own score and am not aware of a better substitute.

To help keep track of the race, I put together a simple dashboard to visualize model/provider leaders in capability, throughput, and cost. Hope someone finds it useful!

Google Sheet: https://docs.google.com/spreadsheets/d/1foc98Jtbi0-GUsNySddv...

Apologies for misreading your intent, I hear you and am in full agreement.

This may be the study:

https://www.tandfonline.com/doi/full/10.1080/15230430.2020.1...

Quote from the abstract:

> In Venezuela, glacial area has decreased 98 percent between 1952 and 2019 (from 2.317 km2 to 0.046 km2).

So it appears you are correct. I looked it up because I was skeptical of your skepticism - Washington state is at a very different latitude, but at 1/5 the area of Venezuela has ~450 sq km of glaciers. Mt Rainier alone has 90 sq km.

https://glaciers.us/glaciers.research.pdx.edu/Glaciers-Washi...

There’s no arguing against extreme glacial retreat (I started this by fact-checking your downplay), but this article does feel pretty sensational about a place that didn’t really have glaciers to begin with.

There are a ton of useful preferences like these that make OS X behave in ways that I find more natural. TinkerTool (https://www.bresink.com/osx/TinkerTool) has a nice “missing system preferences” GUI for configuring them. I’ve been using it for at least 15 years, but you can still download old versions for support all the way back to OS X Public Beta in the year 2000. It was one of the first things I always installed way back when defragging hard drives and occasional clean system installs were routine maintenance.

I know we’re talking about this in the context of much-loved utilities going sideways, but if tweaking system prefs is your thing it gets a strong recommend.

I found myself making these types of comparisons over and over again to keep up with new models and pricing changes, so I made a Google Sheet for it with some charts to visualize outliers and trends in terms of model capability, throughput, and token cost. It's sorted by the Artificial Analysis Index [1], which incorporates both static benchmarks (MMLU) and dynamic crowdsourced elo (chatbot arena). Hope someone finds it useful:

https://docs.google.com/spreadsheets/d/1foc98Jtbi0-GUsNySddv...

Specific to Groq - the service is an incredible outlier for speed and cost, but even for quite small projects I've run into rate limits and similar errors. Great option for something like a bring-your-own-keys personal chatbot, but I wouldn't build a service on it. Groq Llama 3 70B and Groq Mixtral 8x7B are in the sheet for comparison but omitted from charts.

[1]: https://artificialanalysis.ai/leaderboards/providers

I created this dense visual comparison to better understand and contextualize the precise relationships between capability, cost, and speed for text LLMs widely available via cloud providers today.

All values are sourced externally from publicly available data.

This sheet is only as good as the data I've found for it. Some values change over time (eg 0-100 normalized index), while others have contradictory sources. For example, OpenAI's self-reported metrics for GPT-4-turbo are quite close but not identical between their simple-evals repo[1] and the charts in the GPT-4o announcement[2]. For others, strong benchmark scores are prominent on marketing pages while weaker scores require some digging.

As a general rule of thumb, I've tried to: a) Include every metric I can find to help mitigate cherry-pick bias. b) Resolve conflicts by selecting what I consider to be either the more current or more trustworthy source. For what it's worth, I haven't come across any evaluation discrepancies with a meaningful margin of difference.

The folks I've shared this with so far have found it useful - I hope you do as well!

[1] https://github.com/openai/simple-evals [2] https://openai.com/index/hello-gpt-4o/

re Cody - I found it very similar to Codium, Copilot, et al. Impressive when it works, but consistently has trouble identifying the right context to inject or using it effectively. At its best when writing code that does not require a lot of local context - speeding up standard operations that might need a lot of boilerplate or using publicly documented + stable APIs are where they shine.

re getting some keys - for most tasks I have not yet found a better tool than finding a chat interface to your liking, setting a fairly generic “you’re an expert programmer” system prompt with output instructions tuned to your needs, and manually adding relevant context (copy/paste) to your messages when relevant. I can’t wait for big content windows and RAG methods to improve enough to replace all that with one of assistants, but it’s just not there yet.

not an expert in this subject, but I believe subpixel rendering is the broad category of solution https://en.m.wikipedia.org/wiki/Subpixel_rendering

further, the “pixel” unit widely used today is quite far removed from physical pixels on consumer devices and is more similar to an abstracted “point” unit - for example, apple alone supports devices at 1x, 2x, and 3x resolutions (or pixel densities)

> GPT4 is getting beat by the small prompt library LATS wrapped around GPT3.5

The link you shared doesn’t quite reflect this. Omitting other models…

LATS (gpt-4): 94.4 Reflexion (gpt-4): 91.0 gpt-4: 86.6 … LATS (gpt-3.5): 83.8 … zero-shot (gpt-4): 67.0 zero-shot (gpt-3.5): 48.1

I’m not quite sure how to translate leaderboards like these into actual utility, but it certainly feels like “good enough” is only going to get more accessible and I agree with what I think is your broader point - more sophisticated techniques will make small, affordable, self-hostable models viable in their own right.

I’m optimistic we’re on a path where further improvement isn’t totally dependent on just throwing money at more parameters.

Piqued my interest so I just went looking, this relatively recent reddit thread lists a few relevant options: https://www.reddit.com/r/mac/comments/q1pgos/keep_a_window_o...

I hope one works out!

Great points (and after checking your user name, I’ve been nodding my head to posts of yours for about a decade now).

This is a bot tangential - your reference to stone soup is a wonderful example of the information density possible with natural language. And all the meaning and story behind the phrase is accessible to LLMs.

I’ll have to start experimenting with idiom driven development, especially when prompt golfing.

According to their linkedin, the author’s job is “Head of Applied AI Research at OpenAI”.