Going to feed into my own.
Out of curiosity how have your agents evolved and metric changed.
899 karma · joined November 2, 2021
Going to feed into my own.
Out of curiosity how have your agents evolved and metric changed.
I think my only request is to expose the rules engine as a declarative api in v2.
I ended up locally caching the transactions data, using cel/aip-160 as a rules engine then having the script apply the results on the data.
The other use case funnily enough was to ask how much I’m spending on hobby dev.
Make sure I’m not getting carried away across llm subscriptions, as well as cloud costs.
I think the nice part is once you have the cli you just use the agent in your ide as your chat interface.
They have an api and I got llm to write a CLI for it.
That way agent can pretty much pull the data it needs or wants.
One thing I also had it do is build up a series of rules for tagging which then I run a cron for once a day.
Every once in a while I just ask it to look over the rules and make new ones for uncategorized transactions.
(As a side note I think having LLM “memoize” a task through a rule engine or code is really nice pattern)
But once you have the cli with query you can pretty much ask the agent to do anything.
Anyone know of any other similar tools that allow you to track across harnesses, while coding?
Running evals as a solo dev is too cost restrictive I think.
running evals seems like it may be a bit too expensive as a solo dev.
Even just having a hum while an agent is working could alert you when it get stuck.
Or taking your idea further being able to listen to the rate of tokens, or code changes, or thinking.
Sort of like hearing the machinery work, and hearing the differences in different parts of the code base.
Does python sound different than rust or c++ or typescript.
Or some kind of satisfying sounds for code deletions and others for additions. Like Tetris.
And early failures is much more acceptable than I think most people expect.
It felt like I constantly have to go back and either fix things or I just didn't like the results. Like the forward momentum/progress on my projects overally wasn't there over time. Even with tho its cheaper it just doesn't feel worth it, to the point I start to feel negative emotions.
I'm actually a bit worried that I've somehow become to feel more negative emotions with agentic coding. Quicker to feel frustrated somehow when things aren't working.
I guess overall probably was a good decision.
But 7.5x as well as quota limits is pretty hard to swallow.
The annoying thing about the quota limits is they make it really awkward to actually fully utilize the 1500 premium requests you are paying for.
Like if you don’t plan working around the daily and weekly quotas you may not actually be able to utilize your full request allocation.
Claude has the same issue. Single session blows through the quota.
I tried my hand at this once from har file, build a DAG of the requests and responses and how their parameters map into each other, and then try to build a spec.
But it was very brittle. I imagine now with LLMs its much more viable. I always thought generating clients and load tests would be a cool use case.
https://docs.cloud.google.com/docs/quotas/view-manage
Quotas are real time or near real time.
Real time spend limits are probably never going to happen. Actual $ amounts are calculated by a centralized billing system offline in batch.
It sounds easy but it’s bonkers complicated, because of things like discounts, free tiers, committed usage, currency conversions and having to support every payment and deal structure in GCP.
Individual eng teams rarely actually think in dollar amounts, they think in the abstraction which is quotas.
also any open source local or self hosted options?
I think this begs the question, why should we expect AI adoption curve of Google devs to be any different from any other company?
Adoption, has been pretty rapid everywhere from what I can see, the tooling itself is still fairly unstable as everyone is rapidly iterating.
The productivity gains are there, but they seem much more modest than people like to admit (10-20%).
While google definitely has issues, this aint the root cause, even if there where only one root cause.
The first people betting on the outcome they want vs outcome they think will happen.
I think the second obviously make sense. If you had two experts arguing over how a war would go and they where willing to make a bet on it that is good because they are putting “their money where their mouth is” or have “skin in the game”. That’s a good thing.
The first implies manipulation or just a moralistic problem.
The issue I guess is it’s not possible to have one without the other. But I would guess most people are betting because of the second reason and not the first.
Do the work of indexing, or summarization or compression vs just do it retrieval on the raw records.
It could be embeddings, or presorts or indices.
But I think the parent poster is correct. There’s an inherent write vs read tradeoff as well as memory vs compute. Maybe with some generalization vs specialization.
Idk if im explaining properly. But you can always front load more effort on a narrower problem space and then utilize it on that space downstream.
I think with your llm example there is no code/data split. The llm is caching a solution to a narrow problem by writing the code.
But would the extra complexity, hardware, maintenance and associated costs ever be worth it for very small minority of transactions that would only be able to be cash.
It comes with a bunch of extra problems as well especially around abuse.
Better off just implementing gift cards to be bought with cash. Or maybe that’s what op meant?
It’s not even possible to summon one without a phone/app.
And forcing to always orient itself with that repo map first seemed to really help it from tunnel visioning.
What seems to be somewhat working for me
1. Karpathy wiki approach
2. some prompting around telling the llm what to store and not.
But it still feels brittle. I don’t think it’s just a retrieval problem. In fact I feel like the retrieval is relatively easy.
It’s the write part, getting the agent to know what it should be memorizing, and how to store it.
I remember thinking really hard on what to do with color. Except like you say mine is pretty much a naive fft.
https://github.com/aleksiy325/PiSpectrumHoodie?tab=readme-ov...
Thanks for reminding me.
Folders give you hierarchical categories.
You still want tags for horizontal grouping. And links and references for precise edges.
But that gives you a really nice foundation that should get you pretty damn far.
I also now am telling the llm to add a summary as the first section of the file is longer.
https://en.wikipedia.org/wiki/Dead_man_zone - is related to bush fires but seems like it could apply to a battlefield?
https://en.wikipedia.org/wiki/No_man%27s_land
Something more akin to actually being in "measure" or strike distance vs just contested territory in between?
Edit: Sibling comment I think clears it up https://en.wikipedia.org/wiki/Kill_zone
I've been trying a new approach I call CLI first. I realized CLI tools are designed to be used both by humans (command line) and machines (scripting), and are perfect for llms as they are text only interface.
Essentially instead of trying to get llm to generate a fully functioning UI app. You focus on building a local CLI tool first.
CLI tool is cheaper, simpler, but still has a real human UX that pure APIs don't.
You can get the llm to actually walk through the flows, and journeys like a real user end to end, and it will actually see the awkwardness or gaps in design.
Your commands structure will very roughly map to your resources or pages.
Once you are satisfied with the capability of the cli tool. (Which may actually be enough, or just local ui)
You can get it to build the remote storage, then the apis, finally the frontend.
All the while you can still tell it to use the cli to test through the flows and journeys, against real tasks that you have, and iterate on it.
I did recently for pulling some of my personal financial data and reporting it. And now I'm doing this for another TTS automation I've wanted for a while.
I was really excited about the concept and read some papers on it at the time. Then I ran a real world application of it, and writing code in it was just a bad experience.
I think I didn't explain the gradual typing properly. I mean the broader concept of moving developers from one way of doing things to another.
When the vast majority of people and tools are working a certain way. Introducing a whole new thing that requires them to work a whole different way is a hard sell.
So having a way for people to move gradually from point A to point B at their own pace and pick up benefits along the way is really nice.
One example of this is Typescript with gradual typing. Swapping form js to typescript is fairly easy. You can still use your old tools and over time you can start to adapt new features into your code, and over time everyone gets used to this new stricter/better approach.
I think ast based code like this essentially requires you to throw out everything all at once, not just editor, but literally everyone's workflow today and rebuild from the ground up.
I just don't think it will ever be more than a niche thing unless it can loosen up on strictness and play nice with existing tooling.
It was an experimental language that they wanted to replace their current application level language and was built ontop of JetBrains MPS https://www.jetbrains.com/mps/ which has that feature among others.
My personal opinion after working with that lang is all of this is that its theoretically interesting. But a dead end in practice along with visual coding etc.
The universality, simplicity, and legibility of text as interface for both humans and machines is too hard to beat. I think LLMs is just the most recent example of this.
Things that don't work in practice:
- You need a special editor, its heavy and slow, you lose all the ecosystem around them.
- You can't just cat or inspect the raw file, you can't see it in terminal at all.
- You need a new version control system, review system, and people need to learn it.
- You can't use any existing tools for working with code, you are essentially starting from scratch, and lose all the benefits of all the work everyone else is doing around tooling, dev saas etc.
- humans don't think in trees, they don't think in syntactically correct programs. Its actually absurdly frustrating writing a program in only syntactically correct edits. 99% of the time you are coding, your work is probably syntactically and semantically incorrect.
The tooling that is able to either make the right tradeoff around strictness or allow the users to make the that tradeoff is what ends up being used in practice. A good example of this typescript with gradual typing, python with type annotations, etc.
I think these editors just fall in a bit too far right on the strictness scale.
https://jcpsimmons.github.io/Godspeak-Generator
Maybe another word list would be more appropriate however.
Say writing an interesting or novel story.
And was thinking about if feeding in prompts of random words, along with prompts grounding from a simulation would sort of push the llm into interesting directions for implementing an on demand narrative story.
A sort of randomized walk with llm.
I remember watching Terry Davis with this random word generator in his terminal that he would interpret as the voice of God.
Here I guess the seed is the Voice of Dog.