167 karma · joined November 14, 2019
> There are some promising other directions I want to explore, such as coupledness of functions, code churn, cohesion and so on. If you are working on evals and would like to talk, I would be happy to do that: sebastian@earendil.com
Hey Sebastian, I just read your article and it thoroughly resonated with me. I've been working on building something similar to SlopCodeBench, but moreso aimed in the direction of architecture, rather than simple one off "code search functions".
In a nutshell, I'm creating multiple domains of common software architectypes. You can think of these as being as simple as a counter (very common in all architectural explorations worth their weight), todo applications, etc and as complex as an online store, a bank, a wallet, a social communication platform, etc.
Given a single domain, we can extrapolate common functionality that is "higher order" to that domain. Features like data synchronization, functioning offline, sharing information, authentication and authorization, etc all land in this bucket. From a single domain (take the counter, for example), I've laid out my initial plans for the various different levels to concretely observe how bad LLMs are at churn, cyclomatic complexity, poor abstraction planning, etc as follows:
L1: Show a number on the screen L2: Allow a user to click plus or minus and the number responds accordingly L3: Show that number on any device running your software, and keep it in sync with all other devices L4: Ensure that additions and decrements to the count, while a device is offline, will replay to all connected devices once connectivity is reestablished. L5: Introduce the ability to reset the count to zero, and ensure that commutes properly if an offline device triggers a reset or vice versa. L6: Introduce user accounts where I must provide an authorization strategy (login with apple, google, passkey, etc) and now segregate a public counter (which anyone can interact with) from your counter (which only devices authorized with your credentials can interact with. L7: Introduce an action menu whereby I can invoke what is commonly known as a "Command K" menu for the actions that can be performed in the application. L8: Allow me to speak naturally to your application and ask it to "go up", "start over", etc (this exercises your architecture's ability to be "accessible" to agents (and vision disabled folks as well) ... ... ... And so on and so on.
Despite its apparent initial complexity, naive solutions to even the simplest domains will be easy to spot with how many lines were changed vs raw additions (how composable a solution is), that cyclomatic complexity measure you mentioned, how many tokens it took, how many platforms (iOS, android, cli, TUI, react native, react) your application will run on, how long it takes to build, how large the binaries are, how much memory is used during the operation of your software, how semantically similar duplicated code across different platforms etc. From all these different values, we can create a hueristical "architecture score" to benchmark against.
I'm also toying with the idea of enforcing that one must submit a bundle of skills, instructions, scripts, etc that I will exercise with my own harness whereby the submitter has to submit a monetary cost with their submission that a budgeting agent must manage the spend and the agents must yield prior to their submission being scored, lest they forfeit the submission.
I'm just quite tired off all the hype and its exhausting and AFAIK, none of the benchmarks actually produce anything of use. One interesting side effect of aligning incentives in the way I've laid out here is that we will have produced open source, connected software that functions well and solves a whole bunch of business needs that all compose together by definition.
Thanks again for your article, would love to have an e-coffee and chat about if there's potential to collaborate on anything here. Despite how powerful llms are, this is still proving to be a tricky endeavor for me.
Best, Michael
PS: here is a demo of my submission for this first round: https://x.com/technoplato/status/2090902061437030777?s=20
Boy, oh boy do I wish there was a way for an individual to be anonymous but verifiable as a unique individual operating on a consistently used device.
Sybil be damned.
Since bash can evoke any Turing complete program, feasibly anything could be done with one line of bash.
The authors intent as I saw was to show what that line was. Would you have known without this article?
Whether or not people are just being sarcastic, I feel we should definitely not mock but rather encourage any open and free attempts at teaching us cool things that are more novel than the “how to get started on Linux” type bs the market is inundated with.
Bravo to author.
god speed
Contact info on the photo
You could trigger this on any oscillating “body” if that body is oscillating at a certain frequency and amplitude combination.
It just so happens motorcycle engines produce that combination. There would certainly be others. tHe sustained nature of the shaking would also be cause for concern but I didn’t see any mention of duration in the support article
Yeah they actually go into more interesting detail than expected from a support article and detail how (not an expert so it could just be wool over my eyes Apple BS but doesn’t smell that way here) the OIS and closed loop AF work in that article.
Basically sensitive gadgets shouldn’t be exposed to certain frequency / amplitude combinations because sensitive gadgets rely on well, not being shaken out of whack.
What’s going to be the pricing model moving forward. Surely voice and such can’t be free in perpetuity.
Nice work!
Doesn’t have to be that brand obviously but highly recommend checking out the idea at large.
I deeply empathize with the goals of bridging the gap between the incredibly vocal minority opinion holders at both ends of the curve. I’ve been thinking about how to solve this problem and working on my own take, knophy [1], for a couple years now in my free time. Knophy is a content aggregation platform that creates an innovative new economic model whereby voting power is earned by first reading, watching, or listening to the content on which one desires to comment or “upvote” themselves. After consuming a piece of content, a user can then answer community moderated evidence-based questions to earn said voting power. Whenever questions are answered correctly, the creator, moderators of the question bank (the community) and the answerer all earn relative amounts of points with which to vote on content. Should a user develop a surplus of points, they may “burn” those points in exchange for a fungible, cryptocurrency token that they can either sell or speculate against.
This model has implications in solving the problem of content monetization, copyright, siloed walled gardens who rely on riling up their users to increase usage, moderation, and many more major problems which I would love to discuss in further detail if you find them intriguing.
I realize I haven’t done a great job introducing myself in the above paragraphs, but feel that the ideas we have in regards to solving this problem are more important than me blabbing about myself. I’m a full stack engineer specializing in React / React Native and am very excited about using the GRANDStack (GraphQL, React, Apollo, Neo4j Database) to create highly connected and innovative new applications at scale.
I'm currently very happily employed but would love to help innovate in this space in any capacity that would be welcome!
My contact info and some relevant works are linked here: [GitHub - technoplato/portfolio: A portfolio of my work](https://github.com/technoplato/portfolio)
I’ve also sent this via email to the specified address. If any HackerNews people are interested in critiquing or helping with this idea, I’m all ears!
[1] [GitHub - technoplato/knophy: WIP Knophy is a content aggregation platform where voting power is regulated and earned by demonstrating what you know about the content you follow.](https://github.com/technoplato/knophy)
When platforms ban assholes and other platforms arise in the guise of decreasing censorship, said assholes flock to new platforms and infest them a la Ruqqus (though I believe the developer/founders have positive intent)
We need more clever ways of finding the assholes. I mentioned show and tell in my comment above. When you think about how show and tell works, you must earn your right to show and that works fairly well, even when you scale the idea. Does that make more sense?
A retweet, share, and apparently an echo are all semantically close enough to be called identical functionality, save for perhaps upstream metadata associated with the source.
eg on Facebook I don’t know if, once you share a post, the total share count is displayed on the post itself. However on Twitter this share count is shown.
They’re all the same sheep with isomorphically identical data just dressed up in different pairs of clothes.
As someone very curious about language usage, why wouldn’t you just write the word “familiar” there.
I’m genuine curious as to your motivations. Is it brevity? Am I missing additional context this provides?
Surprising insights arise from pondering the philosophy of show and tell-specifically how listening is involved in order to “earn” your turn to speak.
Wonderful documentation is hard to beat from my learning perspective, but I find that level of quality is more rare.
Therefore, I learn better by watching something materialize on video at ~3x speed and often without any speaking.
I think it simply boils down to my level of trust that this person didn’t accidentally omit anything from their article or blog post or whatever and what I see on the screen is what I get. Now, that doesn’t fix when tools change etc, but videos instill a higher level of trust that, if I follow these steps, I will produce this output.
Are they artificially inflated and then deflated to counter act “bot” behavior?
OR
are they organically inflated and then censored?
I just wish I could be a fly on the wall and know. I just can’t believe this would be a damning enough piece to artificially inflate unless there’s a serious game of reverse psychology going on behind the scenes
Edit: I'm a moron and it just dropped a bunch suddenly. Was not removed.
Edit: I'm a moron and it just dropped a bunch suddenly. Was not removed.
"execa": "^4.0.3",
"glob": "^7.1.6",
"mkdirp": "^1.0.4",
"moment": "^2.29.1",
"nopt": "^5.0.0",
"npmlog": "^4.1.2",
"nyc": "^15.1.0",
"p-all": "^3.0.0",
"rimraf": "^3.0.2",
"tape": "^5.0.1",
"unique-slug": "^2.0.2",
"which": "^2.0.2"
Ffmpegs documentation on the other hand forces me to feel as though it’s normal to assume knowledge about a point in a video and what it means if my encoders are different from input to output.