jedwhite

API clients seem like a good use case.

My view is that readability and ease of understanding have a real impact on auditability. Nondeterministic output also clearly has a significant impact on auditability.

The balance between readability and determinism for auditability partly relates to developer philosophy. Tech is famous for religious arguments. I have friends who hate AI coding, and want to avoid nondeterminstic tools at all costs. And other friends whose productivity has increased significantly, and who see the future of programming as natural language.

The quality of AI models and tools like Claude Code is improving fast, and there are many developers who find value in them, myself included. I built this to make life easier for developers who want to use AI tools for automation.

I find it much faster to parse and understand plain language than many code scripts I've seen. It was one of Python's great insights that people spend more time reading code than writing it. And there is a tradeoff in auditability between determinism and the ability to quickly read and understand what systems do.

There are clearly many people who find AI useful, and who are becoming skilled in its use as a tool. This is just a little tool that I put together for myself and other people who fall in that basket.

Learning where to use AI tools appropriately - how to constrain the dangers while maximizing the value - is part of the challenge. From using this particular tool for real work, it fits some use cases well, and can make things easier both to understand and share, as well as to write.

I hope it's useful for some other people wanting to use AI for scripting and automation.

I think that quickly understandable instructions are part of auditability. Not the whole thing, and their use needs to be balanced with safety and security. But an important part of it.

I accept there are plenty of folks who don't see AI tools that way. We're sharing this for people who see the value in this new approach, even though it is a fast-moving field and there are a lot of imperfections.

Any reasonably competent Claude Code user who is careful about setting permissions boundaries is no more going to delete their hard drive than a competent command line user would. There will be things that go wrong with AI, as before it.

In years of tech support, I've personally had to help people who neutered their Windows install or deleted files they needed. Those things happen and I'd argue they come down to skill issues, with AI or without. New tools have a learning curve.

I get that you think that's bizarre to see readability with AI-based tools as more auditable, and I really do understand that perspective.

I think there are a number of interesting ideas raised in the discussion about the use of LLMs and markdown for scripting. I'm doing my best to reply quickly and engage on the substance. I get the intent with pointing out my over-use of common phrases in the replies.

The same as running `rm -rf $HOME`. Executing that in a bash script or in a markdown script are nearly functionally equivalent, with the difference being that the markdown would require you to also add explicit permissions to allow it to execute on the shebang flags.

Claude Code supports a set of flags to control behaviour such as permissions, and both `--permission-mode bypassPermissions` and `--dangerously-skip-permissions` are examples of those.

The claude-run helper supports passing in those flags supported by Claude Code itself that are relevant to a shell-scripting like context.

It also adds a couple of convenience flags (`--aws`, `--azure`, `--vercel`, `--vertex` for cloud API key use).

Thanks, the runme tool looks useful. You could use that along with this approach for testing. Being very meta, it could extract the code blocks from your executable markdown files to test they run correctly as part of the unit tests for a markdown test suite!

The markdown format provides a way to clearly format and structure instructions that Claude Code does well understanding, with clear markers for structure/headings, tables and code blocks.

Making a markdown `.md` text file executable with Claude Code is effective in practice because Claude Code can easily understand the content.

I can never tell :)

But it's a good chance to explore the issue.

In practice after using this for real-world test suites and evaluations, the results with Claude Code if you do this sensibly are remarkably consistent. That's because you can still write the deterministic parts as the `./run_tests.sh` bash script (or `run_tests.py` etc).

So you're using the appropriate tools for the task at hand embedded within both traditional scripts and markdown scripts.

Examples: - A bash script summarizes text files from a path in a loop - A markdown script runs `./test/run_tests.py` and summarizes the results.

Tools like Claude code combined with executable scripts and pipes open up a genuinely new way of doing tasks that are traditionally hard with scripting languages alone. I expect we will see a mix of borth approaches where each gets used based on its strengths, as we're seeing with application development too.

It is a new world and we're all figuring this out.

[Edit for style]

One thing to consider is that the steps in the pipeline can be deterministic (the code executed) while the outputs (summaries, reviews, evaluations, explanations) may be nondeterministic. An example would be summarizing data calculated via a traditional script, and piping it to a report-format markdown script that generates the report and summarizes the results.

I agree that this is a choice by each person using tools like this, and that it is up to each of us as developers whether a tool like this suits the use case at hand.

My own view is that the world is rapidly moving to more human language programming tools, and that system automation and shell scripting will be part of this. There is a wide array of sensible potential use cases I can see between the two polarized views of "never use an LLM' and "let's vibe code system automation".

Thanks, it is definitely an underutilized concept. I think Pete Koomen is right that we will see many more tools adopt this approach. And I hope that Claude Code and Codex add direct support for this themselves too.

You can use this without letting the markdown scripts you write execute any code at all, whether that is via Claude Code or other AI tool in future.

The default permissions are to not allow execution. Which means that you can use the eval and text-generation capabilities of LLMs to perform assessments and evaluations of piped-in content without ever executing code themselves.

The script shebang has to explicitly add the permissions to run code, which you control. It supports the full Claude Code flag model for this.

This is absolutely a new type of nondeterministic tool, so you're spot on there.

One of the key things we realized starting to use it is that the approach allows you to mix deterministic and non-deteministic tools together as part of a composable chain.

So you can, for example, use LLMs for their evaluation capabilities with a natiural language script as part of a broader chain that wraps it in deterministic code, and that also can include and run deterministic code nested within the plain language script.

So it allows us to create pipelines that combine the best of both approaches as appropriate based on the sub-task at hand.

[flagged]

Congrats and welcome to the new public status! Forgive the non-substantive comment but that's awesome :-)

I'm Jed, the author. I wrote this guide up because I'm working on an AI search startup using agents, and have been asked by other founders a few times why their content doesn't show up well with AI search engines like ours or Perplexity or ChatGPT with search, or with agent computer-use like the new OpenAI Operator.

We're ingesting around 30m to 50m pages a day so have learned a lot about what blocks AI crawlers and agents. And I wanted to share some guidelines on how to make sure your startup or documentation or other content shows up as well as possible for agent use.

Congrats on the launch. Adding a memory layer to LLMs is a real painpoint. I've been experimenting with mem0 and it solves a real problem that I failed to solve myself, and we're going to use it in production.

One question that I've heard a few times now: will you support the open source version as a first class citizen for the long term? A lot of open source projects with a paid version follow a similar strategy. They use the open source repo to get traction, but then the open source version gets neglected and users are eventually pushed to the paid version. How committed are you to supporting the open source version long term?

Could this be used for doing team visualization dashboards - like metrics displays for a team with KPIs? Or is that outside the ambit? I'm asking because I feel like there is a gap between the easy set up and configurability of something like Geckoboard and the extensibility and power of something like Grafana. That aside, the idea of team spaces/boards with a dashboard feel that is as easy to use as Notion is neat. And the UX looks really nice. Good luck!

We migrated microservices from Heroku to Porter, and also from standalone VMs and K8s running on AWS to Porter. As a coder trying to do both dev and devops on a tiny team, it was life changing for me.

The key benefits for a small startup team are:

1. Effortless CI/CD: Deploying services on K8s clusters across different clouds becomes trivial. Setup a dockerfile in your repo, point Porter at it, deploy. We mostly run APIs behind AWS API Gateway.

2. Startup credits: You can use your existing credits on AWS, Azure etc.

3. Zero lockin: You can deploy in parallel and switch service providers.

4. Devops expertise: The Porter team have given us next-level hands-on support and help to figure out how to run things optimally. A lot of sensible defaults are built in. As a coder, they have knowledge of how to scale services effectively that (to be blunt) I couldn't match no matter how much time I spent trying to learn it as a lay person.

If you're a K8s and devops master, you probably don't need this. If like me you're a programmer with limited devops skills looking for the fastest and easiest way to just solve deployment and scaling, Porter is close to magic. Plus they have one of the most helpful and friendly teams I've worked with anywhere.

(edit for typo)

This other Axios post today about Meta's news leader stepping down also has some interesting points about the context in which this drop has taken place.

https://www.axios.com/2023/10/03/campbell-brown-meta-news-ex...

“After receiving a recommendation from the boards of the FDIC and the Federal Reserve, and consulting with the president, Secretary Yellen approved actions enabling the FDIC to complete its resolution of Silicon Valley Bank, Santa Clara, Calif., in a manner that fully protects all depositors. Depositors will have access to all of their money starting Monday, March 13. No losses associated with the resolution of Silicon Valley Bank will be borne by the taxpayer.”

Thank you. I haven't seen those and will check!

I purchased a book version many years ago, and it had a very useful practical approach to learn by doing.

Note here - they aren't changing what results are returned when you search with quotes, they are just highlighting what you searched with quotes in bold in the snippet displays in the search results page.

Congrats on the launch! What is your thinking with the on-prem deployments? Would that work along the lines of something like Hugging Face's new Private Hub?

Nice work with the write up and thank you for sharing this. The post is interesting, but I think the problem with the YCRank approach currently is that the labelling appears to be subjective opinion, at least if I understand correctly.

Based on the post, you've trained the classifier by labelling a couple of examples of company descriptions you liked better than each other, based on subjective assessments like "harder to execute" or revenue growth that aren't part of the data you're running the classifier against.

If so, you've done a nice job of training a classifier to predict which companies you personally are more likely to be interested in. To improve this, you could use past YC batch company descriptions and success data to have more useful examples and labels for training the classifier based on past data, and which isn't so subjective. That might produce some interesting predictions that are more generalizable (although I think you may need more data points than the description and basic metadata).

If I've misunderstood, it would be interesting to know a little more detail about how the data was labelled.

I've based this on the following: "To investigate this, I made a neural network, YCRank, trained it on a handful of hand-labeled pairwise comparisons, and then used the learned comparator to sort the companies in the most recent W’22 batch."

And then: "I biased my ranking towards what was “harder to execute” on" and "I also tended to rank favorably companies that were already making monthly recurring revenue with double-digit growth rates".

Those may or may not be good criteria.

Based on that, this is essentially what you could call a "DudeRank Classifier" because as The Dude in the Big Lebowski says, "Yeah, well, that's just like, your opinion, man" :)

As I suggested above, it might be more interesting to label the example pairs and train the classifier based on the original company descriptions of known past successful and unsuccessful YC companies.

Possibly there is some signal in the company descriptions and limited metadata from Demo Day alone sufficient to predict successful companies from a batch.

Good luck!

Disclaimer: I am in the W22 batch. Our startup (Andi) ranks pretty well here. And this also is just, like, my opinion :)

[Edit: You could also test the classifier against historical batches to improve it then also!]

Thank you! You're spot on there :)