tikkun

Autogpt subreddit and discord, and the babyagi discord

What’s the best alternative to iCloud Drive for macOS?

Dropbox is so heavy and uses system resources, I’d love a lightweight one that I can trust

I've used it (no affiliation to the company).

I like it because it's easy. I'm not sure what the alternatives are, but I have the sense they'd all be more complex and more annoying to use (even if they might be cheaper).

Do you have an association with them or just a happy user? Either is fine of course

Here's the note I have on that: “For chatbot interfaces, emerging approach is to have another agent simulating the user (as opposed to a more classic approach based on token prediction probs on chat transcripts, what I think you're referencing). Then still use a model for grading. Only place I've seen this so far: https://github.com/Forethought-Technologies/AutoChain/blob/m... ” - AI Startup Founder

Then there's also monitoring. My notes from the monitoring section are below:

When needed: “it goes hand-in-hand with eval, as you need to be able to turn bad prod generations into failing eval cases for eng to make pass”

Considerations:

1) “Ability to monitor custom metrics (ROUGE [1], Coherence, etc.) and slice-and-dice data - customizability; non-intrusive logging vs proxies, VPC (enterprise-readiness). Plenty of tools w/ basic cost, latency monitoring; very few w/ enterprise-grade customizability, anomaly detection, etc.”

2) “+agent/pipeline tracing, ability to re-purpose data for fine tuning, connection to user feedback, man-in-the-middle approach (proxy) vs SDK integration (we believe SDK is superior so your monitoring vendor can go down without taking down your LLM feature)”

Companies for LLM monitoring: Helicone, Honeyhive, Gentrace, Humanloop, Langsmith, Pezzo

[1]: https://en.wikipedia.org/wiki/ROUGE_(metric)

I've been looking into this question for a bit. [1]

Here's my notes on evals --

Things to consider when comparing options:

1) “Types of metrics supported (only NLP metrics, model-graded evals, or both), level of customizability; supports component eval (i.e. single prompts) or pipeline evals (i.e. testing the entire pipeline, all the way from retrieval to post-processing)”

2) “+method of dataset & eval management (config vs UI), tracing to help debug failing evals”

3) “If you wanted to go deeper on evaluation, I'd probably also add:

What to evaluate for:

- Hallucination

- Safety

- Usefulness

- Tone / format (eg conciseness)

- Specific regressions

Tips:

- Model-graded evaluation is taking off

- Use GPT-4, GPT-3.5 is not good enough [for evals]

- Most big companies have some human oversight of the model-grading

- Conversational simulation is an emerging idea building on top of model-graded eval” - AI Startup Founder

---

Here are a few that people are using for evals at production scale:

* Honeyhive https://honeyhive.ai

* Gentrace https://gentrace.ai

* Humanloop https://humanloop.com

* Gantry https://www.gantry.io

I've done calls with the founders of three of those four, and I've talked with enterprise customers who've been evaluating a couple of those.

I see there's a few others mentioned in this thread (langfuse, truera, langkit/whylabs) that I haven't heard about from customers but also look promising. There's also langsmith which I do know is popular amongst enterprises (enterprises hear of langchain, see that they have a big enterprise-oriented offering) but I haven't talked with anyone who uses it.

Then for evals at prototyping scale there are various small tools and open source tools that I've collected here: https://llm-utils.org/List+of+tools+for+prompt+engineering

[1]: I'm working on an AI infra handbook. Email me, email in profile, if you can review/add comments to my draft. It's 23 pages long :x

Google Docs, Airtable, Apple Notes, GPT Playground, a mail client, reddit, discord

How to? Is there an easy thing with cloudflare or similar and what’s the pricing

What are some better ways to do it?

I'm interested in the same thing and have spent quite a bit of time looking.

Rewind.ai is ok (transcription accuracy is meh)

Voice Memos.app is ok (though no native transcription, and requires stopping and starting)

Otter.ai is ok (though there's a 4 hour limit on recordings, and there's no paid plan that allows for enough recording minutes to do 24/7)

My ideal solution would be that Otter comes out with a Pro 24/7 plan with 60,000 minutes per month and no max recording length, for $60-80/mo.

I would pay for this and have paid for alternatives, though I'd prefer to use an existing company that I've used for a while and that has lots of users, due to privacy/trust, or perhaps a small startup that publishes security reports and does everything on device.

As an aside:

I use 24/7 voice transcription as a kind of "extended context window" (to use an LLM analogy). While I'm working, I talk out loud to myself about what I'm thinking through, which I find allows me to effectively increase my working memory size to be much larger than otherwise. It's quite helpful.

Not quite what you're asking, but see this list I have: https://llm-utils.org/Movies+and+documentaries+about+startup...

e.g. Pirates of Silicon Valley, Startup.com, Silicon Valley - documentary about Fairchild Semiconductor (2013) - suggested by fullshark on HN

The Blackberry movie was also enjoyable

I suspect many companies will do something like this - prepaid credits or tokens for AI features that have high inference costs. Inference costs per user are high, at least much higher than most traditional software costs. This way, the costs are aligned with the usage.

We’ll also see occasional subscription products, but only when it can be done in a way that is comfortably gross margin profitable for most users of a company. (Eg ChatGPT Plus, Claude Pro, Midjourney, 365 Copilot)

This will only change when the cost of inference goes down by a lot.

I think most open source LLM leaderboards are bad - I wish they used different benchmarks to rank the LLMs. The benchmarks they use test metrics that are irrelevant to most users. Also, if you've ever looked inside the benchmark datasets (I have) - it's not the stuff that I'd want dictating the rankings on the LLM leaderboards that influences what LLMs people use!

I have ideas on making better benchmarks (in short: take actual ChatGPT session history, and actual LLM API production use queries, and make benchmarks based on those), but for now, I think this is the best LLM ranking that exists, because it's more of a test of how well the LLM can use tools, and it more closely aligns with my personal experience of how useful different LLMs are in day to day usage.

Here's the ranking from the AgentBench paper: https://raw.githubusercontent.com/THUDM/AgentBench/main/asse...

This is a helpful checklist, thanks for making it

Yes

https://www.ycombinator.com/jobs/role

Large portion of the startups listed here and on other job boards are not remote. If they don’t list any remote roles but do list a city then they’re generally (but not always) fully in person.

A year ago it was tough to find fully in person companies, but now a large portion have moved back to in person.

Side note: if you can, check your thermostat and see if you can set your thermostat’s “fan mode” to on, instead of auto, for all the schedule settings that apply while you’re in the house. Ecobee allows this (comfort settings -> select presets that apply while you’re in the house -> change heat mode fan and cool mode fan to on). It’s the most effective way I’ve found for lowering CO2 for rooms that don’t have windows or when opening a window is impractical.

Aside: This is a great title.

It shows what you made (a dataset of Gumroad products), why/to whom it matters (to spot market opportunities), and it implies how (dataset/scraping). Nice!

Related concept: Enshittification (I've seen this mentioned on HN)

A company starts out focused on building a great product.

Then, they reach product-market fit.

Then, they raise money.

Now, the best way for them to grow is to go upmarket. Sell to enterprises.

Unfortunately, then the resources of the company are directed to features that enterprises buyers care about, because that's the best way for the company to make more money.

The actual product stagnates.

(This does leave an opportunity for bootstrapped founders to come in and swim in the "wake" of B2B startups that built a great product, then let their product stagnate while they went upmarket. Eventually some of those bootstrappers might choose to get bigger and move upmarket themselves, and then the opportunity opens up again for new bootstrappers. Or the bootstrapper can remain niche, focused on winning users who choose the best product and who don't require enterprise features, which is a smaller market but still can be quite profitable and would be a much smaller and simpler to run business.)

Examples: Airtable (it's a great product, I still love it, but it hasn't really improved for me as an end user since about 2016), Notion, Docusign (maybe it wasn't ever a great product, I'm not sure, but it sure hasn't improved for me as a user).

Instead of going from "good to great" it unfortunately seems like many startups go from "great to good"

In addition to Carta (which has a sentiment of being overpriced), there's also https://pulley.com [1], and LTSE Equity [2] [3] (formerly Captable.io). And spreadsheets.

[1]: https://pulley.com/compare/pulley-vs-carta & https://carta.com/carta-vs-pulley/

[2]: https://equity.ltse.com

[3]: https://carta.com/carta-vs-captable-io/

Mercury is great, best bank I’ve used (Mercury - please build personal banking too!)

For federal you do https://www.irs.gov/payments

For California state you do https://www.ftb.ca.gov/pay/index.html

It's a payment, not a filing. Go through the payment links and you'll see that you can select 'estimated tax' as the reason for payment

I'd suggest checking with your accountant, to make sure that you're doing it right. Just say hey here's what I'm planning to do, these amounts, on these dates, using these links, does that look right.

Note that depending on your setup you might need to do multiple. E.g. I do quarterly personal estimated IRS payments, personal estimated FTB payments, and corporate estimated FTB payments.

I hope that entities like large VC firms and accelerators like YC and Techstars are lobbying for software development expenses to continue to be treated like regular expenses, rather than amortized. Are they?

Looks really well executed, nice! I'd shared this idea with a few people. GPT and other LLMs don't allow you to use their output to train competing models, but the implication is that it's fine to use their output to train your own internal alternative models. So you can't sell access to the output as an API, but you can use it to replace your GPT API calls.

My other thoughts to extend this are that you could make it seamless. To start, it'll simply pipe the user's requests to OpenAI or their existing model. So it'd be a drop in replacement. Then, it'll every so often offer to the user - "hey we think at this point there's enough data that a fine tune might save you approx $x/month based on your current calls, click the button to start the fine tune and we'll email you once we have the results" - and then the user gets the email "here are the results, based on that we recommend switching, click here to switch to calling your fine-tuned model" - Helicone and the other monitoring platforms could also offer something similar. (Side note I'm working on an "ai infra handbook" aimed at technical people in software orgs looking to deploy unspecified "AI" features and trying to figure out what to do and what resources they'll need - it's a 20+ page google doc, if anyone can help me review what I have so far please let me know and I'll add you.)

If it's latency/error/speed competitive, and cheaper, and equivalently accurate, then for anyone doing production scale LLM API usage it'd make sense to use something like this - either the fine-tune is worse so you keep using the regular API, or the fine tune has parity plus cost and/or speed advantage, so you switch. (It wouldn't make sense for prototyping scale, because the additional complexity of the switch wouldn't be worth it unless it could save you 4/5 or more figures a year in API costs I'd think.)

No I mean more end-consumer oriented ones. You're probably right, though I still think it'd be nice to see more competition (plus, more competition tends to make companies behave better, because they know users have compelling alternatives).

Most LLM popular libraries are inactive, but a handful are very active. That's a product of it being a hype/popular/new field, where new projects got attention from being new, they didn't need to be ongoing to get attention. (This is fine - it serves as a way to reward people for exploring early on in a new field, which is helpful for new ecosystems)

People saw that an easy way to get "github status" was to create LLM projects. Some of the LLM projects were created to be cool tools that became popular, and the devs don't have as much interest in continuing them.

This is a natural thing to be expected in new and popular fields.

There are still many LLM libraries that are very active.

It's kinda the same as asking - why are most blogs dead projects now? The author had initial excitement, created it, shared it, the excitement/development didn't continue, but also we still have a ton of great blogs that exist on the web from the authors that did continue.

Did you know that new Show HN posts go to /shownew, not /show, and will stay on /shownew until they get a few (maybe 1, I'm not sure) upvotes. I learned this a few weeks ago.

Based on my experience with other AI tools, I am now skeptical of over-promising from AI tools. It could be useful to add screenshots and videos to the landing page, to show a realistic perspective of what Mavex is and isn't capable of, so that I and other users can tell whether it'll be useful for us, and so that it's easier for us to believe the promises.

No, my M1 and M2 devices (and my A15) have all been the same as normal.

I compared alternative analytics things a few months ago and ended up going with Clicky. It's pretty good. But, I don't check it often, talking to users is definitely more useful than analytics to me.