seahyinghang8

founder of castform here, we believe that as agent deployment moves from experimentation phase where cost doesn't matter as much to deployment (what's the margin of serving the request), there will be a rise in interest in optimized models.

<founder of castform here> tldr: we generated synthetic training questions from the gitlab product handbook.

totally agree that this larger corpus with harder to search information would be a good way to stress test - i'm sure we will encounter more interesting problems to solve. love to hear any suggestions of corpus to search against that is not just the public internet

the training run link is also a little buried but here, you can see the comparison against the various models and their exact traces: https://app.castform.com/train/a7a898f6-d802-4908-b044-acb81...

we actually have the test benchmark against luna but no deepseek flash (we haven't added DSFlash into our benchmarking model pipeline)

you can check out the full comparison against all the other models here: https://app.castform.com/train/a7a898f6-d802-4908-b044-acb81...

- founder of castform

founder of castform here again - slightly unrelated to retrieval but on the topic that folks are discussing here, i was actually collecting benchmarking various coding traces for the purpose of training a model router and surprisingly, luna held up very well against sol and terra. it was able to solve close to >95% the that sol can handle at a fraction of the cost. have not benchmarked the OSS models yet but will add the popular ones to the list like Deepseek Flash and Kimi k3 to see how they fare.

will share the full results soon!

we actually have the test benchmark against luna too! it's just not in our title but you can see it in the first diagram below the title. luna does pretty well tbh but sol is just a tad bit better. but luna is way cheaper.

if you want to dive down into the various traces of the benchmark, you can check this out: https://app.castform.com/train/a7a898f6-d802-4908-b044-acb81...

- founder of castform

(founder of castform here) - we didn't get to dive too deep into the dataset we were using for the retrieval in the blogpost for brevity, but we did link the training run (which shows the dataset) here: https://app.castform.com/train/a7a898f6-d802-4908-b044-acb81...

the page shows the exact trace of all the models we are comparing against and the aggregate scores

we generated the question & answer pair from gitlab product handbook (https://handbook.gitlab.com/) since the point is to show that you can generate training questions from raw data corpus (something a company already has today)

(founder of castform here!) - having worked at FAANG / big tech, i totally get this. our example was on gitlab's open source company handbook but i think a real company's corpus is way more messy and has many sources of truth.

a few ideas i have yet to validate are: - prioritize recently updated docs when generating the training questions (assumption those docs are more correct than others) - actually including contradicting documents that talks about the exact same topic might be a good training example - ideally the model should surface all the relevant info it can find, and explain what it has found. (usually contradiction comes from the fact that the later document is the updated stance) - you could also mine high quality Q&A from public slack / communication channels where questions were asked and someone else in the team linked some docs / answer. those are strongly validated "ground truth" answers

i really hope that's the case

you should check out https://openai.com/index/gpt-5-5-bio-bug-bounty/

i use zed 1.0 agent interface and it's pretty great. cool to see that pokegent is modeled after zed's ACP. tho zed has been bugging out recently where the agents just stop working at all so i might give this a try

Xcode 16 recently released swift predictive completion but it requires ARM processor with more than 8GB memory. We made an Xcode "extension" that provides AI completion that works with both Intel and Apple Silicon macs. You can either connect to a model hosted by us for free or a local ollama model of your choice.

If you develop on Xcode, give our extension a try and let us know what you think!

If you are curious about how it works, check out the FAQ section of our docs here: https://seasoned-railway-d87.notion.site/CGFT-Xcode-Extensio...

gamma is lagging on ios safari...

This project was inspired from our jobs where we spent time writing technical documentation on the stuff that we've built. We'd thought it would be cool if GPT4-V could automatically convert a recorded demo of the product into documentation, that's why we built Vid2Doc! Would appreciate any feedback!

Oh I did not make it, here is the link to whom I believe to be the creator: https://github.com/copy/

I discovered it while browsing the web and thought it'd be interesting to share it

Hey HN! We're excited to introduce Charty AI, our attempt to change the way we think about and create data dashboards. Creating dashboards is a time-consuming endeavor, especially if you're not well-versed with the intricacies of data visualization libraries.

How Charty AI Simplifies Dashboard Creation:

1. Natural Language to Code: Imagine if you could just type "Plot a line graph of daily website visitors for the past six months" and get it instantly. With Charty AI, you can.

2. Concurrent Analyses: Juggle multiple data sources and visualizations simultaneously. Compare, contrast, and gain deeper insights without having to switch tabs or scroll up and down.

3. Drag, Drop, & Personalize: Once you have your visuals, tailor them to your preferences. Organize, resize, and stylize your charts to fit the narrative you're building.

Our goal is to make the dashboard creation process as seamless as typing out a sentence. Currently, our supported data sources are csv files but we might include other sources if there is an interest.

Would love to get your feedback, thoughts, and suggestions!

Inspired by Jupyterlite and GPT, I've build an IPython notebook that runs completely in your browser with Pyodide, with a GPT copilot, as part of Svelte's hackathon.

The chat interface allows users to seamlessly move code from the notebook to the chat to ask questions and vice versa, make coding more accessible to everyone.

Link to source code: https://github.com/seahyinghang8/zeus-notebook

Hey HN! We made this extension that lets you run any python code on the web by using Pyodide. You can try it by installing this VSCode extension on GitHub.dev / VSCode.dev. Since it is using pyodide to run python code, there are some limitations which we have listed in the extension’s README page. We made this scrappy extension quickly as a POC for Stanford’s Code in Place program. Let us know if this is interesting and what features should we add next!