HNHacker News
TopNewBestAskShowJobs

jacky2wong

187 karma · joined December 14, 2021

Email: jacky@floatlabs.io
submissionscomments
jacky2wong··on DeepEval – Unit Testing for LLMs
Thanks for the interesting question!

From what I've observed from both academia and posts from a number of top organizations (with seemingly the exception of OpenAI) - all the models are trained on test data. That's why OpenAI seems to report poorer results than the rest but perform the best in practice - they seem to have their own curated dataset.

I have arguably the opposite opinion here - I don't think we can "prevent" cheating but we should instead put in the work to create new exam questions and make it easier to do so. I feel "learning" the exam questions is (IMO) completely fine and should be expected (why should we have a static dataset if it diminishes the quality of the results) - much like having new exam papers every year for students.

What I expect going forward is for the next version of foundation models, there will also be smarter ways to synthetically create datasets from ground truths themselves (we made our first pass at it here - https://docs.confident-ai.com/docs/quickstart/synthetic-data... ).

I feel that data labelling companies will hold an edge and that the cycle of finding new edge cases, labelling, continue training will continue in the foreseeable future (see Karpathy's data engine flywheel in 2021/2022 Tesla AI day).

jacky2wong··on DeepEval – Unit Testing for LLMs
Resolved all issues - thank you!
jacky2wong··on DeepEval – Unit Testing for LLMs
Interesting - I'm not really sure how this would work but I'm happy to discuss this in more detail - I see this being as a suggestion for what to do when tests fail. Happy to discuss this via email if you'd like.

Feel free to email me at jacky@twilix.io

jacky2wong··on DeepEval – Unit Testing for LLMs
Thanks for the enormous amount of interest and questions in this post. I wanted to make a follow-up comment to answer all the questions above about differentiation, where we see issues in the current solution space and clarify the problem we are trying to tackle.

Starting with the problem - the problem we are trying to tackle is to make iteration of LLMs and Agent applications as easy as possible for data teams - not for people to quickly edit and compare prompts (although that is 1 way of doing it). There are a number of solutions out there to help you test prompts etc. but many fail to fit within the data team's workflow (which consists of tooling like Pytest and CLI-first approaches) and do not make it easy to iterate and launch on things like agents.

From conducting 30+ interviews with ML Engineers and data scientists building in this space - all of them want to first build a LangChain agent/RAG pipeline and then build their own internal version of it (due to the fact that LangChain is quick to set up but lacks tooling). A lot of them are encountering issues with developing the right evaluation infrastructure (which DeepEval aims to solve through synthetic data creation and easy-to-use testing tools).

Our product roadmap is to not only build the initial unit testing for LLM but also to make it easy for other developers and other MLEs to quickly iterate off this. This means long-term, our plan is to ensure that our users and customers are able to build the best agents/LLM solutions possible.

And (to be frank) a lot of the existing solutions aren't the best looking, have limited visualisations (or are just dead)

jacky2wong··on DeepEval – Unit Testing for LLMs
See reply above^ :)
jacky2wong··on DeepEval – Unit Testing for LLMs
Agreed. No one should ever have to touch YAML for writing unit tests for LLMs. Ever. Most people writing agents and LLM applications are Python developers/data scientists/ML engineers.
jacky2wong··on DeepEval – Unit Testing for LLMs
Hi thanks for letting us know - please feel free to let us know of any more bugs - we'll make sure to write better docs and fix the code. We're just a few days in!
jacky2wong··on DeepEval – Unit Testing for LLMs
Thanks. I've reviewed all of those solutions and happy to go into more detail in a later post.

Here is our take: None of these solutions are built to fit seamlessly with the data team's workflow to collaborate and work together on (imagine the difference of Modal and WandB had on data / ML teams). We have built and consulted on a number of LLM projects and found that prototypes were super easy to make. Iteration was incredibly difficult. Our goal was to make iteration as easy as writing a unit test.

A number of these projects are missing most Python developer's workflow. For example - we made it compatible with Pytest on day 1 because that's the most common testing suite.

Many competitor tools all make a good pass at these but our roadmap is significantly different and not catered for normal web developers etc. Our focus on building this is to ensure the first step makes sense - that we are super intuitive to add to a data team's Python workflow. (Just 1 assert statement to write a unit test).

jacky2wong··on Tackling the Weaknesses of BertScore
In this documentation for Confident AI, I outline a few weaknesses to BertScore and why it needs to be coupled with entailment to be a useful testing component for LLMs.
jacky2wong··on DeepEval – PyTest for LLMs
Deepeval aims to make writing tests for LLM applications (such as RAG) as easy as writing Python unit tests.

For any Python developer building production-grade apps, it is common to set up PyTest as the default testing suite as it provides a clean interface to quickly write tests.

However, it is often uncommon for many machine learning engineers as their feedback is often in the form of an evaluation loss.

With the advent of agents, LLMs and AI, there is yet to be a tool that can provide software-like tooling and abstractions for machine learning engineers where the feedback loop of these iterations can be significantly reduced.

It is therefore important then to build a new type of testing framework for LLMs to ensure engineers can keep iterating on their prompts, agents and LLMs while being able to continuously add to their test suite.

Introducing DeepEval.

jacky2wong··on Show HN: Structured output from LLMs without reprompting
Point 1 doesn't feel like a good enough reason. The number of tokens outputted as a JSON is so small if you tell GPT to output it properly.
jacky2wong··on A Chatbot for Microsoft Guidance
I built a chatbot for Microsoft Guidance to learn what the Github repository was all about. For this, I parsed the Jupyter Notebooks and indexed them using a sliding window and then hosted the chatbot on my company's server. Posting it here if people find it interesting. There are 2 versions - a simple question-answering bot (Gpt 3.5-turbo) and then a copilot version (which uses GPT-4) under the hood.

If you found this helpful or if you have suggestions on how this can improve, you can send me an email on jacky@twilix.io

Hope this helps people navigate the (somewhat) difficult docs of Microsoft Guidance.

I wrote a guide on how to do it here: https://docs.twilix.io/tutorials/github-integration

jacky2wong··on I Built an AI News Aggregator
There are so many websites popping up with interesting AI news that I wanted to compile them into just 1 website. Just wanted to share with this community. I currently added: - KDNuggets - r/learnmachinelearning - OpenAI - Google AI - Berkeley - MIT - AWS (not sure why - but might as well)

Open to adding more if others are interested :)

jacky2wong··on I reviewed 70 GitHub Repositories
Thanks - that's really odd. Taking a look.
jacky2wong··on I reviewed 70 GitHub Repositories
I’ve been looking around for a website that helped me better evaluate common open-source frameworks and give me unbiased positive and negative reviews but unfortunately I couldn’t find one. I then decided to build my own and reviewed 70 different Github open source repositories, outlining pros and cons of each in a few sentences and then scoring them out of 5 in terms of how I found them. Where possible, I then provided alternatives to these open-source alternatives and reviewed those too.

The reason I built this is because I wanted to share my opinion of software with other people - and in particular help others avoid really bad rabbit holes or find good alternatives that I wish I just knew that I think could have saved me a lot of time and avoid experiencing change cost. The reason I built this is because I couldn't find information readily online and ChatGPT showed me how to build it but I didn't trust it to know the most up-to-date github repositories. Anyways - please check it out and let me know what you all think!

I've also made it so other people can add their reviews of popular open source repositories as well if they would like.

You can check out the website here: colabdog.com

You can check out a video demo to see all the features: https://www.youtube.com/watch?v=nYtVGsXKzxk

jacky2wong··on Prioritize visually with a value/effort board
I wanted to share a free tool that I built for better value/effort prioritisation to help everyone with discovering influencers. You can find a YouTube video of it here: https://www.youtube.com/watch?v=AOLsemUKSE0

Please share feedback on what others think can be improved!

jacky2wong··on Show HN: More Aesthetic GitHub Repo Search
You can watch a video demo here. https://youtu.be/y0uzgl7aDf4
jacky2wong··on Show HN: Side-Project: FlowChartGPT – Turn Text into FlowCharts
Hi everyone - I've updated the website, fixed all the bugs and migrated it over to a new React framework with new added functionality. I have also included screenshots of results and a demo video of what to expect upon signing up.
jacky2wong··on Launch HN: Pynecone (YC W23) – Web Apps in Pure Python
This is great. I received a lot of comments on my previous Python web apps saying it looked kind of scammy or it looked really old. Glad this came out :) Nothing beats true productivity like a frontend Python stack.

Streamlit was nice but it was built for newbie web developers - not web developers who didn't want to touch JS/TS - whereas this hits that niche quite well.

jacky2wong··on Show HN: Turn Text to Flowcharts
Ohhh - okay. I can build that for you - what's the use case? That would help me judge the quality of the infographic!
jacky2wong··on Launch HN: Vellum (YC W23) – Dev Platform for LLM Apps
Congrats on launching!

I personally think the target audience for this is a little hard to find when compared to products like langchain that do something similar already (i wouldn't be surprised if you guys built on top of this).

As a developer, I wouldn't have much difficulty spinning a Colab instance and running Langchain (takes a few minutes) and get it up and running compared to a solution like yours. Would be awesome to get a pros/cons table of a solution like yours compared to Langchain so developers can best figure out how to dedicate their time without having to try both tools.

jacky2wong··on Show HN: I trained an AI model on 120M+ songs from iTunes
Just tried it - I Walk Up by Luke Christopher and Memories is not too bad! Admittedly - encountered quite a few problems as part of this though - with a few random pop songs in the list.
jacky2wong··on Replacing a SQL analyst with 26 recursive GPT prompts
Having worked in large corporate enterprises where the visualisation of data engineering and navigation to the relevant code-base was incredibly difficult - I see a lot of value in this. I think this is an absolute game-changer for engineers due to the often outdated documentation of the pipelines otherwise!
jacky2wong··on Show HN: Turning books into chatbots with GPT-3
Interesting project - interesting way to analyze data using generative models - I wonder how confident it is in its answer though.
jacky2wong··on Cal.com: Open Scheduling Infrastructure
What ends up happening is that B2B companies prefer 'value' over 'simplicity' or user experience and end up building features without proper product design. This is required for them to expand their B2B offering and hence end up being 'clunky'. It's inevitable - much like most enterprise softwares.
jacky2wong··on How to Stop Procrastinating on Your Goals by Using the “Seinfeld Strategy”
Sounds like something that only fits creatives IMO
jacky2wong··on How We built a $1M ARR open source SaaS
Great read! Thanks for posting
jacky2wong··on Show HN: Jless, a command-line JSON viewer
This is awesome! A great add to our command line utilities!
jacky2wong··on StackOverflow Is Down
Back up :)
jacky2wong··on Ask HN: Would you hold two full-time remote jobs?
Depends on speed of the individual. Definitely think if you can do it and balance outcomes on both - I don't see any problem here. Issue arises if there are conflicting interests! Having worked multiple jobs before - definitely very doable.
← PreviousPage 2 of 3Next →