HNHacker News
TopNewBestAskShowJobs

Joschkabraun

26 karma · joined April 9, 2020

submissionscomments
Joschkabraun··on Anthropic's Haiku Beats GPT-4 Turbo in Tool Use
What do you mean with being a bad model?

If the model is really good at tool use, then it will broadly useful as it needs capabilities to generate the tool definition. So, there should be some transferability.

Joschkabraun··on Anthropic's Haiku Beats GPT-4 Turbo in Tool Use
Ah yes. Have you tried out instructor [0] or Guidance [1]?

[0]: https://github.com/jxnl/instructor/

[1]: https://github.com/guidance-ai/guidance/tree/main

Joschkabraun··on Anthropic's Haiku Beats GPT-4 Turbo in Tool Use
Interesting. Do you have any benchmarks?
Joschkabraun··on Anthropic's Haiku Beats GPT-4 Turbo in Tool Use
All models got the same prompt fed which was essentially "Question: {question}". And then the API's accept the function call definition
Joschkabraun··on Anthropic's Haiku Beats GPT-4 Turbo in Tool Use
Have you tried the new beta tool use API? In the experiments I ran there were almost no issues parsing the function call response (similar to GPT-3.5-turbo & GPT-4 turbo)
Joschkabraun··on Open-source tools for LLM monitoring and observability
https://docs.parea.ai/observability/logging_and_tracing isn't open-source for LLM monitoring but the evaluation metrics to assess LLM app quality are: https://github.com/parea-ai/parea-sdk-py
Joschkabraun··on Ask HN: How are you testing your LLM applications?
Co-founder of Parea here, thanks for the mention!

We offer testing/evaluation in development ([1], [2]), and production ([3]). You can use pre-built evals ([4]) or create your own ([5]). Do logging ([6]) and go from trace to playground ([7]). For TypeScript & Python.

[1]: https://docs.parea.ai/evaluation/offline/experiments

[2]: https://docs.parea.ai/platform/test_hub/benchmarking

[3]: https://docs.parea.ai/evaluation/evals-in-trace

[4]: https://docs.parea.ai/evaluation/list_of_autoeval_functions

[5]: https://docs.parea.ai/evaluation/evaluation-functions/create

[6]: https://docs.parea.ai/observability/logging_and_tracing

[7]: https://docs.parea.ai/observability/open_trace_in_playground

Joschkabraun··on The first playground to experiment with the GPT-4 Vision API
Happy to launch the first LLM/LMM prompt playground which allows experimenting with the GPT-4 Vision API! Curious to hear what you guys are thinking
Joschkabraun··on When gradient descent is a kernel method
Reminds me of "Every Model Learned by Gradient Descent Is Approximately a Kernel Machine" by Pedro Domingos: https://arxiv.org/abs/2012.00152
Joschkabraun··on How has the ChatGPT model changed from March to June?
Yes, GPT-4 is in general much more tuned to respect the system message.