roh26it

What do you use for the onboarding guide?

Well written, but does this all change in a claude-ified world?

Looks solid, going to try it out.

I'm going to be __that guy__, but just ask - is the functionality set similar to llamaparse or is this llamaparse + llm?

So, while time to first token is lower, throughput might also be lower in most cases?

What are the trade-offs you've made to achieve this?

The most interesting findings (at least from my perspective) - OpenAI measured "honesty" that could become an interesting model eval others should adopt. - Using instruction heirarchy to _avoid_ prompt injections - When pushed to pursue misaligned goals, it tried to disable oversight in 5% of tests!

I've been thinking about AI guardrails in production for almost 7 months now, and I'm convinced that using them to "control", "secure", or "block" LLM outputs is not the best use for them .

Using them to "guide", "accelerate", and "empower" your AI development process? Now we're talking.

Let me explain why I believe this shift in perspective could change how we build AI...

Most people see AI guardrails as training wheels. But what if we saw them as a nitrous boost for our AI hot rods instead?

When you're not worried about your AI going off the rails, you can push it to its limits. You can experiment with wilder ideas, test bolder hypotheses, and innovate at lightning speed.

Want to train on spicy data? Guardrails have your back. Thinking of an aggressive model architecture? Guardrails keep you in bounds. Controversial prompt? Guardrails ensure you don't cross the line.

Am I in the right direction, or am I just going off the rails?

Also wondering if this would become expensive very fast if it ends up using S3 with a large number of PUT calls

thanks for the support!

We use a bunch of caching mechanisms on the LLM requests themselves and extend the same to guardrails now.

So there's 2 levels of cache - the LLM request itself might be cached (simple and semantic) and the guardrail response can be cached as well.

We use a mix of a distributed kv store and a vector DB to actually store the data

Being one of the most downloaded datasets on Huggingface, I was a little bit surprised by how dirty this dataset was. Plus it had very limited information and some incorrect classifications as well.

For an internal experiment on building a "Truthful Evaluator", we picked up this dataset and tried fine-tuning a model on these 8000 odd examples.

Realised that it needed: 1. Cleaning up 2. Some reclassification

But, most importantly - it lacked context data. It only had a link pointing to the source which was also absent for a few rows.

We scraped the internet for the link in the dataset, matched it to the question and narrowed down on a small context to be added to the main dataset.

Releasing it publicly so that someone else may avoid the 2-3 days of pain of wrangling with this data.

Here's a mega guide on keeping costs low with LLMs - https://portkey.ai/blog/implementing-frugalgpt-smarter-llm-u...

tl;dr: - Keep prompts short, combine prompts or make more detailed prompts but go to a smaller model - Simple and semantic cache lookups - Classify tasks and route to the best LLM using an AI gateway

Portkey.ai could help with a lot of this

At Portkey, this is a problem we deal with quite a bit. Also the reason that Datadog and the traditional observability vendors did not work for LLM use cases since they're not built to handle large volumes of data.

We've done this through a careful combination of Clickhouse + MinIO for fast retrieval of log items + selected retrieval from the MinIO buckets.

Cost becomes a very big factor when managing, filtering and searching through TBs of data even for fairly small use cases.

One thing we lost in the process is full-text search over the request & response pairs and while we try to intelligently add metadata to requests to make searching easier, it isn't the complete experience yet. Still WIP as a problem statement to solve and maybe the last straw here. Any suggestions?

Openrouter packages APIs and most companies prefer having individual relationships with AI vendors. Choosing an AI gateway might be another way to go

Individual subreddits have usually been my go-to place to discuss and understand the nuances of academic papers. Love LocalLLama for discussions on generative AI papers.

_/\_

thanks! we spent very long building it out properly. Time to integrate weaviate generators now?

While I love and use Python everyday, I do believe that JS is a lot more popular and mainstream for web development today.

that's an awesome repo! We just announced our seed round today, cool coincidence :D

You beat us to posting Portkey to HN first! Thanks @gaocegege.

Having built LLM systems for the past 3 years, we painfully realised the need of DevOps for LLM systems, now emerging as the LLMOps/FMOps category. We're dead focused on the production side of things.

@HN - when building with LLMs, are you looking for LLMOps tools? How do you choose? What's most critical?

You folks should put up your connectors on the website, where all do you collect context from

Love this idea, but it becomes a race for integrations or some context isn't available. I personally like using Albus (https://www.springworks.in/albus/) quite a bit. You folks seem to have way more integrations!

Cool idea, like if Tinder & PH had a baby?

I would just start the slide, without even clicking on "Start Discovery Queue"

Beautiful website, but why don't you have a domain for this?

Pretty excited to launch this to the world! Have heard on multiple groups where people are trying to find various grants and credits programs. Hope this serves as a ready reckoner.

Please feel free to contribute.

Might be temporary, we've also randomly received 403s for short periods of time with no real explanations. Becomes ok with a retry.

Can you share evidence on shorter look-backs though?

While building LLM apps, we mostly cared about cost management as that completely sucked in the OpenAI dashboards.

We've today tried to create the perfect set of dashboards that can serve as a starting point for any AI builder to find control over the LLM requests. Very curious to get feedback from this community.

Accuracy & Hallucinations is one of the MAIN challenge for adopting LLMs in production. Evaluations as part of CI/CD and in real-time are very good counter-measures.

While amazing techniques & libraries exist for this, there's little literature on how to use them in production. Tried writing a detailed blog that decodes the OpenAI's evals framework and goes step-by-step through how to use it to your advantage.

You can learn how to use the eval framework to evaluate models & prompts to optimise LLM systems for the best outputs.

Langchain seems to be an active python repository as well

Have you tried Freshsales? It's loaded with automation but still has a DIY feel no akin to other enterprise crms

Disclosure: I work there and would love feedback.