HNHacker News
TopNewBestAskShowJobs

eskibars

379 karma · joined December 28, 2015

https://zeroquarry.com
submissionscomments
eskibars··on Is seat-based pricing dead?
Seat-based pricing isn't dead, but it's on life support, though it has nothing to do with AI now or realistically in the future.

Consumption based pricing is becoming/already the norm for 2 reasons. As a SaaS operator, my storage/compute/etc costs tend to go up with your usage. If I can bill by the metrics that affect my cost, then:

1. It feels more fair to the buyer (pay for what you use) and

2. Investors and finance folks like it because I can build a pricing model with consistent gross margins

This whole thing is hinged on the multitenant SaaS model generally though, and it being so prominent. For example, if I ship self-managed software, I no longer genefally incur hardware usage costs (except my support costs probably go up roughly linearly to usage).

AI doesn't affect any of this reasoning.

eskibars··on Better RAG Results with Reciprocal Rank Fusion and Hybrid Search
Vectara
eskibars··on Better RAG Results with Reciprocal Rank Fusion and Hybrid Search
Having worked building out a RAG SaaS platform for the past year and having worked on the vendor side of several keyword-based search systems in the past 10 years, I can say it's absolutely necessary to have some kind of hybrid search for most use cases I've seen.

The problem is that most people don't have experience optimizing even 1 of the retrieval systems (vector or keyword), so a lot of users that try to DIY build end up with an awful time trying to get to prod. People are talking about things like RRF (which are needed) but then missing other big-picture things like the mistakes everyone makes when building out a keyword search (not getting the right language rules in place) and also not getting the right vector side (finding the right embedding models, chunking strategies, etc).

I recognize I have a bit of a conflict of interest since I'm at a RAG vendor, but I'll abstain from the name/self-promotion and say: I've seen so many cases where people get this wrong, if you're thinking RAG you really should be hiring a consultant or looking at a complete platform from people that have done it more. Or be prepared to spend a lot of cycles learning and iterating

eskibars··on DuckDuckGo was down
You don't need to look hard to find this happen. OpenAI has had several API outages this year
eskibars··on Ask HN: Founders who offer free/OS and paid SaaS, how do you manage your code?
I worked in product at Elastic working on Elasticsearch, which ended up trying several different things (some of which I know the broader OSS-driven community didn't like, so I'm prepared for downvotes here).

We had started off by using a series of private repos for our non-OSS features. At build time, we had a pretty complicated suite of build tooling that took all of the repos and integrated them and ran the integration tests. The biggest reason why Elasticsearch went to single repo with all of the enterprise software included (knowing that the OSS-focused community was going to backlash on it) was just how much of a hassle that was. It bifurcated everything and the results were that paying customers were upset the most.

For example: have an issue with the product? You could file a GitHub issue in public and it would get looked at quickly, but if you were a paying customer with an issue with one of the enterprise features, you had to file an issue with the support team because the GitHub repo was separate/private, and then support would file an issue in the private GitHub repo, and you'd lose complete visibility with it. It was even worse for free/trial users: they had to file an issue on our forums (since they didn't have an established support contact), after which they lost visibility. Integration tests were also hellish.

I later worked at another open core company, which at first maintained a separate codebase for its paid offering, and synchronized changes over from the open source product periodically, and things were much worse: a ton of bugs kept popping up and getting the paid offering out the door was always extremely slow because you were always playing catch up on the OSS offering.

You can go down the submodule route, but you end up relying on pretty complicated build tooling. If that's in your team's realm of capabilities, it's not a bad route to go. If you have no need for "OSS," then you can choose a non-OSS license that provides whatever restrictions you care about and bundle it up together, which is easier, but might dissuade certain users.

Regardless, licensing enforcement isn't really an issue if you're working on a B2B product: there are certain demographics that won't pay for your software, but they won't pay if you have licensing enforcement either. B2C can be a different story.

eskibars··on The Stainless SDK Generator
I've worked on 4 different (pretty heavily used) developer-focused products in my career in product management, and we hand-built SDKs for each of them. This experience is so common: in all cases, a lot of the leadership went into it thinking it would be an easy-to-maintain "background task" to maintain 4-6 SDKS because we generally hired pretty well seasoned and hard working developers

Inevitably in all cases language-specific problems exactly like this happened, but also just some frustrating drift. One of those heroic workhorses would go out on parental leave and maybe a backup would go on vacation or out for extended sick time and the SDKs would start drifting away from the API. Someone would be stuck maintaining code in their absence. At one point, I (a product manager at the time) got tasked with maintaining a Perl SDK while a developer was out on leave for several months because my background is in software development and I was the only person that could really maintain it in the company.

I'm happy to see tools like this come about because it seems everyone underestimates how much work SDKs take to really build functional SDKs. You can get bad ones cheaply by just putting a few people on and accepting bugs and drift. The counter-argument against these tools that I've generally heard is that they'll also produce worse ones cheaply or require effectively the same maintenance to figure out a machine-generated problem as a human-generated problem.

eskibars··on I bought 300 emoji domain names from Kazakhstan and built an email service
I bought xn--mn8h9e.ws a couple years ago just for fun. I think it's fun to own an emoji domain, but what I can say definitely say is that it's still a bad idea to own one if you want to get emails to it.

Popular thick email clients still struggle with utf8 domains and I've fiddled around with several email providers that just have complete failures trying to send as well.

I tried pasting the email address in a bunch of popular services (LinkedIn, Instagram, etc) as my email or the domain as my homepage and most of them treat it as invalid, and I found some legitimate breaking bugs in their services in trying it out.

Edit: case in point, just noticed HN also falls in the camp of unsupported emojis in the text body, so another example :). Added the punycode instead

eskibars··on [dead]
I really wanted to understand where this community is at in their generative AI journey. Are you in production? Just looking around? Have you tried and ran into serious problems? Is it living up to the hype but too expensive?

I built a little poll at https://forms.gle/WezLRpGUkiF6Za4w6 and have it set up to auto-share the responses with any respondents so we can all learn together.

I'd love any input!

eskibars··on Insecure Features in PDFs (2021)
Are there any sample documents for these types of attacks? I'd love to test my own service.
eskibars··on How to Reject a Feature Request
On the PR front, a lot of the issues can be fixed before a PR is ever submitted: make it clear in the PR process that there should be an issue open before a PR. That gives you an opportunity to discuss with the community before coffee gets written and guide a committer on what the expectations are (and are not).

Some users will fork your code to implement something they see as necessary, so if you're going to reject an issue, the first thing you need to do is understand why the user is requesting it, and what their options are without an implementation

eskibars··on Show HN: Chat-focused RAG with automated memory management
Hi HN!

I lead product at Vectara. Yesterday, we launched Vectara Chat, which is a set of APIs to help organizations combine chatbots with RAG (Retrieval Augmented Generation). One of the most common things we've heard from the community of users trying to build chatbots is that they hallucinate (and hence the release of our hallucination evaluation model: https://huggingface.co/vectara/hallucination_evaluation_mode...). One of the things we've realized when working with customers on RAG systems (especially multi-turn) is that a lot of overemphasis of effort into "which model," "which vector DB," and so on, but then miss incredibly important things like "how do you make sure things like keywords -- non semantic-based bits of context -- influence results" and "how do I prompt engineer" and "how do I rewrite queries appropriately to include history?"

This is our attempt to tackle that: you basically just give the API a conversation ID and a turn ID and Vectara will store the history and rewrite the queries using modern LLM techniques to provide the end user with the answer to their question.

We've just launched this feature: we started off as a LLM-based semantic search company, and then launched a RAG solution in May 2023, and we think this multi-turn Chat capability is the next stage of our offering. We'd love to hear feedback from the community!

A longer form blog post is at https://vectara.com/blog/vectara-chat-revolutionizing-chat-d...

eskibars··on Are we at peak vector database?
That's exactly what Vectara is (full disclosure, I work there)
eskibars··on Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?
Besides the other comments in this thread, I'd really recommending looking at least first to the (relatively new) "Managed index" in LlamaIndex: https://docs.llamaindex.ai/en/stable/community/integrations/... . These handle combining the retrieval with the generative side. I've seen a lot of users both get frustrated and get bad results by trying to write their own glue to string together various components of retrieval and generation and these are much easier to get started with
eskibars··on What Is Retrieval-Augmented Generation a.k.a. RAG?
Having worked on a (production, as-a-service RAG system), what we've learned is that fine tuning is not good for really adding/removing facts but is great for changing the form of the output. Anyscale has a great blog on this: https://www.anyscale.com/blog/fine-tuning-is-for-form-not-fa... (I work at Vectara, not Anyscale, but appreciate the details and thought they put into this blog)

Fine-tuning can help with certain areas of knowledge acquisition, but it's costly and frankly doesn't work that well when the knowledge you're trying to provide goes "against" the data that was trained into the core system. e.g. try to fine tune a model that there is a cure to <disease X> if you're a research org and it's incredibly hard to do that because the base model may be so convinced otherwise.

Probably the most difficult thing though is using fine-tuning to get a model to "forget" or "not respond to" something that it shouldn't take a stance on. I always ask a fine-tuned model: "how do you calculate the fourth interior angle of a triangle." This is obviously a nonsense question, but helps to show how most LLMs will happily tell you to sum the interior angles of your first 3 angles and then subtract from 180. A well-tuned RAG system will say "sorry, I don't have information for that." It shows how it's less "guess-y" and hallucinated

eskibars··on Show HN: Trains.fyi – a live map of passenger trains in the US and Canada
It'd be great. My understanding is that there's basically no live data for freight trains unfortunately
eskibars··on Show HN: Trains.fyi – a live map of passenger trains in the US and Canada
I love it! I made a mashup of video feeds and live train locations to show you video feeds from trains that are about to show up: https://train.api.connelly.casa/

This gives me lots of ideas of additional agencies to include! Maybe we should join forces :)

eskibars··on Amtrak Explorer
Personal pet project for my son who loves seeing trains: I'm trying to get a system set up to auto-rotate through live webcams when the trains are most likely to show up through their live location status. https://train.api.connelly.casa/
eskibars··on Show HN: Boomerang, a new embedding model for RAG and semantic search
So far, we haven't really focused on code ingestion. We've had a few users try it out for that use case, but we code ingestion and generation is a bit different. We've found a lot of users have success in the natural language areas (ingesting enterprise content, ecommerce content, etc) and then building chatbots on top of the all-in-one API
eskibars··on Show HN: Live Train Webcams
I'd love to hear them!
eskibars··on Bob and Juliet? Rag vs. Finetuning an LLM
I've seen a lot of this, and other problems with fine tuning, that result in real business outcomes. For example:

  - I've seen research orgs have a *lot* of difficult to fine tuning out something that might be taken as "fact" based on the original training data.  As science advances, but the training may be based on (and reinforced through) several years of suddenly outdated info due to new research, it's really difficult to "fine tune that old knowledge out".  Meanwhile, RAG from what I've seen seems to use the language model for *language/structure* but the retrieval for the facts/details when done correctly.
  - I've seen a fair number of copyright issues with fine tuning, because it's really difficult to get a LLM to not return something that looks like the original source except to change its form.  Meanwhile, RAG focuses the output on whatever data you put in
It's of course possible to trick a RAG system into producing either of these issues -- especially if the prompt doesn't attempt to account for them -- but it seems to me to be much harder
eskibars··on How do domain-specific chatbots work? A retrieval augmented generation overview
IMHO I think the best points you make here are the last 2. To contrast the first 2, I have seen that many users can provide really good capabilities with RAG -- especially combined with some kind of keyword/hybrid blending so that hyper-specific terms still come up.

Of course, it's difficult for most organizations to actually test their results in a meaningful and cost-effective way, which is why we see so much of this "deploy, cross fingers, wait for a call from the CEO" behavior.

eskibars··on No one wants to talk to your chatbot
Also, many times I've had to wait 5 minutes for an agent to respond at all (presumably because they're way oversubscribed) and then had the "chat with an agent" thing time out and disconnect me entirely after 10 minutes is so frustrating.. Yeah, I went and grabbed a water/coffee/went for a bio break because your agent hasn't responded to my last message for 7 minutes. But then you disconnect me after 5 minutes of "inactivity" and ask me to hop on a new chat with the next agent that will not have any history from the previous chat? I could do with a lot less of that in my life.
eskibars··on No one wants to talk to your chatbot
I think there are multiple facets to this argument (both for and against). Yeah, a lot of chatbots are so "stupid" or at least so obviously non-human that as a user, I have absolutely no desire to interact with them. They waste my time and I end up doing the same thing as I sometimes need to do with automated phone systems: press the virtual equivalent of "0" to try to get connected with a real human.

But that is starting to change: some chatbots can now start understanding and interacting like humans. As a user, when that's the case, I don't personally care what is powering the thing behind the scenes. In fact, I'd generally prefer a bot if it's as good as a good human: the number of times I've had 45 minute or longer sessions with some human support agent that: 1) Just didn't listen to what I was looking for 2) Had difficulty communicating because I started a chat on an evening/weekend and got routed to someone who had English as a second language 3) Couldn't actually figure out how to solve some problem, so I had to start a new conversation of the same substance the next day 4) Didn't actually log the notes of my chat for the next agent, so I had to repeat myself etc

is just completely off the charts and it's anecdotally gotten worse in my experience in the past few years.

eskibars··on Avoiding hallucinations in LLM-powered applications
I wasn't, and I'm not sure how you got that out of what I said. I'm not claiming "understanding," "sentience," etc.

I'm claiming there's a great deal of work in the realm of research that I'm excited by: research I expect to be done by humans. I do expect/am supposing that result may be LLMs of a different nature: to apply guard rails around generative LLM systems, but that's not to anthropomorphize them: just to suppose their purpose.

The term "hallucination" does anthropomorphize LLMs, but I think that's now accepted nomenclature in the industry, at least for the time being, and it's helpful to have some standard nomenclature to describe some of the benefits and problems.

eskibars··on Avoiding hallucinations in LLM-powered applications
I know the article doesn't go into this particular element, but I do wonder how much opportunity is still in front of us for adversarial LLM systems that try to detect/control for hallucinations. I'm pretty excited by the research in LLM explainability and quantitative measures on how accurate generative LLMs are measured

(Full disclosure: I work at Vectara, where this blog was published)

eskibars··on Ask HN: Who is hiring? (May 2023)
Vectara is hiring a number of positions: https://jobs.lever.co/vectara

Vectara is a SaaS platform built from the ground up to support LLM-focused retrieval (think LLMs for search applications) as well as using that to surface higher-level business use cases like building chatbots, web/app/product search, and research-focused applications that are "grounded" in truth based on that retrieval. The team is fairly seasoned and comes from backgrounds like likes of Google, Elastic, and Cloudera. The product is real, and we already have paying customers, though the company is still very early. If you want to have a fiddle around with the product to see what you'd be getting yourself into (or to give us feedback so we can give a better product!), we have a pretty generous free tier at vectara.com -- no CC required.

Open positions:

In engineering: software engineer (remote OK), ML engineer (Bay area preferred), and DevOps (remote OK)

Outside of engineering: first tech writer, director/head of PMM

Compensation ranges for each position are listed in the job posting

I lead product and have the open tech writer + product marketing positions, but can connect you with the engineering team / our recruiter there if these positions seem interesting. You can reach out to me at shane[at]vectara[dot]com

eskibars··on Building a semantic search engine in Rust
> I'm sure people have thought about these things: what technical challenges exist in improving search in these areas? is it just a matter of integrating engines like the one that's linked here? Or maybe keyword searches are often Good Enough, so no one is really clamoring for something better

Several technical challenges.

First, keyword searches have a lot of history that has led to a huge amount of tuning that users have gotten used to (mostly for the better in terms of results but mostly for the worse in terms of difficulty of configuration). For example, keyword systems have evolved over decades to have synonyms (unidirectional and bidirectional), a huge number of stemming algorithms for various languages (and some that cross languages), dictionaries for decompounding, various ngram/shingling methods, phrase matching and term overlap analysis, and the ability to combine all of these together with tunable weights, etc. These have generally resulted in a lot of keyword systems continuing to be "as good as it gets" for a long time. People generally like fiddling with these knobs/dials because it gives them a sense of control...until they realize the combinatory mess they get themselves into where they're essentially human hyperparameter tuning systems. Recently, some additional steps have come to take the "human" out of that with automated systems, but even then, most systems aren't set up to "learn" what synonyms to potentially introduce, whether/when/how to take word order into place, and in particular when/how these can/should combine together and when they shouldn't.

Semantic large language models "solve" some of these problems (automatic synonyms, built in linguistic understanding of root words, etc) if you build them right, but they have a lot of hidden technical depth. Most people try to throw something like BERT into their search and find the hardware costs and complexity go through the roof in ways they weren't ready to handle. And there's history weighing on the expectations for the operators ("where's my synonym configuration," etc) and the answers are very different ("go through a fine tuning step for your model") or sometimes nonexistent on most commercial platforms (how do you ensure only relevant results are returned)? And because the semantic/large language models don't know everything in the world, OOTB models still do underperform relative to keyword on certain query types (those heavy on obscure people names, etc) -- until they're retrained.

There's good research and companies/products coming out though that are changing a lot of this. See https://docs.google.com/spreadsheets/d/1L8aACyPaXrL8iEelJLGq... for example where the BM25 rows are traditional keyword and rows 8+ are zero-shot language models, and you can start to see that in some of the recent developments, semantic/neural/large language models are starting to outperform keyword on the things keyword used to be better at. My sense (though I'm biased) is these solutions are going to rapidly evolve to eliminate many of these technical challenges.

Disclosure/source: I led product management for Elasticsearch for several years and am currently leading product management for Vectara (a neural search SaaS platform)

eskibars··on Will keyword (BM25, TD-IDF) be replaced for search by Neural Search?
I agree there's a place for both right now. Curious if you think that's going to be the case forever? I've been seeing huge, nearly step-function increases in precision gains for neural systems recently, and I'm wondering how much longer that's going to be true. Keyword has taken the title for this so far (but doesn't need anything as complex as BM25/TF-IDF IMO in a lot of cases)
eskibars··on Show HN: Where SF public transit breaks rules
I've been collecting SF MUNI's GPS info (including speed, current position) for the past 2 months and comparing it to what should be happening (speed limit, route). This is a heatmap of where MUNI speeds most often.

Full source available at https://github.com/eskibars/Oddball

← PreviousPage 3 of 3