mtricot

Great to see you here!

Let me see what's up and fix that!

Just want to call out a couple of nuances in our methodology. In general, we tried our best to do apples-to-apples comparisons where we could, and gave ourselves a discount where we couldn’t. Unsurprisingly, it’s a challenge to find MCPs for various vendors (which is another reason we are trying to solve this). Here’s a video walkthrough of the benchmark harness:https://www.loom.com/share/9d96c8c64c1a4b7fad0356774fc54acc

Where the comparison wasn't valid or not apples-to-apples:

Gong and Zendesk: no official native MCP exists, so we used the most popular community implementations we could find. We were only able to benchmark Gong Search as the Gong MCP does not have a Get tool call.

While our Search testing yielded the same number of records on either path, vendor-specific search implementations means results aren’t identical. Contents are similar in general, so the ratios remain directionally correct.

The general test set:

2 scenarios (Retrieval and Search) across 4 connectors isn’t a huge test set. While we hope to extend this over time, we’ve made the harness public so anyone can contribute in the meantime. Let us know if you find any MCP with better results!

Where the vendor MCP wins or ties:

Salesforce showed the smallest win at 16%. This is primarily because Salesforce, unlike many vendors, uniquely provides great search support out of the box with their SOQL.

We see identical records for Get. As noted, Search returns different sets of identical counts. Airbyte uses fewer tokens because the Salesforce records contain mandatory metadata (type and url).

Where the vendor MCP is costly to context:

Zendesk is a great example of this. The extreme gap is because the Zendesk MCP (reminder - a community alternative) returns the entire API response in search results. This averages to 9KB per record against our production Zendesk account!

Airbyte’s implementation provides filtering, which allows agents to retrieve the minimal data needed to achieve the outcome, explaining the drastic gap.

Talking about going back memory lane :) The initial name of the project was "conduit"...

correct

Not at the moment but let me bring that to the team so we can brainstorm what it could look like.

When reading the tutorial, we are describing one stack to build a specific app. But the stack is made of building blocks that you can replace with others if you need to.

- Airbyte has two self-hosted options: OSS & Enterprise

- Langchain: OSS

- OpenAI: you can host an OSS model if you want to

- Pinecone: there are OSS/self-hosted alternatives

No good reason. Does "it made the post's title too long" work?

Isn't it the dream? Today there is a lot of stack that needs to be built to enable what you're describing. This is actually what we are doing with that post. What foundations do we need to build so that the UX for the end user is what you're describing. Will take some time to get there :)

It depends.

Airbyte comes in 3 flavors: OSS, Cloud, Enterprise.

For OSS & Enterprise, data doesn't leave your infra since Airbyte is running in your infrastructure. For Cloud, you would have to allow some IPs to allow us to access your local db.

For the purpose of the tutorial that we built, it really comes down to the type of data that you're using.

If you have data with PII:

One option would be to use Airbyte and bring the data into files/local db rather than directly to the vector store, add an extra step that strips the data from all PII and then configure Airbyte to move the clean file/record to the vector store.

The option that jmorgan mention is relevant here, using a "self-hosted" model.

On the roadmap! We want to get more clarity on how to fit the Embedding part in the ELT model. Once we figure it out we will add it to PG.

Thanks! I agree with your point. There is a lot of tuning that needs to happen, including context aware splitting and any other kind of transformation before the unstructured data gets indexed. This is one of the big challenge of productionizing LLM apps with external data. So far we are using internally since the team as experience dealing with building these connectors and that becomes a great co-pilot.

The great thing we get by plugging this whole stack together is that we get all the refreshed data as more issues/connectors get created.

I am sure we can build something around that. Going to take a look at it. Thanks for mentioning it.

Shouldn't have any limits here. Can you let us know how it goes?

Great question :) We want to get to value as fast as possible. I am certain that at some point we will need to go deeper with those integrations and they will likely require to be separate destinations. It will also depend on how they differentiate from each others, we will need more granularity with configurations.

Let me know how that works out for you and if you would add anything to this tutorial!

Super thrilled to see two open source projects come together!

We do support separate workspaces but we don't have auth yet. However because there is an API available, you can always build the account segmentation and permissionning at your app level and only interact with Airbyte using the API.

Thanks! let us know if we can help.

Let us know if we can help on slack (https://slack.airbyte.io) #connector-development or we can schedule a coding call call with you

In the end, data is transported as electrical current so it is not completely off :)

Data engineers, data scientist & analytics engineers. People that be users of Airbyte but also can be contributors.

For data engineers, the objectives is to help them empower other roles to be data fluent and have the freedom to move data without bugging data engineers

YC does play an important role to get connections to VCs but at the end of the day, "YC" is a signal for VCs, not a criteria.

VCs spend time looking at the team, the past achievements, the product and most importantly the existing users. They also try to invest in industries that they know about.

In our case the team experience was important. We had solved the problem internally at other companies (and the scars that come with it!).

In one of John's response, he mentioned that we've been talking to many VCs. The reason was that we were looking to talk to the ones who understand deeply the problem and the market we're addressing. No matter how good your product or deck is, if you're pitching a calendar app to a VC who is specialized in deep tech, you probably won't get them on your cap table.

we keep on improving the experience (priority #1 :) ). Would love to get your feedback there!

Here are some articles: https://docs.airbyte.io/tutorials/building-a-python-source https://docs.airbyte.io/tutorials/toy-connector https://docs.airbyte.io/integrations/custom-connectors

That's exactly why we are approaching the problem with open source. It changes the dynamic of how it gets adopted. we've been in your shoes where a tool is being pushed Top-Down and now you have to deal with a super complex, super expensive, rigid & half working product.

Instead Airbyte gets adopted by engineers, data scientist... to solve one problem and then the usage expands from there. We can improve the product based on the feedback we get from the real users.

And if a feature, a connector is not there, anyone can actually add it!

Just fixed it! Thanks!

Don't hesitate if you have questions :) It was 13 days but it was INTENSE!

Perfectly timed comment :) we are currently writing a tutorial at the moment to showcase how to use Airbyte/DBT/Metabase together

Thanks! will look at it!