benjaminwootton

That is a fair point and I can see how the amount of prompt engineering would reduce. However, even a human has to be told how to solve a task, what tools are available to them, and how to stay within guardrails. Prompt engineering is how that is done.

The article tries to make the point that they sound like nonsense buzzwords, but I believe there is some actual depth there.

I would be interested whether or not you agree with that central point?

I’m not sure it requires studying, but it isn’t easy and it definetly requires practice.

You need to write incredibly detailed descriptions as to how the agents should behave, how they should reason and how they should use the tools available to them.

The language is quite esoteric, and every model behaves differently.

It’s funny how a lot of the practices at small scrappy software companies persist even at Temenos $X00m scale.

Sales guys selling stuff that doesn’t yet exist, doing funky deals to hit the quarters numbers, selling far too much bespoke work for a product, and very unpredictable and shoddy implementation projects once the license is sold.

Ive seen it all at small software companies, but shouldn’t a core banking system vendor have a slightly more rigorous approach.

This is an old sub-thread now, but I would agree that dbt is a very simple idea and easy to use at its core. Store your models in text files, in git, add a dag. It’s easy to use and very lightweight so I don’t see any reason to roll your own.

Then you get things like testing, documentation, modularity if you want it.

I’ve always had an inkling that at least half of pay per click traffic is bots.

I must have tried PPC with 20 different businesses that I’ve ran or helped with, and I don’t think I’ve ever really moved the needle with it.

The vast majority of traffic just bounces even with every best practice in the book followed.

I feel like it’s a game that everyone has to play but theres a little emperors new clothes about it where nobody will admit it’s a waste of money.

dbt was a huge leap forward in this regard.

It enables source code control, modularity, testing, CI/CD, environments, documentation, git branching, local development experience.

Not a fanboy, but it's who reason for being is basically to enable "software engineering practices for data".

I don't think it's either.

Modern Data Stack is a set of characteristics - cloud based, SaaS, consumption based pricing, ELT bias, open source bias, component based etc.

AirByte is all of them. It's just a fact rather than a criticism.

Have you tried the PostgreSQL table engine?

Or Debezium to Kafka for a more hand coded solution?

GitHub actions must have eaten most of the DevOps tooling market. It’s pretty good and it’s fine for the vast majority of pipelines.

I built this demo of using LLMs to query databases, knowledge bases, and most interestingly create PDFs. It’s targeted at financial services but similar could be achieved in many industries.

Very pleased with how it turned out as it really brings the potential of LLMs to life IMO.

https://www.youtube.com/watch?v=r8MyAxyPJsA

Isn’t it difficult for one person to market multiple products or websites? It’s hard enough getting traction for one product, let alone a portfolio?

I agree that sqlite has a number of similar benefits - openness, table abstractions and concurrent transactions. It’s also a library so close to how delta, iceberg and hudi are implemented.

I’m glad it’s had an uptick in interest recently but I haven’t yet seen it mentioned for analytics yet.

I assume it’s row rather than column oriented?

Databricks popularised the concept and explain it very well - https://youtu.be/g11y-kJHr3I?si=j8FAkFsIjScHv24f

It’s a technology independent pattern though.

Date lake can be thought of a file system. Imagine 100 CSVs in folders, usually stored on S3.

Data Lakehouse involves adding things like the ability to query via SQL, the ability to update/insert/delete, transactions.

Where before people needed warehouses for BI and lakes for data science, they can now have only one approach.

It’s likely to be a big trend as data moves to this format and arrangement and the DBMS vendors like Snowflake have to play nicely with it.

They are closely related.

A lakehouse would be a collection of tables.

There would be some SQL engine such as Trino, Databricks, ClickHouse brokering access to these tables.

The lakehouse might be organised into layers of tables where we have raw, intermediate, processed tables.

The concept of a lakehouse doesn’t really work if you don’t have transactions and updates, which these table formats enable.

I always say, Lakehouse is a stupid name but an amazing concept and I’m sure the data industry will go this way.

Thanks for the upvotes.

Here is a more technical deep dive comparing the three main open formats - https://www.onehouse.ai/blog/apache-hudi-vs-delta-lake-vs-ap...

It’s true but it’s still depressing if the market has a bad week and you know that you are down by 6 figures. It bounces back in the end but it’s a source of background stress.

Do you have any podcasts you would reccomend with this type of content?

I built a demo of something similar, using LlamaIndex to query data as it streamed into ClickHouse.

I think this has a lot of real world potential, particularly when you move between the query and a GenAI task:

https://youtu.be/F3Eup8yQiQQ?si=pa_JrUbBNyvPXlV0

https://youtu.be/7G-VwZ_fC5M?si=TxDQgi-w5f41xRJL

I generally found this worked quite well. It was good at identifying which fields to query and how to build where clauses and aggregations. It could pull off simple joins but started to break down much past there.

I agree with the peer comment that being able to process and respond to error logs would make it more robust.

I'm invested in VUSA ETF through Vanguard UK, but point taken.

Different funds will have different mandates, but generally they are trying to beat their benchmark index and charge 2 & 20 for the privilege.

If they can't beat the S&P 500 which I can invest in with Vanguard at 0.07% at relatively low risk, then why would you invest with them?

It’s funny how in a lot of these corporate missteps it’s the response which tells you more about the culture than the original incident. This is more likely to turn me off using them than the original incident.

Apologising and saying you will put processes into place to prevent it happening again is corporate comms and emotional 101, and it would have blown over in a week.

I quite like how France defend their culture. Some of these rules sound silly, but if they slow down the world homogenising then they aren’t without benefit.

It depends on lifestyle and expenses, which is probably a function of how you lived before retirement. There is no right answer.

The name is easy to poke fun at, but I think it’s a real problem. A lot of companies use data lakes to store data and warehouses to serve BI to tools like Tableau or PowerBI. They then up copying data between the two.

Querying a lake directly and having transactions, governance etc against one set of data (a data Lakehouse) can really simplify the stack and take out cost.

It tends to be more library dependencies than live clusters.

A lot of data lakes are managed using Hadoop and Spark so I think it’s just an artefact of that.

In the end I can’t see why you wouldn’t just be able to create and manage Iceberg files directly from a standard Python/JS/Java without that legacy.

This is a big deal in the database world as delta, iceberg and hudi mean that data is being stored in an open source format, often on S3.

It means that the storage and much of the processing is being standrdised so that you can move between databases easily and almost all tools will eventually be able to work with the same set of files in a transactionally sound way.

For instance, Snowflake could be writing to a file, a data scientist could be querying the data live from a Jupyter notebook, and ClickHouse could be serving user facing analytics against the same data with consistency guarantees.

If the business then decide to switch Snowflake to Databricks then it isn’t such a big deal.

Right now it isn’t quite as fast to query these formats on S3 as a native ingestion would be, but every database vendor will be forced by the market to optimise for performance such that they tend towards the performance of natively ingested data.

It’s a great win for openness and open source and for businesses to have their data in open and portable formats.

Lakehouse has the same implications. Lots of companies have data lakes and data warehouses and end up copying data between the two. To query the same set of data and have just one system to manage is equally impactful.

It’s a very interesting time to be in the data engineering world.

I suspect most of the cost in the OpenSearch vector database?

I agree it is high and it is not exactly serverless. You pay by the hour.

Bedrock itself is charged based on tokens exchanged.

I think Pinecone is a cheaper database for hobby and small business projects, though I haven't looked into it.

I used the RAG knowledge bases for most of my testing described above.

I got a toy demo up and running with continuous pre-training but haven’t evaluated it unfortunately.