montanalow

Thanks for the feedback. I've updated that paragraph to clarify the product placement is for Llama. I don't really think Meta needs our help to much...

If you want to see all the SQL functions and tables Korvus depends on, check out the pgml extension.

https://postgresml.org/docs/open-source/pgml/

Yep, one of our other projects, pgcat is exactly to help make the horizontal scaling as easy as possible.

https://github.com/postgresml/pgcat

Most managers and organizations miss the fact that engineers are often motivated by solving puzzles in ways like this, because it's fun. If you want to accomplish big challenges, quickly, make it fun for the engineer. Working long hours on projects like this doesn't burn engineers out, it sharpens their skills, broadens their knowledge and grows the organizations capabilities long term.

This sort of cultural difference of exploration and letting work be fun is one of the big things that accounts for the differences in velocity between big co's and little co's. Does your work give you energy to the point where not only you love doing it, you want to tell everyone else about it?

You can do all of that in a single SQL query, with pgml.embed() and then pgml.train() a custom reranker with xgboost, to pgml.predict() the conversion score of a search result based on click-through-rate, or other objective.

If you'd like free hosting, feel free to reach out. I'm one of the founders at postgresml.org.

This is an SDK built to interact with PostgresML, which provides ML & AI _inside_ a Postgres database. Clients in this case don't perform inference, rather the server does. You could run the open source server locally, or connect to one running in the cloud.

If anyone would like a free PostgresML T-shirt, we just did our first run. Feel free to email me with your shipping info and size. It'd also be nice to get to know you a bit if your email address isn't obvious.

Quantization allows PostgresML to fit larger models in less RAM. These algorithms perform inference significantly faster on NVIDIA, Apple and Intel hardware. Half-precision floating point and quantized optimizations are now available for your favorite LLMs downloaded from Huggingface.

We've been working on a Python SDK[1] for PostgresML to make it easier for application developers to get the performance and scalability benefits of integrated memory for LLMs, by combining embedding generation, vector recall and LLM tasks from HuggingFace in a single database query.

This work builds on our previous efforts that give a 10x performance improvement from generating the LLM embedding[2] from input text along with tuning vector recall[3] in a single process to avoid excessive network transit.

We'd love your feedback on our roadmap[4] for this extension, if you have other use cases for an ML application database. So far, we've implemented our best practices for scalable vector storage to provide an example reference implementation for interacting with an ML application database based on Postgres.

[1]: https://github.com/postgresml/postgresml/tree/master/pgml-sd... [2]: https://postgresml.org/blog/generating-llm-embeddings-with-o... [3]: https://postgresml.org/blog/tuning-vector-recall-while-gener... [4]: https://github.com/postgresml/postgresml/tree/master/pgml-sd...

Full disclaimer, I work on PostgresML, and I'm not a MindsDB expert.

They both do ML in the database, but we've been at least as focused on scalability for Postgres workloads as ML functionality, with our PgCat project. It's a pooler that gives us load balancing, sharding, failover etc to handle large clusters of many machines for application and inference workloads to scale horizontally.

OTOH MindsDB interconnects with just about every data source out there, which you may consider an advantage.

There is a lot of latency involved shuffling data for modern/complex ML systems in production. In our experience these costs dominate end-to-end user experienced latency, rather than actual model or ANN algorithms, which unfortunately limits what is achievable for interactive applications.

We've extended Postgres w/ open source models from Huggingface, as well as vector search, and classical ML algos, so that everything can happen in the same process. It's significantly faster and cheaper, which leaves a large latency budget available to expand model and algorithm complexity. In addition open source models have already surpassed OpenAI's text-embedding-ada-002 in quality, not just speed. [1]

Here is a series of posts explaining how to accomplish the complexity involved in a typical ML powered application, as a single SQL query, that runs in a single process with memory shared between models and feature indexes, including learned embeddings and reranking models.

- Generating LLM embeddings with open source models in the database[2]

- Tuning vector recall [3]

- Personalize embedding results with application data [4]

This allows a single SQL query to accomplish what would normally be an entire application w/ several model services and databases

e.g. for a modern chatbot built across various services and databases

  -> application sends user input data to embedding service
      <- embedding model generates a vector to send back to application
  -> application sends vector to vector database
      <- vector database returns associated metadata found via ANN
  -> application sends metadata for reranking
      <- reranking model prunes less helpful context
  -> application sends finished prompt w/ context to generative model
      <- model produces final output
  -> application streams response to user

[1]: https://huggingface.co/spaces/mteb/leaderboard

[2]: https://postgresml.org/blog/generating-llm-embeddings-with-o...

[3]: https://postgresml.org/blog/tuning-vector-recall-while-gener...

[4]: https://postgresml.org/blog/personalize-embedding-vector-sea...

Github: https://github.com/postgresml/postgresml

RAM and the Postgres shared buffers, as well OS page cache can also be important factors when there is an active working subset, e.g. the active sessions on a website might be reused hundreds of times per session, with only relatively small portion of them active at any time.

Shared RAM across all cores can be a reason to use fewer larger machines rather than more smaller ones in such a case. Postgres does give you options either way though.

Another suggestion: Don't build you identity around a language or platform. They come and go. Except SQL. It's been around for longer than either of us.

Low effort comment that didn't read the post.

- Multiple formats were compared

- Duckdb is not a production ready service

- Pandas isn't used

You seem to be trolling.

How are you doing online ML inference, without fetching data?

Stateful services are indeed more painful to manage than non stateful ones. Ignoring state (data fetch time) for ML as if the model artifact is the only important component is... not a winning strategy.

You get it. 1 tier is better than 2 tier. Python can't be 1 tier, unless it loads the full dataset which is not generally feasible for production online inference cases. PostgresML is 1 tier, and supports the traditional Python use cases.

The converse is also true. We may be missing diagnosis for people born in September, who may struggle but not quite meet the definitions which often include academic performance.

What may be missing from cloud, is alignment of incentives. If you waste more compute, you increase their profit margins. That would explain things the author questions like general latency increases.

I think what you're missing is that XGBoost is worthless without data to use for inference. That data can come from in process, or over the wire. One is fast, one is not.

As a contributor, I think it's interesting when comments focus on the language (Python vs Rust) vs the architecture (local vs remote). Inference is embarrassingly parallelizable, with Python Flask or Postgres replicas. I think the interesting thing is that data retrieval costs tend to dominate other costs, and yet are often ignored.

ML algorithms get a lot focus and hype. Data retrieval, not as much.

Those well defined slices of pie you’re referring to are often exactly the political boundaries OP is referring to.

This is the degenerate form of Conways Law.

We weren’t expecting 35x speedup. The difference was pretty surprising.

We're launching the PostgresML Gym today, a free/hosted version of Postgres with our machine learning extension built in. It enables you to train supervised ML algorithms (regression, classification) on real time data in your database, and make predictions without any additional infrastructure.

We'd really appreciate it if you could kick the tires let us know how we can make ML more approachable.

Go to huggingface.com and start with some of the tutorials. The operational/engineering skill sets alone are all you need to treat modern ML models like any other black box API/SDK.

Yep, they will use the specialized hardware by default if CUDA/cuBLAS has been installed on the system.

My point is that the ships telemetry was not an issue. Blaming electric vehicles with no basis, or lack of telemetry with clear evidence to the contrary is…something.

The entire crew was rescued. The article is nonsense.

This article is pure speculation by an insurance company that isn’t involved, and admit they have no idea what happened. FUD.

Landed a PR to enable GridSearchCV and RandomSearchCV. https://github.com/postgresml/postgresml/pull/83