edublancas

TIL there is pgvector and pgvecto.rs

Cool work!

I've been working on a similar product. Users can select between Streamlit/Shiny: https://editor.ploomber.io/ - so not necessarily for BI (although you can use it for that), but more broadly focused on data apps.

has papermill deprecated the ipython runtime? I used papermill extensively in the past and I never saw that in their docs.

Papermill is great but has quite some limitations because it spins up a new process to run the notebook:

- You cannot extract live variables (needed for testing)

- Cannot use pdb for debugging

- Cannot profile memory usage

You can do all of that with ploomber-engine (https://github.com/ploomber/ploomber-engine).

Disclaimer: I'm the author of this package

I'd say how much is good enough highly depends on your use case. For something that still has to be reviewed by a human, I think even .7 is great; if you're planning to automate processes end-to-end, I'd aim for higher than .95

thanks a lot for the feedback! you're right, this is much better input data. I'll re-run the code with these tables!

there isn't. but you can connect X or LinkedIn.

I might add a subscribe button once I get some time :)

nothing, I strip out all the HTML tags and pass raw text

author here: I'm working on a follow-up post where I benchmark pre-processing techniques (to reduce the token count). Turns out, removing all HTML works well (much cheaper and doesn't impact accuracy). So far, I've only tried gpt-4o and the mini version, but trying other models would be interesting!

author here: I'm working on a follow-up post. Turns out, removing all HTML tags works great and reduces the cost by a huge margin.

You can get this for pretty much any language by re-using Jupyter kernels; here's a Python example: https://hidden-truth-8699.ploomberapp.io/

If you're looking to try this in a Jupyter notebook: https://jupysql.ploomber.io/en/latest/integrations/chdb.html

We'd love to support this in Ploomber Cloud!

We already support frameworks like Streamlit, Gradio, Shiny, Solara, Voila, among others. Ping me if you're interested!

https://docs.cloud.ploomber.io/en/latest/intro.html

If you're looking for a place to deploy Quarto projects for free, check out Ploomber Cloud.

We added support recently and have some nice features like password protection: https://docs.cloud.ploomber.io/en/latest/apps/quarto.html

Nice! This looks a lot simpler than traditional Shiny code.

I've just deployed an app to Ploomber Cloud: https://dry-fire-8233.ploomberapp.io/

Docs: https://docs.cloud.ploomber.io/en/latest/apps/shiny-express....

The chDB team is great! We worked with them recently to add chDB support to JupyterLab via a %%sql magic: https://jupysql.ploomber.io/en/latest/integrations/chdb.html

Solara is a fantastic project! The user experience is excellent, and I can quickly get ideas up and running. Solara can potentially become the framework for shipping ML/AI apps.

We (Ploomber) have been working with the team to help their community easily deploy applications in our cloud platform: https://docs.cloud.ploomber.io/en/latest/apps/solara.html

chDB is a great project!

We recently worked with the chDB team to add support for Jupyter notebooks: https://jupysql.ploomber.io/en/latest/integrations/chdb.html

Ploomber (W22) | NYC (hybrid) | Full-time

Job openings: https://www.workatastartup.com/companies/ploomber

Ploomber is hiring engineers to build a platform to deploy AI applications.

We're well-funded (raised a seed round last year) and have thousands of weekly active users in our open-source projects (https://github.com/ploomber/)

For those who don't know, Jupyter has a bash kernel: https://github.com/takluyver/bash_kernel

And you can run Jupyter notebooks from the CLI with Ploomber: https://github.com/ploomber/ploomber

Congrats on the 1.0 milestone!

A few months ago, we worked with the team to bring Hydra to Jupyter, you can check out the tutorial here: https://docs.hydra.so/analyze/jupyter

JupySQL's GitHub: https://github.com/ploomber/jupysql

Yes! We have some basic autocomplete features: https://jupysql.ploomber.io/en/latest/howto/autocompletion.h...

If anyone is looking for something like this in Python/Jupyter, check out JupySQL: https://github.com/ploomber/jupysql

How are people managing the existence of data frame APIs like pandas/polars with SQL engines like BigQuery, Snowflake, and DuckDB?

Most of my notebooks are a mix of SQL and Python: SQL for most processing, dump the results as a pandas dataframe (via https://github.com/ploomber/jupysql) and then use Python for operations that are difficult to express with SQL (or that I don't know how to do it), so I end up with 80% SQL, 20% Python.

Unsure if this is the best workflow but it's the most efficient one I've come up with.

Disclaimer: my team develops JupySQL.

How are you running your SQL queries?

If you use notebooks: my team is working on JupySQL, a tool to improve the SQL experience in Jupyter. https://github.com/ploomber/jupysql

If you use Jupyter, check out JupySQL. It allows you to break down long queries in multiple cells so you can test them individually: https://jupysql.ploomber.io/en/latest/compose.html (Disclosure: my team develops JupySQL)

Reminds me of JupySQL, which allows to compose large SQL queries: https://jupysql.readthedocs.io/en/latest/compose.html

Disclaimer: My company develops JupySQL

Yeah, we're also looking into it and we'll probably add it as a backend in the future!

Is querying the underlying SQL database officially supported in MLflow? Last time I used it, it wasn't documented. I took a look at the database and it wasn't end-user friendly.

Agree. Once you have a team, you need to have a service they can all interact with. This release is a first step, we want to get the user experience right for an individual and then think of how to expand that to teams. Ultimately, the two things we're the most excited about are 1) you don't need to add any extra code (and it works with all libraries, not a pre-defined set) 2) SQL as the query language