paddy_m

I'm certain the host heavy lifting is done by numpy which is a python wrapper around Fortran and C. The visualization heavy lifting is done by pygfx/wgpu-py. wgpu-py has C. I think wgpu-py compiles to WASM to run in the browser. More and more packages are taking this route.

[1] https://github.com/pygfx/pygfx [2] https://github.com/pygfx/wgpu-py

I'd be curious to hear more about your EDA workflow.

What I want for EDA is a tool that let's me quickly toggle between common views of the dataset. I run through the same analysis over and over again, I don't want to type the same commands repeatedly. I have my own heuristics for which views I want, and I want a platform that lets me write functions that express those heuristics. I want to build the inteligence into the tool instead of having to remember a bunch of commands to type on each dataframe.

For manipulating the plot, I want a low-code UI that lets me point and click the operations I want to use to transform the dataframe. The lowcode UI should also emit python code to do the same operations (so you aren't tied to a low-code system, you just use it as a faster way to generate code then typing).

I have built the start of this for my open source datatable UX called Buckaroo. But it's for tables, not for plotting. The approach could be adapted to plotting. Happy to collaborate.

Really nice post introducing your library.

When would you reach for a different library instead of fastplotlib?

How does this deal with really large datasets? Are you doing any type of downsampling?

How does this work with pandas? I didn't see it as a requirement in setup.py

Does this work in Jupyter notebooks? What about marimo?

I have read about the CL condition system. I get how restarts are useful for interactive programming.

I don't see how they could be used in most cases where you want a program to run without a programmer intervening. Could you list some more usecases.

Location: Boston

Remote: Yes

Willing to relocate: Yes

Technologies: talking to users, python, pandas/numpy, jupyter, js/ts

Résumé/CV: https://www.linkedin.com/in/paddymullen/

Email: paddy@paddymullen.com

In my next role, I want a broad mandate to make a meaningful impact within an organization by developing products that address genuine business challenges, with a preference for data related problems.

Recently I built the data table for Jupyter/Pandas/Polars that I have wanted for over a decade. The open source Buckaroo (https://github.com/paddymul/buckaroo) data table combines a performant table, summary statistics, and a low code UI to expedite common data analysis tasks.

This sounds interesting. Is this meant to run in pipelines or be used interactively?

Are you building something open source? Link to the repo?

I am building Buckaroo [1], the data table UI in Jupyter that I have always wanted. I know how to use df.head(), df.describe(), sort, and run histograms on columns. I just got tired of typing 5 bits of code to minimally inspect each dataframe every time I looked at it.

So I built buckaroo, it combines a high performance scrollable table (built on top of ag-grid), with summary stats, and histograms. All of this is customizable and extensible. I recently built a dataframe compare tool [2] on top of buckaroo that uses coloring to show differences between dataframes intuitively.

Get in touch if you want to talk tables, data science tooling, or exploratory data analysis.

[1] https://github.com/paddymul/buckaroo

[2] https://www.youtube.com/shorts/u3PW6q36ufo

Did you write your own table?

Million+ row handling is nice. Once I figured out window based display of tables, documentation and a bunch of other pieces of architecture became simpler. Previously I sent the whole dataset or a sample of it.

--

Is Scripton user extendable? Do you expect ipywidgets/anywidgets based projects to plug in?

What are your plans for a data table UI?

Shameless plug: I created the open source Buckaroo table for jupyter (embeddable in other contexts) with histograms, summary stats, search... I love talking tables if you want to get in touch.

I like 80 columns, I can tolerate 100 or 120. I get really annoyed with formatting standards, JS/TS in particular that waste a whole line for a closing brace. Standard aspect has screens more limited vertically than horizontally.

When dealing with tabular data, particularly test data, I find most formatting lacking. I want to be able to specify blocks that align on the decimal point. Especially when dealing with lists of dicts. This makes reading test fixtures much more intuitive than default indentation styles.

Has anyone seen a formatter where you can specify a block be formatted in that manner?

  [{'a':  3.89, 'b':  10},
   {'a': 12.3,  'b': 233}]

instead of

  [{'a': 3.89,
    'b': 10},
   {'a': 12.3,
    'b': 233}]

I have know idea if the project author would agree with those sentences, I was just proposing them.

I made a similar comment. I think they come up because people are genuinely interested in a project and trying to offer the creator a fresh perspective. When creating a project where your the domain expert it's so easy to get stuck in your own head, and then you start explaining the project to a newcomer diving into deep details when they don't understand the starting point.

Until a project has a lot of traction (think docker, react, django not uv or jq) it's very safe to assume that every visitor to your page doesn't understand the background.

I was going to ask the exact same question. The title makes it sound like something I might be interested in, then I visited the page and I have no idea what it does.

After some brief reading of docs, I'm trying to write one sentence explanations. Maybe this will be helpful to you

What

Preserves is a specification and set of libraries in popular languages that lets you reliably exchange data between XML, JSON and EDN.

Who

Preserves is built for (data engineers|data framework writers) to reliably interchange data.

Why

Formats like JSON in particular are imprecise. Preserves forces you to deal with these vagaries up front

What else?

With P-Expressions you can search a preserve compliant datasource much like you would query JSON with JQ

Who Not? Who shouldn't use this

This will not help a data analyst exchange data between CSV and Excel

README suggestions:

Put the animated gif at the top

Add subtitles to the gif explaining what you're doing.

Thank you. I can always xargs grep for a function name, at worst. Dir() in python at a debugger for other things. With YAML, kubernetes and other devops hotness, I frequently can’t even find the relevant scripts/YAML that are executed nor their codebases.

This also happens with configuration based packaging setups. Python hatch in particular, but sometimes node/webpack/rollup/vite.

Which makes it much harder to break the circuit vs AC

I think pandas was so ambitious syntax wise and concept wise. But it got be a bit of a jumble. The index idea in particular is so cool, particular multi-indexes, watching people who really understand it do multi index operations is very cool.

IMO Polars sets a different goal of what's the most pandas like thing that we can build that is fast (and leaves open the possibility for more optimization), and clean.

Polars feels like you are obviously manipulating an advanced query engine. Pandas feels like manipulating this squishy datastructure that should be super useful and friendly, but sometimes it does something dumb and slow

Have you tried polars? It’s a much more regular syntax. The regular syntax fits well with the lazy execution. It’s very composable for programmatically building queries. And then it’s super fast

From a brief look, I think he's trying to position it for "Exploratory DOM Analysis". The intro demo looks scheme/smalltalk like in that you are creating the structure and primitives as you go (as demonstrated by changing 'ivory' to 'red' and watching the syntax highlighting change in realtime).

I understand how we got here, but it's a shame that javascript frameworks and libraries aren't easier to just play with in the browser. It's just JS, you should be able to play with it quickly in a lightweight environment. This approach excels at that. This approach brings back the whimisical possibilities of HTML/JS. I'd love to see more stuff like that and less TS, rollup, webpack,...

edit:Actually after reading a bit, this is being proposed for data analysis. I think that's a poor fit for this approach

They have a claim for $1.5B, they are going after all Alex Jones assets which are much less than $1.5B.

University surplus stores are always fun to browse. Lab equipment, machine tools, great stuff.

Nice work!

Do you have any plans for data cleaning?

I am working on a somewhat similar open source project. I intend to add heuristic data cleaning. With the UI I want to be able to toggle between different strategies quickly - strip characters from a column to treat it as numeric, if less than 2% or 5% of values have a character, fill na with mean, interpret dates in different formats - drop if the date doesn't parse. The idea bing that if it's really quick to change between different strategies, you can create more opinionated strategies to get to the right answer faster.

Happy to collaborate and talk tables with anyone who's interested.

I would pay money, $1+ postage to maybe $5 + postage to physically print out tweet streams and other articles and mail them to some old people I know.

I'm thinking specifically of DieWorkwear on twitter, but others too.

Could you give some examples?

Curious to hear about other people's experience with Tanstack. I built a complex app [1] around AG-Grid and here are my thoughts:

I would recommend AG-Grid for most data apps until a well supported alternative comes out. My understanding is that Tanstack Table is aimed at generic apps that happen to have a table. Perspective's [2] table component is most likely more capable than AG-Grids, but it needs significantly more documentation and examples.

AG-Grid has a particular way of building things that seems to map to well to pulling data from a database or basic webservice. AG-Grid works very well for this use case. The AG-Grid enterprise features are built around this type of data model too.

If your data is small enough to fit into memory (less than 50 MB), it works very well. AG-Grid doesn't have a connector to arrow or parquet (I will eventually write my own). For larger datasets JSON generation and parsing performance bottlenecks. Additionally Parquet/Arrow offers much better types.

Configuring AG-Grid is tricky, especially when you want config to be declarative. If you want to define your own custom rendering components, getting them to work properly per column when you want different renderers for pinned rows is tricky (my solution [3])

Support for the community version of AG-Grid is practically non-existent. I understand that they have an enterprise business model that charges for support. They don't seem to be interested in answering community bug reports or comments, even for generically relevant bugs.

I have a decent amount of trust in AG-Grid's community offering [4], I don't see them cannibalizing community features to put them in their enterprise offering. Their enterprise features are the main development focus of the team, particularly the charts. The charts offering isn't impressive compared to bokeh, plotly, or vega. It would work for line of business apps, but not for a userbase familiar with better analytic platforms.

All that said. AG-Grid has excellent documentation and examples. They examples are swappable between Javascript, Typescript, React, and Angular.

[1] https://github.com/paddymul/buckaroo

[2] https://github.com/finos/perspective

[3] live example https://buckaroo-data.readthedocs.io/en/latest/examples/#/DF...

[4] https://blog.ag-grid.com/javascript-jabber-podcast/

I just saw this. yep, totally promoted my project instead of following this guideline.

If anyone here has an open source project that they want a docs or README review from fresh eyes, please reach out. Establishing what problem your software solves, for who, with no initial context is a fun challenge. I don't pretend to be an expert, but I enjoy looking at new stuff.

To your point about LLM based approaches have the huge adoption advantage in that you don't need to understand a lot to write into a text box.

A tool like buckaroo requires investment into knowing where to click and how to understand the output intitially.

I would love if we could seed LLMs with specific books, or give our own weightings to sources. I'm sure most books are in there already, I would happily pay extra (and want it to go to original authors) for known provenance of advice. Even pass royalties back to the original authors. For PyData code, I'm always looking at Effective Python/Polars.

I have a tool [1] that is tackling some of the same problems in a different way.

I had some core views that shaped what I built.

1. When doing data manipulation, especially initial exploration and cleaning, we type the same things over and over. Being proficient with pandas involves a lot of recognition of patterns, and hopefully remembering one with well written code (like you would read in Effective Pandas).

2. pandas/polars is a huge surface space in terms of API calls, but rarely are all of those calls relevant. There are distinct operations you would want on a datetime column, a string column or an int column. The traditional IDE paraidgm is a bit lacking for this type of use (python typing doesn't seem to utilize the dtype of a column, so you see 400 methods for every column).

3.It is less important for a tool to have the right answer out of the box, vs letting you cycle through different views and transforms quickly.

------

I built a low code UI for Buckaroo that has a DSL (JSON Lisp) that mostly specifies transform, column name, and other arguments. These operations are then applied to a dataframe, and separately the python code is generated from templates for each command.

I also have a facility for auto-cleaning that heuristically inspects columns and outputs the same operations. So if a column has 95% numbers and 1% blank strings, that should probably be treated as a numeric column. These operations are then visible in the lowcode UI. Multiple cleaning methods can be tried out (with different thresholds).

[1] https://github.com/paddymul/buckaroo

[2] https://youtu.be/GPl6_9n31NE?si=YNZkpDBvov1lUYe4&t=603 Demonstrating the low code UI and autocleaning in about 3 minutes

[3] There are other related tools in this space, specifically visidata and dtale. They take different approaches which are worth learning from.

ps: I love this product space and I'm eager to talk to anyone building products in this area.