2,239 karma · joined April 9, 2008
A couple of questions:
What type of data person are you aiming this tool to be useful for, data analyst, data scientist, or data engineer. I'm guessing data analyst who wants to use PyData instead of Tableau?
How data aware is your system? Is it sending dtypes of columns, and maybe actual values to the LLM?
How do you deal with python code gen and hallucinations?
Do you plan to make a jupyter extension with this tech?
You're a troll, but I'll feed you. I adapted Peter Norvig's excellent lispy2.py [0] to read json. I call it JLisp [1].
Lispy2 is a scheme implementation, complete with macros that executes on top of python. I made it read json, really just replacing () with []. and defining symbols as {'symbol': 'symbol_name'}. I built it because it's easier to get a webapp to emit JSON then paren lisp. I also knew that building an interpreter on top of lisp meant that I wouldn't back myself into a corner. There is incredible power in the lisp, especially the ability to transform code.
[0] https://norvig.com/lispy2.html
[1] https://github.com/paddymul/buckaroo/blob/main/tests/unit/li... #tests for JLisp
Matt Harrison wrote Effective Pandas, the guide to writing elegant pandas code and avoiding many common pitfalls. Keep up the good work
https://www.amazon.com/Effective-Pandas-Patterns-Manipulatio...
https://buckaroo-data.readthedocs.io/en/latest/examples/inde... I need to clean up those examples.
Recently I have been working on the low-code UI. I find myself looking up and typing the same pandas (and polars) incantations over and over again to explore and modify data. The low-code UI allows simple transforms of data (search, remove column outliers, show only outliers of a column, group by) to be perfromed with just clicks. You can also view the generated python code. This is powered by a json-flavored lisp interpreter, but users never have to type lisp code.
Something like `if check_versions(pandas__gt="2.0.0", pandas__lt="3.0.0"):`
After cursory googling I couldn't find one, do you have a public notebook gallery?
What mix of JS packages do you see? Could that be built into one uber package?
This works hand in hand with autocleaning that looks at a dataframe and suggests jlisp cleaning commands.
[1] https://youtu.be/GPl6_9n31NE?si=JJPtk6K6FNX0sboi&t=603 [2] jlisp.py, based on Peter Norvig's lispy with a reader modified to accept JSON
Also, in the north east, tuned skis absolutely matter, whatever level you are. My wife has been learning to ski and had trouble with ice. So I put on her beginner skis, and said, ahh no wonder, went back and tuned them. Then I put her on my slalom skis because she didn't trust her skis to initiate a turn (slalom skis are famously quick to initiate), and she could turn them quickly. Then I tuned one edge of each slalom ski with a more aggressive tune, but she could swap back to regular if she wanted. Finally, she was skidding her turns too much, so I put her on my GS skis, which forced her to patiently start the turn then follow it through. She now has a good idea of when she's having trouble with the ski shape, the conditions or the ski tune after 35 total days of skiing.
If you put a beginner or intermediate on race skis, make sure the tune is mellow enough. Slalom skis at 0.5 and 86, will hunt when not turning, and can grab an edge throwing you to the ground. At the steeper angles (86/87 vs 88/89) it is much harder to skid the ski for speed control. When you head out with a beginner, you can always bring a diamond stone and detune the tip/tail, on the mountain even. They will notice the difference and appreciate it.
https://buckaroo-data.readthedocs.io/en/latest/articles/rela...
The other feature I have played with in this area is auto-cleaning. Auto-cleaning looks at individual columns and emits cleaning commands to the low-code UI. Different cleaning strategies can be implemented and toggled through.
Buckaroo takes the view that being opinionated is good, so long as you can toggle through opinions to get the right combination of cleaning, display, or post-processing that you are looking for quickly. All of the features of buckaroo are also built to be easily extendable by users.
This feature saw very little use, so I haven't developed it much (I had to disable it after some refactorings). The lowcode UI is demonstrated at the end of the youtube video linked above.
The next performance gain web tables comes from using a binary encoding instead of JSON, particularly arrow. Perspective uses arrow (in addition to rendering to canvas).
IME building buckaroo on top of ag-grid, I can render the table with up to about 300k elements very performantly with just JSON. Rendering speed is a non factor because only 50 rows are rendered at a time. Moving to arrow-js should be about 3 times faster for the entire system (python serialize, js deserialize, js render). Beyond 900k elements, you really want to lazily load from the server as the user scrolls. The memory usage for just the data in the browser tends to slow things down. (I am working on a library and benchmark for different serialization techniques).
Perspective seems to be the most performant html table. It is more focused on extremely fast updates than styling, although it looks good.
Glide is a newcomer that also renders to canvas.
I have built a different table library for jupyter called buckaroo. My approach has been different. Buckaroo aims to allow you to interactively cycle through different formats and post-processing functions to quickly glean important insights from a table while working interactively. I took the view that I type the same commands over and over to perform rudimentary exploratory data analysis, those commands and insights should be built into a table.
Great tables seems built so that you can manually format a table for presentation.