2,239 karma · joined April 9, 2008
Already 85% of commuters to lower Manhattan take public transit. Of the remaining 15%. An analysis found that only 2% of working poor New Yorkers would pay the charge. Otherwise low income New Yorkers would overwhelmingly benefit from the better transit funding
https://www.nrdc.org/bio/eric-goldstein/busting-myths-new-yo...
I created Buckaroo to provide a better table viewing experience inside of notebooks. I also built a low code UI and auto cleaning to expedite the wrote data cleaning tasks that take up a large portion of data analysis. Autocleaning is heuristically powered - no LLMs, so it's fast and your data stays local. You can apply different autocleaning strategies and visually inspect the results. When you are happy with the cleaning, you can copy and paste the python code as a reusable function.
All of this is open source, and its extendable/customizable.
Here's a video walking through autocleaning and how to extend it https://youtu.be/A-GKVsqTLMI
Here's the repo: https://github.com/paddymul/buckaroo
The best measure of a transit project is "How many people use this per day". ie is it doing something valuable.
Note: I don't know of a solution for this other than more holistic government service planning. I do think it's valuable and good that those in need of government services can get there without a car. But it isn't always the sole fault of transit agencies that they have low ridership slow busses.
What are you using for processing (polars)?
Marketing note: I'm sure you're proud of P Core/V Core, but that doesn't matter to your users, it's an implementation detail. At a maximum I'd write "intelligent execution that scales from small files to large files".
As an implementation note, I would make it simple to operate on just the first 1000 (10k or 100k) rows so responses are super quick, then once the users are happy about the transform, make it a single click to operate on the entire file with a time estimate.
Another feature I'd like in this vein is execute on a small subset, then if you find an error with a larger subset, try to reduce the larger subset to a small quick to reproduce version. Especially for deduping.
hmm. fair enough. I have heard the short trip stat bandied about a lot. Having spent time with people in the suburbs, even close in suburbs, the stat makes sense... if you exclude commute to work. When I visit my parents in stroadville, a trip to the store is 2 miles each way and should be easily bikeable, but bike infra is non existent so everyone drives.
One note about the framing. The average US Driver excludes everyone who isn't a driver
This requires the most enforcement in front of NYC precincts. I truly don't know how such a corrupt organization could be reformed.
Even in sprawling suburbia, most trips a person takes are under 3 miles, eminently bikeable, but the bike infrastructure and built environment sucks for that. So people drive, from parking lot to parking lot.
The fact that the US is huge doesn't mean that the majority of miles driven are on long trips.
I built Buckaroo as a data table UI for Jupyter and Pandas/Polars, that first lets you look at the data in a modern performant table with histograms, formatting, and summary stats.
Yesterday I released autocleaning for Buckaroo. This looks at data and heuristically chooses cleaning methods with definite code. This is fast (less than 500ms). Multiple cleaning strategies can be cycled through and you can choose the best approach for your data. For the simple problems we shoudn't need to consult an LLM to do the obvious things.
All of this is open source and extensible.
[1] https://youtube.com/shorts/4Jz-Wgf3YDc
[2] https://github.com/paddymul/buckaroo
[3] https://marimo.io/p/@paddy-mullen/buckaroo-auto-cleaning Live WASM notebook that you can play with - no downloads or installs required
Thing is, they could have killed it by 1998, without ever releasing anything, that would have killed the other architectures it was trying to compete with. Instead they waited until 2020 to end support.
What the VLIW of Itanium needed and never really got was proper compiler support. Nvidia has this in spades with CUDA. It's easy to port to Nvidia where you do get serious speedups. AVX-512 never offered enough of a speedup from what I could tell, even though it was well supported by at least ICC (and numpy/scipy when properly compiled)
I have been working on Buckaroo, a table UI for dataframes that runs in jupyter notebooks. It's much more TraderUI, with sparkline size histograms, and decent baked in formatting for numeric columns.
I have heard that rail is heavily subsidized, and trucking is possibly taxed. But their operating procedures have to account for a lot of it.
My 2000 2500 has a hood approximately at my belly button. Most current 1/2 tons have a hood height at my nipple (I'm 5'8).
A Pillars have gotten significantly thicker too, making it harder to see pedestrians at corners.
Then there is the expanding cab and shrinking bed. The visibility in a standard cab truck is phenomenal, you are surrounded by glass. Higher beds are also harder to use, which doesn't matter for most truck buyers because they buy them as a statement, not for utility.
I was surprised when I specifically checked on Pierce that they are owned by Oshkosh and publicly traded. [2][3]
[1] https://www.nytimes.com/2025/02/17/us/fire-engines-shortage-...
The days are long but the years are short.
The team is responsive and they care about getting it right. Having a sane file format for serializing notebooks is an example of this. They are thinking about core problems. They are also building in the open.
The core jupyter team is very unresponsive and unfocused. When you have a bug, you need to figure out which one of many many interelated projects caused the bug, issues go weeks without a response. It's a mess.
Then there are the proprietary notebook like environments. VSCode notebooks, and google colab in particular. They frequently rely on opaque undocumented APIs and are also very unresponsive.
What shortfalls in Tabix/datagrip were you trying to fix?
What’s the prime use case were you trying to improve on. What did users do before, how is that better with agx?
What do you mean by data explorer? A Table browser? Exploratory data analysis like - ydata-profiling?
How does this connect to clickhouse? Does user data go through your server?
What table library are you using?
For larger datasets, real scripts are a better idea. I expect my stuff to work with datasets up to about 1Gb, caching is easy to layer on and would speed up work for larger datsets, but my code assumes the data fits in memory. It would be easier to add caching, the make sure I don't load an entire dataset into memory. (I don't serialize the entire dataframe to the browser though).
As an example, I frequently want to run analytics on a dataframe. More complex summary stats. So you write a couple of functions, and have two for loops, iterating over columns and functions. This works for a bit. It's easy to add functions to the list. Then a function throws an error, and you're trying to figure out where you are in two nested for loops.
Or, especially for pandas, you want to separate functions to depend on the same expensive pre-calc. You could pass the existing dict of computed measures so you can reuse that expensive calculation... Now you have to worry about the ordering of functions.
So you could put all of your measures into one big function, but that isn't reusable. So you write your big function over and over.
I built a small dag library that handles this, and lets you specify that your analysis requires keys and provides keys, then the DAG of functions is ordered for you.
How do other people approach these issues?
It seems like with a restart system you could do something like this, in a generic reusable library way, correct me if I'm wrong. On network failure, register a callback with some OS level network activity watcher, once the network has resumed working, continue execution as normal.