Hehe used to do R IMO you are right about ggplot but I strongly disagree about pandas. I fing love it. Would love to understand you troubles with it though, after using it for 4 years daily mabye I can offer some perspective ;)
Then when doing basic operations like "group by" you end up excessively elaborate indexes that are in my experience useless and always need to be manually squashed to something coherent.
It's a common joke for me that whenever even a seasoned Pandas user cries out "gaarrr! why isn't this working!?" I just reply "have you tried reset_index?"... this works in a frighteningly large number of cases.
Generally, I've even preferred Spark to pandas, though it's hardly less verbose. Coming from R, it's much slower than data.table and nowhere near as slick and discoverable as dplyr. Its system of indices is a pain that I'd rather not deal with at all (and, indeed, I can't think of another data frame library that relies on them). I hate finding CSVs that other data scientists have created from pandas, because they invariably include the index ...
Handles time series really well, though.
Recently I've been using polars (https://github.com/pola-rs/polars). As an API I much, much prefer it to pandas, and it's a lot faster. Comes at the cost of not using numpy under the hood, so you can't just toss a polars data frame into a sklearn model.
That being said: > I hate finding CSVs that other data scientists have created from pandas, because they invariably include the index ...
This is also default in R, with row numbers (like I have ever needed them). To be fair, it's gotten better since people stopped putting important information in rownames.
Polars looks interesting, thanks for the recommendation!
Ideally you should be using the parquet format which will use the binary format, preserve column types and indexes [df.to_parquet(<file>); df = pd.read_parquet(<file>)]
You can get away from a lot of problems by simply avoiding text files
More generally, the API is large, all-consuming and not consistent. sklearn is best in class here, I rarely need to look things up whereas the pandas docs autocomplete in my browser after one or two characters.