Polars
pola.rs
pola.rs
And the rest follows the same line.
Anyone could ELI5 what this is and for what needs it is a good solution to use?
EDIT: So an alternative implementation of Pandas DataFrame. Google gave me [0] which explains:
> The pandas DataFrame is a structure that contains two-dimensional data and its corresponding labels. DataFrames are widely used in data science, machine learning, scientific computing, and many other data-intensive fields.
> DataFrames are similar to SQL tables or the spreadsheets that you work with in Excel or Calc. In many cases, DataFrames are faster, easier to use, and more powerful than tables or spreadsheets because they’re an integral part of the Python and NumPy ecosystems.
/pun
I'm so sorry.
...ehh, not quite. R and its predecessor S have Pandas beat by decades. Pandas wasn't even the first data frame library for Python. But it sure is popular now.
Out of curiosity, what was/were the previous libraries?
It seemed that gmfawcett was indicating that there was a dataframe library in _Python_ that existed prior to Pandas. I was curious what that library was/is, as I'd not heard that before.
[1] https://datapythonista.me/blog/pandas-20-and-the-arrow-revol...
Spark tended to do this and it makes complete sense for distributed setups, but apparently is still faster locally.
Only for SQL databases, so not really. Source: have been running dplyr since 2011.
https://arrow.apache.org/cookbook/r/manipulating-data---tabl...
Python is literally designed to be easy to program with in general.
Well, it turns out when you’re dealing with terabytes of data and TFLOPS, the programming becomes more important than the math. Not all R devs are happy about this and they are very loud about it.
But it shouldn’t really surprise anyone. That is literally how those languages are designed.
Most of the R devs I know like this are just butthurt they are paid less and refuse to switch because they’re obstinate, or they’re a little scared they’re being left behind. first group is all over the place, but the second group tends to skew older of course
Look, I started with R and use mostly Python these days, but this is not really a fair take.
R is (still) much, much, much better for analytics and graphing (the only decent plotting library in python is a ggplot clone). The big change (and why Python ended up winning) is that integrating R with other tools (like web stuff, for example) is harder than just using Python.
pandas (for instance) is like an unholy clone of the worst features from both R and Python. Polars is pretty rocking, though (mostly because it clones from Spark/dplyr/linc).
It's another example of Python being the second best language for everything winning out in the marketplace.
That being said, if I was starting a data focused company and needed to pick a language, I'd almost certainly build all the DS focused stuff in R as it would be many many times quicker, as long as I didn't need to hire too many people.
R is fine. The issue is more in the ecosystem (with the aforementioned exception of the tidyverse).
One could also argue that dataframes are basically in-memory database tables. And in that case, S and SQL probably tie in terms of the creation timeline.
I think this is overblowing the similarities to matrices. Matrices have elements all of the same type, while data.frames mix numbers, characters, factors, etc. You certainly cannot transpose a data.frame and still have a data.frame that makes sense. Multiplying rows would not make sense either, since within one row you will have different types of data. Unless you have a data.frame that is all numeric, but in that case one should probably be using a matrix in the first place.
They still have their advantages with row/column labels, NaN handling etc. These are not operations I am speculating about by the way. I am most familiar with pandas and the dataframe there has transpose, dot product operations and almost all column operations have their correspondence in rows (i.e. you either sum(axis=0) or sum(axis=1)).
Statisticians are funny!
Mainly because R sucks for anything that isn’t statistics.
So so true.
I was working on an adhoc project that needed a quick result by the end of the day. I had to pull this series of parquet files and do some quick and dirty analysis. My first reflex was to use python with pandas, quick and easy. Python could not handle the datasets, too large. I decided to give R and data.table a go and it went smoothly. I am usually a python user but from time to time I feel compelled to jump back to R and data.table. Phenomenal tool.
Python is great for getting things into a data.frame. But once in a data.frame, R is so much easier to work with for all things stats and ml-related.
Polars can be an overkill for small/medium dataset, but since I have been bitten by corrupted/badly formatted CSVs/TSVs I love the fact that Polars will throw the towel & complain about types/column number mismatches etc. And the fact that it can scale up to millions of rows on a modest workstation compensates the fact that sometimes one can spend hours finding a proper way to manipulate a dataset.
> As a hobby project I tried to build a DataFrame library in Rust. I got excited about the Apache Arrow project and wondered if this would succeed.
> After two months of development it is faster than pandas for groupby's and left and inner joins. I still got some ideas for the join algorithms. Eventually I'd also want to add a query planner for lazy evaluation.
You can ignore my comment. It doesn't make sense.
> "Polars is written from the ground up with performance in mind"
It is a common thing to see, I thought it would be funny to imagine the opposite.
I get that these are general purpose tools with a lot of use cases, but some real quick examples of "this is a good use case" and "this is a bad use case, maybe prefer SQL/nosql/quasisql/hadoop/a CSV file and sed" would be really helpful, please.
On the other hand, how can you become a target user if you don't know that a product category exists?
If you have the problem "I want to analyze a bunch of tabular data", you'll start researching and asking around about it, and you'll quickly discover a few things: 1. people do this with (usually columnar / "OLAP") sql query interfaces, 2. people usually end up augmenting that with some in memory analyses in a general purpose programming environment, 3. people often choose R or python for this, 4. both of those languages lean heavily on a concept they both call "data frames", 5. in python, this is most commonly done using the pandas library, which is pervasive in the python data science / engineering world.
Once you've gotten to that point, you'll be primed for new solutions to the new problems you now have, one of which is that pandas is old and pretty creaky and does some things in awkward and suboptimal ways that can be greatly improved upon with new iterations of the concept, like polars.
But if you don't have these problems, then the solution won't make much sense.
Programming language documentation doesn't all start with "programming languages are used to direct computers to do things"; it is assumed the target audience knows that. Database documentation similarly doesn't start out with discussing what it means to store and access data and why you'd want to do that.
It's always hard to know where to draw this line, and the early iterations of a new idea really do need to put more time into describing what they are from first principles.
I remember this from the early days of "NoSQL" databases. They spilled lots of ink on what they even were trying to do and why.
But in my view this isn't one of those times. I think "DataFrames" are well within a "lingua franca" that is reasonable to expect the audience of this kind of tool to understand. This is not an early iteration of a concept that is not widely familiar, it is an iteration of an old, mature, and foundational concept with essentially universal penetration in the field where it is relevant.
Having said all that, I came across this "what is mysql" documentation[0] which does explain what a relational database is for. It's not the main entry point to the docs, but yeah, sure, it's useful to put that somewhere!
0: https://dev.mysql.com/doc/refman/8.0/en/what-is-mysql.html
I am the target audience for Polars and have been meaning to try it for several months, but I keep procrastinating about because I feel residual loyalty to Pandas because Wes McKinney (its creator) took the time to write a helpful book about the most common analytical tools: https://wesmckinney.com/book/
Polars is also a wonderful project!
Polars is a DataFrame interface on top of an OLAP Query Engine implemented in Rust using Apache Arrow Columnar Format as the memory model.
most data engines and dataframe tools these days use Apache Arrow, it's a bit orthogonal
Newbies are your best target audience too! They aren't already ingrained in a system and have to learn a new framework. They are starting from yours. If a newbie can't get through your docs, you need to improve your docs. But it's strange to me how mature Polars is and that the docs are still this bad. It makes it feel like it isn't in active/continued development. Polars is certainly a great piece of software, but that doesn't mean much if you can't get people to use it. And the better your docs, the quicker you turn noobs into wizards. The quicker you do that, the quicker you offload support onto your newfound wizards.
Interesting, I've personally found them quite good and compared to datafusion or duckdb they're dramatically better. I agree pandas has better docs, but one of the strengths of polars is that I find I often don't need the docs due to putting lots of careful thought into designing a minimal and elegant API, not to mention they're actually care about subtle quirks like making autocomplete, type hinting, etc. work well.
But I do think making good docs is quite hard. You usually have multiple audiences that you might not even be aware of. Which makes one of the most important things to do is keep an open ear to listen for them. It's easy to get trapped thinking you got your audience but you're actually closing the door to many more groups (unintentionally). It's also just easy to be focused on the "real" work and not think about docs.
Hell, I can do pretty complex integrals and derivatives and now so much of that seems trivial to me now but I did struggle when learning it. Don't shame people for not already knowing things when they are explicitly trying to learn things. Shame the people that think they know and refuse to learn. There's no reason to not be nice.
Having done a lot of teaching I have a note, don't expect noobs to be able to articulate their problems well. They're noobs. They have the capacity to complain but it takes expertise to have clarify that complaint, turning it into a critique. I get that this is frustrating, but being nice turns noobs into experts and often friends too.
Nobody is suggesting that you should be an expert on data frames "through divine intervention". But the place to expect to learn about those things is the many articles, tutorials, courses, and books on the subject, not the website of one specific new tool in the space.
If you're really interested in learning about this, a fairly canonical place to start would be "Python for Data Analysis"[0] by Wes McKinney, the creator of pandas and one of the creators of the arrow in-memory columnar data format that most of these projects build atop now.
This is a (multiple-) book length topic, not a project landing page length topic.
Sure. I mean devs can do whatever they want. But the package is a few years old now and they do frequently advertise, so I don't think it makes make it more approachable for... you know... noobs.
This is a bit difficult of a conversation too, because you've moved the goal post. I've always held the context of noob, but now you've shifted to just be dismissive of noobs. Totally fine, but different from the last comment.
> But the place to expect to learn about those things is the many articles, tutorials, courses, and books on the subject, not the website of one specific new tool in the space.
I actually disagree. This is the outsourcing I expressed previously, but it's clear from the number of complaints that this is not sufficient for a novice. You do seem passionate about this issue, and so maybe you have the opportunity to fill that gap. But I very much think that official documentation is supposed to be the best place. Frankly because it is written by the people who have a full understanding of the system and how it all integrates together. I'm sure you've run into tons of Medium tutorials that get the job done but are also utter garbage and misinform users. It isn't surprising when most of these are written by those in the process of learning, and are better than nothing, but they are entirely insufficient. The whole point of creating a library is to save people time. That time includes onboarding. For example of good docs, I highly recommend the vim docs. Even man pages are often surprisingly good.
No, I'm sorry, this is getting ridiculous. I'm not being dismissive of noobs, I'm saying "noobs should seek introductory material when attempting to learn an entirely new subject, like books, courses, or tutorials on the subject matter".
It's just so freaking weird for you to expect every single tool in some space to create that introductory material.
I promise you that the ruby on rails website did not assume total ignorance of the term "web application" when I first came across it as a "noob". I was a total noob at ruby on rails, but I had to understand why I might be interested in "web applications, but easier".
I could spend all day coming up with examples that are just like this. And this is not some kind of failure of imagination in how to document specific projects, it's just specialization. The website of a new tool for something that has been done a bunch of times over multiple decades is not the right place to put the canonical text on what the thing you're doing is; you put that in a book or in college courses or other kinds of training materials.
Unless what you have made is a brand new entirely unfamiliar thing (which is very rare) with no introductory materials for your brand new novel concept available anywhere, it makes more sense to focus your documentation on "why choose this specific solution over the other ones people are already familiar with" rather than "what even is the thing that we're doing here from first principles". Sure, add some links to the best introductory materials, but don't try to write them yourself, that's crazy!
> I actually disagree. This is the outsourcing I expressed previously, but it's clear from the number of complaints that this is not sufficient for a novice. You do seem passionate about this issue, and so maybe you have the opportunity to fill that gap.
No, I'm not passionate about this issue. I think people who actually want to learn things will continue doing research and reading books and taking classes to learn about new subjects, and that people who just want to complain will continue to do so. There is no "gap" to fill. There are tons of great materials that will describe in great depth what "data frames" are, and how to work with them, for anyone who is even the tiniest bit interested.
> I very much think that official documentation is supposed to be the best place. Frankly because it is written by the people who have a full understanding of the system and how it all integrates together.
I think what you seem to be confused by is the difference between this one library - polars - and an entire large subject - tabular data analysis using data frames. It certainly does make sense for the polars website to document the polars library, which (in my view) it already does. But if you want to learn the subject, you need to do that in the normal way that people have always learned new subjects. I'm sorry, because you seem resistant to this, but again, the way to do that is with books and courses, not by reading the documentation of one tool comprising a tiny sliver of a very large subject.
> I'm sure you've run into tons of Medium tutorials that get the job done but are also utter garbage and misinform users.
No, Medium tutorials should not be your go-to source for learning about a new subject! Your go-to source should be books and courses.
This is why I keep commenting here. I want to get through to you that you seem to be going about the acquisition of knowledge in a very weird and fundamentally misguided way. It just isn't the case that knowledge is mostly found in the documentation of tools! There is way more foundational knowledge to learn than it would ever make sense for every little tool to document themselves.
This is, in a very literal sense, why people write books about things, and why schools exist. We don't teach algebra by linking to the Mathematica documentation.
A little bit context would have helped to have attracted a lot more noobs.g
I'd also recommend this book: https://wesmckinney.com/book/. It's not about polars, but you'd be able to transfer its ideas to polars easily once you read it.
Or is diff aggregation its own thing? (I tried searching for the term, but didn't find much.)
For instance you've got a time series with an odometer value and you want the a delta from the previous sample to compute the every trip.
Shouldn't the good communication happen when the boss tells you to learn polars? Like, why are you telling me this, boss; what is it that you need done?
I for example hate website that try to serve newbies, newbies have a lot of content if they are interested, it's not that all the web needs to serve them
Thijs Nieuwdorp and I are currently working on the O'Reilly book "Python Polars: The Definitive Guide" [1]. It'll be a while before it gets released, but the Early Release version on the O'Reilly platform gets updated regularly. We also post draft chapters on the Polars Discord server [2].
The Discord server is also a great place to ask questions and interact with the Polars team.
[1] More information about the book: https://jeroenjanssens.com/pp/
[2] Polars Discord server: https://discord.gg/fngBqDry
source: we use this feature in beta as part of the enterprise copilot license to summarize Teams calls. Yes, it listens to us talking and spits out bullet points from our discussions at the end of the call. It's so good it feels like magic sometimes.
note on copilot: any capable model could probably do it. I just said copilot because it does it today.
There’s a reason people prefer it: it’s better.
That's a very "black and white" view, of any issue.
It could be that it's better for that specific person(s) who is having the current conversation. But what about after that?
It used to be that you could search Google and if it was answered in a forum post, you'd find it via Google. But since a lot of it are closed behind locked doors (like Discord), it becomes really hard to find, even with Discords own search.
Everything being ephemeral could help someone right now, but what about the others in the future, wanting the exact same answer?
I built a paper trading app for stocks and options and Discord was the primary place where the users talked. The subreddit was almost completely empty, nobody responded to tweets, the Instagram was thriving but there was no sense of community because you couldn't tell if anyone commenting actually used the product or a meme just showed up in their feed.
Did I have to repeat myself a bunch? Yes, but that's fine, especially because the answer sometimes changed (rapidly), like "No we don't support multiple trading accounts per user" > "Yeah I implemented that last week, it's in the release notes, to add a second account..."
For a mature product that's not changing as much and more or less has all its features built out, it makes sense to branch out into more structured forums that are easily searchable, especially as you progress through different versions and users are looking for answers to past versions.
What I'm hearing is that you think Discord is better than a forum, because people talked in Discord but they didn't talk in the forum, a forum which you didn't have?
Do you have comparable experience building a community via a traditional forum VS doing so with Discord? As far as I can tell, it doesn't seem like you have tried a normal forum, yet you say Discord is better for community building.
[0] https://docs.pola.rs/user-guide/
[1] https://github.com/pola-rs/polars/tree/main/docs/user-guide
This whole thread just comes across as unmitigated pedantry to me.
Compare that to Pandas which starts with the basics, "pandas is a fast, powerful, flexible and easy to use open source data analysis and manipulation tool, built on top of the Python programming language." It then leads you to "Getting started" guide which features "Intro to pandas" that explains the core concepts.
Sadly its not only tech products, but also things like security disclosures too.
It always follows the same pattern:
- Spend $X time coding/researching something.
- Spend $not_enough_time documenting it.
- Spend $far_too_much_time thinking about / "engaging with the community" in deciding on a cute name, fancy logo and cool looking website.It’s a shame because I like to read about new tech or project and try and learn more, even if I don’t understand it completely. But there’s just nothing here for me.
This must be what normal people go through when I talk about my lowly web development work…
You can much more easily compose the operations you want to run.
Just think of it as an API for manipulating tabular data stored somewhere (often parquet files, though they can query many different data sources).
Polars actually has SQL supported, so you can mix and match: e.g. construct DF from SQL.
It's in the documentation.
I just learned pandas recently, and I would have said this same thing. Not because I read through the numpy code, but because I read the documentation.
Is it wrong? Can't a user pick up a new tool and trust some documentation without reading through 100's of libraries built on libraries.
When was last time someone traced out every dependency, so they can confidently say something is "Myth".
If your data fits into numpy arrays or structured arrays (mainly if it is in numeric types), numpy is designed for this and will likely be much faster than pandas/polars (though I've also heard pandas can be faster on very large tables).
Pandas and Polars are designed for ease of use on heterogeneous data. They also include a python 'Object' data type which numpy very much does not. They are also designed more like database (e.g. 'join' operations). This allows you to work directly with imported data that numpy won't accept - after which Pandas uses numpy for underlying operations.
So I think the point is if you are running into speed issues in Pandas/Polars, you may find that the time-critical operations could be things that are more efficiently done in numpy (and this would be a much bigger gain than moving from Pandas to Polars)
I haven't played enough to know all it's benefits, but yes it's the next logical step if you are in the space using the above mentioned libraries, it's something one will find.
Marketing is also very concerned with understanding who your target audience(s) are and speaking their language.
I think talking about "DataFrames" is exactly that; the target audience of this project knows what that means. What they are interested in is "ok but who cares about data frames? I've been using pandas for like fifteen years", so what you want to tell them is why this is an improvement, how it would help them. Dumbing it down to spend a bunch of space describing what data frames are would just be a distraction. You'd probably lose the target audience before you ever got to the actual benefits of the project.
The PMs literally simplified things so much they simplified the product right ought of the docs.
> Quick install > Polars is written from the ground up making it easy to install. Select your programming language and get started!
There's enough subtle and breaking changes that it is a bit frustrating. I really think Polars would be much more popular if the learning curve wasn't so high. It wouldn't be so high if there were just good docs. I'm also confused why there's a split between "User Guide" and "Docs".
To all devs:
Your docs are incredibly important! They are not an afterthought. And dear god, don't treat them as an afterthought and then tell people opening issues to RTFM. It's totally okay to point people in the right direction without hostility. It even takes less energy! It's okay to have a bad day and apologize later too, you'll even get more respect! Your docs are just as important as your code, even if you don't agree with me, they are to everyone but you. Besides technical debt there is also design debt. If you're getting the same questions over and over, you probably have poor design or you've miscommunicated somewhere. You're not expected to be a pro at everything you do and that's okay, we're all learning.
This isn't about polars, but I'm sure I'm not the only one to experience main character devs. It makes me (and presumably others) not want to open issues on __any__ project, not just bad projects, and that's bad for the whole community (including me, because users find mistakes. And we know there's 2 types of software: those with bugs and those that no one uses). Stupid people are just wizards in training and you don't get more wizards without noobs.
It's a complete paradigm shift.
There's honestly not much that probably could've been shared at the point polars was conceived. Maybe now there's a little more (due to the arrow backend) but still very little probably.
As of last week, I have a need to process tabular data in Python. I started working with polars on Friday, and I have an analysis running across 16 nodes today. I find it very intuitive.
While that may be, I think it would make sense to describe the project in a succinct way on the first page a visitor lands.
You don't go car parts manufacturer expecting an explanation of what an intercooler is.
If they'd called them flurzles you wouldn't feel like you should understand if it's not something you work with.
You'd really have to be a complete data engineering newbie to not understand it I think?
I mean, where do you draw the line? You wouldn't expect a software tool like this to explain what it is in language my grandma would understand, I don't think?
I do occasionally use Pandas in my day job, but I honestly think very few programmers that could have use for a data frame library would describe themselves as a “data engineer” at all.
In my case, for example, I’m just a physicist - I don’t work with machine learning, big data, or in the software industry at all. I just use Pandas + Seaborn to process the results of numerical simulations and physical experiments similarly to how someone else might use Excel. Works great.
Anyway, probably the interesting things about polars are that it's like pandas, but uses a more efficient rust "backend" called Arrow (although I think that part's also in pandas now) and something like a "query planner" that makes combining operations more efficient. Typically doing things in polars is much more efficient than pandas, to the extent that things that previously required complicated infrastructure can often be done on a single machine. It's a very friendly competition, created by the main developer of pandas.
As far as I can tell everybody loves it and it'll probably supplant pandas over time.
I've been using pandas heavily, everyday, for something like 8 years now. I also did contribute to it, as well as wrote numpy extensions. That is to say, I'm fairly familiar with the pandas/numpy ecosystem, strengths and weaknesses.
Polars is a breeze of fresh air. The API of pandas is a mess:
* overuse of polymorphic parameters and return types (functions that accept lists, ndarrays, dataframes, series or scalar) and return differently shaped dataframes or series.
* lots of indirections behind layers of trampoline functions that hide default values behind undocumented "=None" default values.
* abuse of half baked immutable APIs, favoring "copy everything" style of coding, mixed with half supported, should-have-been-deprecated, in-place variants.
* lots and lots of regressions at every new release, of the worst kind ("oh yeah we changed the behavior of function X when there is more than Y NaNs over the window")
* Very hard to actually know what is delegated to numpy, what is Cython/pandas, and what is pure python/pandas.
* Overall the beast seemed to have won against its masters, and the maintainers seem lost as to what to fix versus what to keep backward compatible.
Polars fixes a lot of these issues, but it has some shortcomings as well. Mainly I found that:
* the API is definitely more consistent, but also more rigid than pandas. Some things can be very verbose to write. It will take some years for nicer simpler "shortcuts" and patterns to emerge.
* The main issue IMHO is polars' handling of cross sectional (axis=1) computations. Polars is _very_ time series (axis=0) oriented, and most cross sectional computations require to transpose the data frame, which is very slow. Pandas has a lot of dedicated axis=1 implementations that avoid a full transposition.
Sure, but many others are natively axis=1-aware and avoid full transposition.
> Axis=1 belongs in matrices, not in heterogeneous data.
I'm not sure to understand what that means. Care to elaborate?
> They are a performance footgun.
You don't get to only solve the problems that are efficient to solve...
> We make the transpose explicit.
Yes, but when you do mixed time series / cross sectional computations, you cannot always untangle both dimensions and transpose once. Sometimes your computation intrinsicely interleaves cross sectional and time series. In these case, which happen a lot in financial computations, then explicitly fully transposing is very slow.
I've been recently getting into DevOps stuff, and this is exactly what that sounds like to me. Same thing whenever I look at any new web framework:
Okay, cool, so your product brings the power of "WazzupJS" to "the edge"... But it also has "teleports" like "Blimper" but without all of the downsides of "Vormhole Blues"?
I'm sure that's really useful, but I kinda wish I knew what the product actually does
Polars is implemented in Rust and uses Arrow to represent some data in memory.
Pandas is often used in a way that results in a lot of processing happening in Python. Data processing in Python is comically inefficient and single-threaded. Beating that sort of pipeline by 10-100x is not that difficult if you have optimization experience and are able to work in a more suitable language.
Detailed Comparison Between Polars, DuckDB, Pandas, Modin, Ponder, Fugue, Daft - https://news.ycombinator.com/item?id=37087279 - Aug 2023 (1 comment)
Polars: Company Formation Announcement - https://news.ycombinator.com/item?id=36984611 - Aug 2023 (52 comments)
Replacing Pandas with Polars - https://news.ycombinator.com/item?id=34452526 - Jan 2023 (82 comments)
Fast DataFrames for Ruby - https://news.ycombinator.com/item?id=34423221 - Jan 2023 (25 comments)
Modern Polars: A comparison of the Polars and Pandas dataframe libraries - https://news.ycombinator.com/item?id=34275818 - Jan 2023 (62 comments)
Rust polars 0.26 is released - https://news.ycombinator.com/item?id=34092566 - Dec 2022 (1 comment)
Polars: Fast DataFrame library for Rust and Python - https://news.ycombinator.com/item?id=29584698 - Dec 2021 (124 comments)
Polars: Rust DataFrames Based on Apache Arrow - https://news.ycombinator.com/item?id=23768227 - July 2020 (1 comment)
> A couple of personal points that I may as well insert here. The account I'm now using, dang, was used briefly in 2010 by someone who didn't leave an email address. I feel bad for not being able to ask them if they still want it, so I'm considering this an indefinite loan. If you're the original owner, please email me and I'll give it back to you. The reason I want the name dang is that (a) it's a version of my real name and (b) it's something you say when you make a mistake.
You're probably too late.
Like a decade too late it seems
From the 2014 thread, it was a (four years and) three months offer:
“I was thinking that because it's just software, we could restore the old account to its exact prior state. But the attention that accrues to a moderator account would make that impossible. So I guess I'll keep the offer open for three months, after which we can make some other arrangement.”
Polars feels better than pandas in every way (faster + multi-core, less memory, more intuitive API). The library is still relatively young which has its downsides but in my opinion, at minimum, it deserves to be considered on any new project.
Easily being able to leverage the Rust ecosystem is also awesome - I sped up some geospatial code 100X by writing my own plugin to parallelize a function.
It's slightly ironic that you mention this, because I always thought the biggest problem with Pandas was its documentation. Case in point: did you know there's a way to join data frames without using the index? It's called "merge" rather than "join".
Pandas was originally very heavily inspired by R terminology and usage patterns, where the term "merge" to mean "join" was already commonplace. If I didn't already know R when I started learning Pandas (~2015), I don't think I'd have been able to pick it up quickly at all.
chatgpt (even the free tier) solved that problem for me. I ask it what I want in sql terms (or just plain english) and it tells me the pandas spell invocation. It even started to make sense after a few kLOC...
The join syntax and semantics is one of the trickiest parts and is under discussion again recently. It's actually one of the key parts of any data transformation platform and is foundational to Relational Algebra, being right there in the "Relational" part and also the R in PRQL. Most of the PRQL built-in primitive transforms are just simple list manipulations like map, filter or reduce but joins require care to preserve monadic composition (see for example the design of SelectMany in LINQ or flatmap in the List Monad). See this comment for some of my thoughts on this: https://github.com/PRQL/prql/issues/3782#issuecomment-181131... That issue is closed but I would love to hear any comments and you are welcome to open a new issue referencing that comment or simply tagging me (@snth).
Disclaimer: I'm a PRQL contributor.
There's an open issue for Substrait but I don't think that anyone's started any work on that yet.
I also think it’s awesome you guys have a duckdb integration - maybe I’ll try it out.
Not a big deal because I just read the docs but it was annoying that I couldn't have copilot just spit out what I need.
Forgive me for ranting here, but have people forgotten how to bootstrap their own knowledge about a new library? Taking notes isn't hard. Making a personal cheat-sheet isn't hard. I say all this AND I use LLMs very frequently to help with technical work. But I'm mindful about the tradeoffs. I will not let the tool steer me down a path that isn't suitable.
I'm actually hopeful: there is an unexpected competitive advantage to people who are willing to embrace a little discomfort and take advantage of one's neuroplasticity.
Continuing to choose it in the face of - in their own words - a better option, is a bit mind-boggling to me.
I still imagine the opposite impact... Welcome to no-moats-lang.io! So, you've created yet another new programming language over the holidays? You have a sandbox and LSP server up, and are wondering what to do next? Our open-source LLMs are easily tuned for your wonderful language! They will help you rapidly create excellent documentation, translators from related popular languages, do bulk translation of "batteries" so your soon-to-be-hordes of users can be quickly productive, and create both server and on-prem ChatOverflowPilotBots! Instant support for new language versions, and automatic code update! "LLM's are dynamite for barriers to entry!" - Some LLM Somewhere Probably.
Once upon a time, a tar file with a compiler was MVP for a language. But with little hope of broad adoption. And year by year, user minimum expectations have grown dauntingly - towards extensive infrastructure, docs, code, community. Now even FAMG struggle to support "Help me do common-thing in current-version?". Looking ahead, not only do LLMs seemingly help drop the cost of those current expectations to something a tiny team might manage, but also help drop some of the historical barriers to rapid broad adoption - "Waiting for the datascience and webdev books? ... Week after next."
We might finally be escaping decades of language evolution ecosystem dysfunction... just as programming might be moving on from them? :/
Not making a judgement, just finding it interesting.
Anyway, for what is worth, Copilot learns fast in your repos, very fast.
I use an extremely custom stack made of TS-Plus a TypeScript fork that not even the author itself uses nor recommends and Copilot churns very good TS-Plus code.
So don't underestimate how good can copilot can get at the boilerplate stage once he's seen few examples.
That sounds really interesting and valuable, I just have no idea where to start.
Thus it works even for relatively obscure stuff like this https://github.com/ts-plus
About me: I've used and disliked the Pandas API for a long time. I'm very proactive about continual improvement in my learning, tooling, mindset, and skills.
This is...such a strange take. To follow your logic to an extreme, everyone should use a very small handful of languages that are the "best" in their domain with ne'er a care for their personal comfort or preference.
> for your own sake
They're sticking with Pandas exactly for their own sake since they like being able to use Copilot.
> for the sake of open source innovation
Ohh by all means let's all be constantly relearning and rehashing just to follow the latest and greatest in open source innovation this week.
Tools are designed to be _used_ and if you like using a tool _and_ it does the job you require of it that seems just fine to me; especially if you're also taking the time to evaluate what else is out there occasionally.
Doubly so if it involves copilot. There's no way to get training data without people writing it. This sound like a direct application of a greedy algorithm: trading long term success for short term gain. That's not the ideal way to live.
I also like thinking about this as a feedback loop (as explained by systems dynamics), since it provides nice concepts for how systems change over time.
(a) For one's own sake, please pick the better tool as evaluated over suitable timeframe (perhaps an hour or two, after you've got some familiarity and muscle memory for the API) instead of only a brief evaluation (e.g. only 15 minutes).
(b) Better open source tools (defined however you want), which benefit us all, get better uptake when people think beyond merely the short-term.
The essence of my argument is "think beyond the short-term". Hardly controversial.
Don't miss the context: LLMs are giving people even more excuses for short-term thinking. Humans are terribly tempted for short-sighted "victories".
https://blog.jupyter.org/bringing-modern-javascript-to-the-j...
I'm not personally a Data Science guy, but considering how early the JS/Jupyter ecosystem is, it was surprisingly quick to get pola.rs-based analysis up and running in TypeScript.
The JS bindings certainly need a bit of love, but hopefully now that it's more accessible we'll see some iteration on it.
For anyone who doesn't use those libraries, they are all targeted towards out-of-core data processing (i.e. computing across multiple machines because your data is too big). Comparing them to a single core data frame library is just silly, and they will obviously be slower because they necessarily come with a lot of overhead. It just wouldn't make sense to use polars in the same context as those libraries, so seeing them presented in benchmarks as if they are equivalents is a little silly.
And on top of that, duckdb, which you might use in the same context as polars and is faster than polars in a lot of contexts, isn't included in the benchmarks.
The software engineering behind polars is amazing work and there's no need to have misleading benchmarks like this.
Duckdb.query(sql).pl() is much faster than duckdb.query(sql).df(). It’s zero copy to Polars and happens instantaneously while Pandas takes quite a while if the DataFrame is big. And you can manipulate it like a Pandas DataFrame (albeit with slightly different syntax).
It’s greater for working with big datasets.
I use Pandas from time to time and I'll probably try this out, but I always find myself wishing I'd just started with shoving whatever data I'm working with into Postgres.
It's not like I'm some database expert either, I'm far more comfortable with Python, but the facilities for selecting, sorting, filtering, joining, etc tabular data are just way better in SQL.
But you may want to look into DuckDB, which has a sql implementation that is not ad hoc, bug ridden, slow, or incomplete (though I honestly don't know about the formality of its specification). And it is compatible with polars :)
DuckDB allows you to work on your Polars and Pandas (and any data on Arrow format) directly using SQL without needing any data copying or duplication.
The duckdb-prql allows you to use PRQL (prql-lang.org) which gives you all the power and universality of SQL with the ergonomics and DX of Polars or Pandas (IMHO).
Disclaimer: I'm a PRQL contributor.
Postgres would be an order of magnitude slower than OLAP query engines for the types of queries that people do with them.
this is a better approach to Python dataframes and SQL
Unfortunately polars was not it. Too many bugs on standard operations, unreliable interoperability with pandas (which is an issue since so many libraries require pandas dataframes as inputs), the API is also very verbose for a modern dataframe library, though it's still better than pandas.
Hopefully these will get resolved out over time but for now I had the best luck using duckdb on top of pandas, it is as fast as polars but more stable/better interoperability.
Eventually I hope the Python dataframe ecosystem gets to the same point as R's, where you have a analytics-oriented dataframe library with an intuitive API (dplyr) that can be easily used alongside a high-performance dataframe library (data.table).
The rest of the tidyverse stuff is OK (like forcats), but the overall ecosystem is a little weird. The focus on "tidy" data itself is nice up to a point, but sometimes you want to just move data around in imperative style without trying to figure out which "tidy verb" to use, or trying to learn yet another symbol interpolation / macro / nonstandard eval system, because they seem to have a new one every time I look.
Pandas is a real workhorse overall. Data.table is like a fast sports car with a very complicated engine, and Pandas is like a work van. It's a little of everything and not particularly excellent at anything and that's ok. Also its index/multiindex system is unique and powerful. But data.table always smoked it for single-process in-memory performance.
Until DuckDB and Polars, there was no Python equivalent of data.table at all. They're great when you want high performance, native Arrow (read: Parquet) support, and/or an interface that feels more like a programming library than a data processing tool. If you're coming from a programming background, or if you need to do some data processing or analytics inside of production system, those might be good choices. The Polars API will also feel very familiar to users of Spark SQL.
For geospatial data, Pandas is by far superior to all options due to GeoPandas and now SpatialPandas. There is an alpha-stage GeoPolars library but I have no idea who's working on it or how productive they will be.
If you had to learn one and only one, Pandas might still be the best option. Python is a much better general-purpose language than R, as much as I love R. And Pandas is probably the most flexible option. Its index system is idiosyncratic among its peers, but it's quite powerful once you get used to using it, and it enables some interesting performance optimization opportunities that help it scale up to data sets it otherwise wouldn't be able to handle. Pandas also has pretty good support for time series data, e.g. aggregating on monthly intervals. Pandas also has the most extensibility/customizability, with support for things like custom array back ends and custom data types. And its plotting methods can help make Matplotlib less verbose.
I've never gotten past "hello world" with Julia, not for lack of interest, but mostly for lack of time and need. I would be interested to hear about that comparison as well.
DataFrames.jl, DataFramesMeta.jl, and the rest of the ecosystem are outstanding. Very similar to pandas, and much ... much faster. If you are dealing with small (obviously subjective as to the definition of small) dfs of around 1000-10000 rows, sticking with pandas and python is fine. If you are dealing with large amounts of real world time series data, with missing values, with a need for data cleanup as well as analytics, it is very hard to beat Julia.
FWIW, I'm amazed by DuckDB, and have played with it. The DuckDB Julia connector gives you the best of both worlds. I don't need DuckDB at the moment (though I can see this changing), and use Julia for my large scale analytics. Python's regex support is fairly crappy, so my data extraction is done using Perl. Python is left for small scripts that don't need to process lots of information, and can fit within a single terminal window (due to its semantic space handicap).
Is there any kind of decent support for plotting with data frames? Or does Plots.jl work with it out of the box?
tldr: Julia is my favorite.
I could never click with R. It is true that data.table and dplyr and ggplot are well done and I think we owe a debt of gratitude to the community that created them. But the language itself is ... not good. But that's just, like, my opinion!
Pandas I also have really never clicked with. But I like python a lot more than R, and pandas basically works. For what it's worth, the polars api style is more my thing. But most of the data scientists I work with prefer the pandas style, :shrug:.
But I really like this part of Julia. It feels more "native" to Julia than pandas does to python. More like data.table in R, but embedded in a, IMO, even better language than python. The only issue is that Julia itself remains immature in a number of ways and who knows whether it will ever overcome that. But I hope it does!
But it's a lot more fun when you realize that it's an homoiconic array language with true lazily-evaluated F-exprs (not Rebol/Tcl strings).
I probably wouldn't use it for EDA or research, but I have started to use it in certain production scripts for the better performance.
R dplyr + data.table is still my favorite data manipulation experience. I just wish we had something like Matplotlib in R: ggplot is too high level, base graphics are too low level. Also Scikit-Learn is much more modular than Caret, which I don't really miss using.
plotly could be worth a try, i use its python bindings and much prefer it to matplotlib, but i don't know much about the quality of it's R API
IMHO this gives you the DX of dplyr / Polars / Pandas combined with the power and universality of SQL because you can still execute your queries on any SQL compatible query execution engine of your choice, including Polars and Pandas but also DuckDB, ClickHouse, BigQuery, Redshift, Postgres, Trino/Presto, SQLite, ... to name just a few popular ones.
I'd love to hear your thoughts, either in a Discussion on Github (https://github.com/PRQL/prql/discussions) or on our Discord (https://discord.com/invite/XWxbCrWr)!
Disclaimer: I'm a PRQL contributor.
The API is still seeing some (expected) breaking changes, and could become a maintenance burden across multiple projects. But the API already feels more consistent, and overall seems to be going in the right direction.
Looks like you ought to set aside some time to do updates each quarter, but I do wonder how much breakage can there be in practice, most changes seem pretty niche.
Rust Polars is able to handle this using Lazy DataFrames / Streaming without issue.
A basic approach which worked was sequentially loading each df from the filesystem, iterating through record hashes, and incrementing a counter; however the runtime was an order of magnitude greater than my final implementation in Polars.
I've found being able to express ideas clearly in code (to aid comprehension now and in the future) to be much more important than shaving off a few seconds of run time.
For this reason I think speed alone is not a strong sell point, except specifically in cases where execution times really matter.
Analogous somewhat to how ruby/rails might be a 'slow' language/framework (e.g. 600ms when another framework might be 200ms) but multiples faster in facilitating the expression of complex ideas through code, which tends to be the far bigger problem in most software projects.
That being said, there are many, many times when I would be willing to rewrite code to make it faster or more memory efficient. Just yesterday I rewrote a method from Pandas to Polars to take advantage of that.
Time might be cheap (while developing) but memory is expensive. Similarly, if you're writing data science jobs for production, you care about both.
Since a lot of people use pandas to explore and experiment with datasets, having your workflow speed limited to a few operations an hour is hard to defend. That’s where the value proposition of polars and similar solutions lie IMO.
Good example; I can see how efficiency would matter for workflows like that.
I work with dataframes in the 10's-100's millions of rows (mostly in tidyverse, but also pandas and base python and R), and find most data wrangling operations are close to instant on modern laptops. Plotting is another story (not sure if polars helps there).
So the case for efficiency is weak at the 10-100 million row dataframe size (unless doing some intense computations), but gains strength as the size of the dataframe grows.
Would be a fun aside to test all these frameworks side by side with some 1m/10m/100m/1bn row joins, filters, summary calcs, maps etc to get some concrete data on where efficiency starts to become noticeable and starts to matter. I think at sub 100m rows it probably doesn't. Not for the kinds of operations I do anyway.
Most of my life I've had databases of like 1000. Now I have a big one of about 500K! So for me, speed is almost a non-issue. But that is my specific field.
I never liked pandas much due to its annoying API, indices and single thread only implementation (we usually get a 10x performance boost at least and for me that also means improvements in productivity). Also, pandas never handled NULLs properly. This should now work with the pyarrow backend in pandas, but we can’t read our parquet files generated by PySpark. With polars it mostly just works, but we use pyarrow to read/write parquet.
Overall I can recommend it, conversion from/to pandas DataFrames was never an issue as well.
Quivr (https://github.com/spenczar/quivr) is an alternative approach that has been working for me. Maybe types are good!
1: https://kevinheavey.github.io/modern-polars/method_chaining....
Because you ideally describe everything in terms of lazy operations, it actually internally keeps track of all your data types at every step until materialization and execution of the query plan. Because of that, you're not going to have the same kind of data type issues you might have in pandas. There are also libraries based off pydantic built for polars dataframes validation (see patito, though it's not mature yet).
This actually means you can have zero copy swaps between polars, pandas and duckdb with very little runtime impact.
Definitely a big help if your using a non-pandas df library since the conversion to pandas when needed is now really cheap.
If you're just reading a small CSV to do analysis on it, then there will be no human-perceptible difference between Polars and Pandas. If you're reading a larger CSV with 100k rows, there still won't be much of a perceptible difference.
Per this (old) benchmark, there are differences once you get into 10 million rows/500MB+ territory: https://h2oai.github.io/db-benchmark/
I'm not sure how much faster I could go, since ~1 second/dataset allowed me to answer some questions that I had that required scanning values for a few parameters. The biggest wins for me were in grouping and merging operations.
I'm a complete convert now. The API is simpler and more obvious IMO, and the ability to compose expressions (`polars.Expr`) is awesome. The performance benefits are nice and what motivated me in the first place, but I'm more swayed by the aforementioned benefits.
- I can't rule out that a pandas wizard couldn't have achieved the same speed-up in pandas
- polars code was slightly more verbose. For example, when calculating columns based on other columns in the same chain, in pandas, each new column can be defined as a kwarg in a single call to `assign`, whereas in polars, columns that depend on other must be defined in their own calls to `with_columns`
- handling of categoricals in polars seemed a little underbaked, though my main complaint, that categories cannot be pre-defined, seems to have been recently addressed: https://github.com/pola-rs/polars/issues/10705
- polars is not yet 1.0, breaking changes will happen
Edited to add: also, if you’re using a lazy dataframe, you can just naively do the same operation twice (once to store it in a named column and once again in the subsequent computation), and Polars will use common subexpression elimination (CSE) to prevent recomputing the result. You can verify this is true using the `.explain()` method of a lazy dataframe operation containing the `.with_columns()` call.
But yeah, polars is awesome, I'm all in on it.
Perhaps I can focus on a subset of this processing and write this up since it seems like there's at least some interest in real examples. As pointed out in a reply to a sibling comment, I don't guarantee that my starting code is the best that pandas can do -- to be honest, the runtime of the original code did not line up with my intuition of how long these operations should take. Maybe someone will school me but either way switching to polars was a relatively easy win that came with other benefits and feels right to me in a way that pandas never did.
Nowadays, we write a pure pandas version, and when the data needs to be 100X bigger and faster, change almost nothing and have it run on the GPU via cudf, a GPU runtime that fully follows the pandas API. Most recently, we port GFQL (Cypher graph queries on dataframes) to GPU execution over the holiday weekend and it already beats most Cypher implementations. Think billions of edges traversed per second on a cheap 5 year old GPU.
We're planning the bigger than memory & multi node versions next, for both CPU + GPU, and while cudf leans towards dask_cudf, plans are still TBD. Polars, Ray, and Dask all have sweet spots here.
That, plus parallelism, probably explains the performance difference. If anything, 60x sounds conservative to me.
My only critique of the Rust crate is that it's not as well documented as the Python API and the Rust API required a lot more unwrapping and error handling than other Rust crates which was quite tedious.
but now I've been using it 99% exclusively for the past 2–3 months. I would say it's ready for prime time! I don't get any @TODO seg faults anymore lol
I don't know if I notice the speed difference over Pandas most of the time, but I do find the way of expressing transformations way more intuitive than Pandas, it's more similar to SQL.
It does use pyo3. Though basically anytime you use a python lambda (the equivalent of pandas apply) on a row basis, it will be limited by python speed/the GIL.
Excellent work!
1. Querying and Data processing in Polars or DuckDB
2. Metadata (for transactions, time-travel) table formats (Iceberg, Hudi, Delta)
3. Storage with Parquet on S3.
With Polars/DuckDB, you can process up to 100+ GB of data at a time on a single VM, turn it off when you're not using it. It's a lower overhead, lower time-to-value stack than Spark. And even if you have TBs of data, so long as you only process ~100GB at a time, it works fine.
https://motherduck.com/blog/big-data-is-dead/
Is a good read
Happy to assist and answer any questions I can.
Regarding polars, would you have time to answer these questions?
1) How are polars supported by popular data science packages, e.g., for plotting?
2) I know it is a bit silly: is there a way to get around typing `pl.col`, etc. all the time?
3) Besides `tidypolars`, are there any reasonable packages that add support for dplyr-style pipes or operation chaining?
2 - If you just want the columns, there is no need to make an expression; just pass the string. There are shortcuts like pl.sum(col) instead of pl.col(col).sum(). I guess you could shortcut, c = pl.col, if you really hate typing it...
3 - Polars supports chaining and encourages it out of the box. Anything specific you are looking for?
1. `from polars import col` which at least shortens each use by a few characters 2. For columns that I use very frequently, define a "constant" for them e.g. ID=col("id").
This lets you do df.groupby(ID) instead of df.groupby(pl.col("ID")). Another advantage of defining these column "constants" is that it makes it much easier to refactor to rename all usages of the column (without needing to check whether each string "id" is being used as a column name vs. something else)
They all stay at your computer and in your RAM though.
I wonder, what is the benefit of using this compared to a truly distributed engine like spark? If your job barely scales maybe that is what you need rather than maximizing your cores or memory usage?
(Note, I have not worked with spark, so it is an honest question.)
Matt Rocklin (the dask creator) has put some effort into benchmarking across all these recently. You can see the presentation here: https://youtu.be/5YXiRl41DKA?si=5Pt9XQHp5Y1cG7tM
If you don't feel like watching a whole presentation, the main take away is local is faster, until it isn't.
Polars outperforms spark by a wide margin, until data size gets sufficiently large that things swap over.
Some operations can be performance sensitive without having scaling issues. So I guess those kind of jobs are great candidates for something like polars.
I think I am a guy who prefer reliability and versatility over speed.
(When I was a kid you'd say Alfa Romeo vs Volvo)
For disclosure I'm a minor contributor to dask so probably am a little biased.
I guess one side I probably haven't put forward though is that the memory footprint of something like dask/spark is higher because of its overheads. If you don't have scalable resources, then a polars / duckdb option would probably be your most reliable choice (I.e. the one that'll hit the fewest memory errors in the given architecture)
By the way, that pie chart on the home page is a crime.
Not sure where this fits in to any workflow tbh, with sufficiently large datasets, you will inevitably need spark (which has same API as pandas)
Note: I do like Polars but have not used it in work setting.
When would you use dataframes (e.g Pandas, Polars...) and when would you use tensors (Pytorch, TF...)?
Are the usecases completely distinct or is there overlap?
you still transform data in tensors, but generally that's one-hot encoding or transposing or other transformation done right before model training. before that, you might use a dataframe to cleanup strings, aggregate timeseries data, etc.
hope that makes sense. so yes there's some overlap, but generally they're distinct toolsets that would be used together for an end-to-end ML project
> Polars will remain MIT-licensed and the company will sponsor and accelerate the open-source development of Polars.
I bet that they will change to a custom business source available license sooner or later.
From the about page: [0]
> We successfully closed a seed round of approximately 4M$, which was lead by Bain Capital Ventures.
Waiting for the day that a seperate closed-source version of polars gets built / forked privately and improvements do not make it back into the source.
Seems like fairly strong assurance to me
Quick history lesson: MT owned blogging, got VC backed, changed the license because capitalism, and everyone fled to WordPress which powers over 40% of the Web today. I think looking at the way Matt has run WP and the level of sensitivity he's had to have towards OSS/GPL in order to maintain the ecosystem and community, is instructive when it comes to the viability of an OSS AI VC backed C corp.
should be parallelism
Conclusion: Polars gives you indigestion.