strings as objects and integers turning into floats when NaNs are introduced have been a much bigger annoyance to me, than it ought to.
I'm excited to try out the new pyarrow dtypes, but it also sounds confusing that there are now 2 classes of types
I'm excited to try out the new pyarrow dtypes, but it also sounds confusing that there are now 2 classes of types
I've implemented a function for schema based processing JSON documents for both vanilla python and pyspark that makes the process really easy. It'll take a schema and a document and product a list of flat dictionaries for python or a data frame for pyspark. Vanilla python is really streamable and keeps memory overhead low so it was actually faster than the pandas based workflows that it replaced.
Nah, you rightly are annoyed. When I am writing unit tests, it is especially annoying to fix the type.