Pandas extension arrays
pandas-dev.github.io
pandas-dev.github.io
Today, we default to using pydata gpu arrow tools like BlazingSQL or Nvidia RAPIDS directly. They ~guarantee perf, and subtle yet critical for maintaining our < 100ms SLA, the Arrow format stays clean. (Ex: don't want a column schema to get coerced to something less structured.) We'll use Pandas as a fallback for when they lack features or are hard to use.
The ideal would be to use Pandas directly. Today it is a bit of a crapshoot on whether schemas will break across calls, and the above libraries are really replacements, rather than integrated accelerator extensions. So thinking like this project get us closer to predictable (and GPU-level) performance within pandas, vs fully replacing it. So cool!
If you don't understand and want to understand, you can always politely ask for an explanation.
For a more technical discussion of the performance implications of one possible implementation, see http://wesmckinney.com/blog/bitmaps-vs-sentinel-values/
Not allowing nullable type in raw integer `Series` is the least bad solution this problem. If you really want nulls in numerical data, either use floating point or `IntegerArray`.
n = 2**25
data = np.random.choice(list(string.ascii_letters), n)
df = pd.DataFrame({
'string': data,
'arrow': fl.FletcherArray(data),
'categorical': pd.Categorical(data),
'ints': np.arange(n)
})
- String: 1.58 s
- Arrow: 886 ms
- Categorical: 406 ms