Once you read and prune out the dataset to only include the relevant data, then what's left for the GPU to do?

We are not shipping all of the data as a whole to the gpu. We are going to be releasing some whitepapers that explain this in more detail but lets get a few things clear. Data is sent to the gpu compressed since it is compressed when it is stored. We can decompress VERY quickly on GPU's (30-50GB/s is easily achievable on a K80) and because each of our columns are compressed using one of our cascading compression algorithms (which everyone offers the best in terms of compression and throughput). We are a column store and only send over the columns that are being used in processing. So for example

select id, name, age, avg(income) from people group by gender

In this case only the income and gender columns would actually be sent to the gpu and they would do so in a compressed fashion to increase the "effective" bandwidth of data over PCIE. Even more interesting is that id, name, and age, would be pulled from our horizontal store instead of our compressed columnar store in order to minimize the number of iops necessary to fill the result set.