We do cryptocurrency mining on an industrial scale and constantly see single bit errors from hardware that is brand-new without modifications.
That's very surprising and interesting. How do you detect these single bit errors?
That's very surprising and interesting. How do you detect these single bit errors?
Detection of false negatives: Compare solution distribution and frequency to expected models; switch to debug kernels if outside tolerance.
However, this works because the mining problem space is stateless and follows strict mathematically predictable models.
A DB is stateful and the answers generally can't be verified without consulting a secondary copy, which is why I'm super curious how they would engineer correctness and reliability in a cost-effective way using GPUs.
I'm assuming you are using GeForce cards and not Tesla cards which have an ECC memory protection mode?
I've tried to collect some statistics on GPU memory errors rates but have found them to be normally extremely rare. The only time I've reproducibly seen them is due to faulty hardware, where the errors become highly reproducible and the GPU needs replacement. The other theoretical cause of bit flips is supposed to be random errors due to cosmic radiation but I've never been able to observe that using memory testing software (though I did only run the experiments in AWS).
Could it be that you have faulty or low grade GPUs? I assume these are all low-cost OEM parts, given your application? Or maybe there's something odd about your data center environment?
Regarding the GPU database application, I think the answer is to just use the Tesla grade GPU with ECC memory enabled.
AMD's hardware specifications are more open too which lets one build your own shader compilers and get direct access to the iron.
Dont the GPU specs allow for a certain lossiness in the math? Or like, at least they don't conform to IEEE 794 float specs w/ regard to order of operations, precision, degradation, etc.
So like, do a shitload of math ops in a glsl shader with a deterministic outcome, render the result to a texture, take the texture back and make sure the RGBA values match bit for bit with the numbers you expected?
Or to detect single bit errors in the gpu's local memory or caching just attach textures, read data into them, read it back, render back, etc etc.