14 November 2016

Bit Grooming Improves Precision/Compression Ratio

Bit Grooming: Statistically accurate precision-preserving quantization with compression, evaluated in the netCDF Operators (NCO, v4.4.8+)

Science

We developed an algorithm called Bit Grooming that significantly reduces the volume of data while retaining the desired precision and eliminating statistical biases often associated with lossy compression.
 

Impact

Bit Grooming reduces the storage required by 40-80% and does not require any special software to read. This allows models to export higher time-frequency or spatial resolution data without increasing storage costs.
 

Summary

We introduce Bit Grooming, a lossy compression algorithm that removes the bloat due to false-precision, those bits and bytes beyond the meaningful precision of the data. Bit Grooming is statistically unbiased, applies to all floating point numbers, and is easy to use. Bit-Grooming reduces data storage requirements by 25-80%. Unlike its best-known competitor Linear Packing, Bit Grooming imposes no software overhead on users, and guarantees its precision throughout the whole floating point range. 
 

Contact
Charles S. Zender
University of California Irvine