14 November 2016

Bit Grooming Improves Precision/Storage Ratio

Bit Grooming: Statistically accurate precision-preserving quantization with compression, evaluated in the netCDF Operators (NCO, v4.4.8+)

Science

We developed an algorithm called Bit Grooming that significantly reduces the volume of data while retaining the desired precision.

Impact

Bit Grooming reduces the storage required by 40-80% and does not require any special software to read. This allows models to export higher time-frequency or spatial resolution data without increasing storage costs.
 

Summary

We introduce Bit Grooming, a lossy compression algorithm that removes the bloat due to false-precision, those bits and bytes beyond the meaningful precision of the data. Bit Grooming is statistically unbiased, applies to all floating point numbers, and is easy to use. Bit-Grooming reduces data storage requirements by 25-80%. Unlike its best-known competitor Linear Packing, Bit Grooming imposes no software overhead on users, and guarantees its precision throughout the whole floating point range.

Contact
Charles S. Zender
University of California Irvine