litl

TL;DR

litl (Lightweight InTerface for Lossy compression) is a small Python framework I built for writing and comparing lossy compressors. You write a compressor as a single class, and litl gives you a command-line tool, a file format, and quality metrics. Install it with pip install litl.

Problem

Comparing a neural compressor against a classic one, like SIREN against TTHRESH in my INR blog post, takes the same steps for every compressor: load the data, time the run, compute PSNR and SSIM, work out the compression ratio. litl does those steps, so the compressor only has to compress and decompress.

One class

A compressor is a class with four methods: about and blob_class describe it, and compress and decompress do the work. Here is a complete one that rounds every value to 8 bits:

import numpy as np

from litl import DataWrapper
from litl.blobs import ArrayBlob
from litl.compressors import Compressor, CompressorAbout


class Quantize(Compressor):

    @classmethod
    def about(cls):
        return CompressorAbout(name="Quantize", description="Rounds every value to 8 bits", version="1.0.0")

    @classmethod
    def blob_class(cls):
        return ArrayBlob

    @classmethod
    def compress(cls, data: DataWrapper, config: dict):
        array = data.numpy()
        low, high = float(array.min()), float(array.max())
        step = (high - low) / 255

        quantized = np.round((array - low) / step).astype(np.uint8)
        return ArrayBlob(quantized), {"min": low, "step": step}

    @classmethod
    def decompress(cls, blob: ArrayBlob, meta: dict):
        array = blob.array * meta["step"] + meta["min"]
        return DataWrapper(array.astype(np.float32))

A blob can be an array, raw bytes, or a whole PyTorch model, so the compressed file can be a neural network.

Try it

Here is that class in a folder with some data: a smooth 64×64×64 field saved as field.npy. The terminal below replays output recorded from litl 0.1.3, so go ahead and type.

In those recordings 1 MB goes in and a 46 KB file comes out, at a PSNR of 53. The same four commands (compress, decompress, evaluate, info) work for any compressor: to try a different one, change the first argument.

The .litl file

Whatever your compressor returns gets wrapped into a .litl file, together with what litl needs to open it again later: the litl version, which compressor made it and its version, and the metadata the compressor returned.

$ litl info field.litl
File field.litl was compressed with quantize.py:Quantize v1.0.0 using litl v0.1.3
Metadata:
{'min': -1.499533772468567, 'step': 0.011761049195831897}

See more