Compression ​

compress & L4 are nice Go packages for compression. Brotli is amazing.

Zstandard is great.

Use Squoosh to compress images without losing quality. Brotli-WASM is great compression library.

Notes ​

• Can compress PNG images well with pngquant.
• LZ4 compression is fast

Links ​

• Zstandard - Fast real-time compression algorithm. (HN) (HN) (HN) (HN)
• Orz - General purpose data compressor written in rust.
• FiniteStateEntropy - New Generation Entropy coders.
• Snappy - Fast compressor/decompressor.
• Snappy Node - Fastest Snappy compression library in Node.js.
• UPX - Advanced executable file compressor.
• gzip-js - Pure JavaScript implementation of the GZIP file format.
• Blosc - Blocking, shuffling and lossless compression library. (HN)
• wasm-flate - Fastest compression and decompression in the browser.
• zlib - General purpose data compression library.
• zlib-ng - zlib data compression library for the next generation systems.
• LZ4 - Extremely Fast Compression Algorithm. (Web) (HN)
• How LZ4 works (2016)
• Handbook of Data Compression (2010)
• Faster integer compression in Go with SIMD using StreamVByte codec
• fast_rsync - Optimized implementation of librsync in pure Rust.
• SDefl - Small bare bone lossless compression library in ANSI C (ISO C90) that implements the Deflate (RFC 1951) compressed data format specification standard.
• Pixz - Parallel, indexed xz compressor.
• Flywheel: Google’s Data Compression Proxy for the Mobile Web
• Opus - Codec for interactive speech and audio transmission over the Internet.
• minizip - Zip manipulation library written in C that is supported on Windows, macOS, and Linux.
• Comparing zstd/gzip/bzip2/lzma/xz Compression (2020) (Lobsters)
• ZFP - Compressed format for representing multidimensional floating-point and integer arrays.
• PyTorch Implementation of the CVPR'19 Paper "Practical Full Resolution Learned Lossless Image Compression"
• Lossless compression of English messages using GPT-2 (HN) (Code)
• NNCP: Lossless Data Compression with Neural Networks (2019) (HN)
• Visualizing gzip compression with Python (2020) (HN)
• The Hitchhiker's Guide to Compression (Code) (HN)
• Miniz - Single C source file zlib-replacement library.
• Rationale for a Large Text Compression Benchmark (2009)
• LZSA - Byte-aligned, efficient lossless packer that is optimized for fast decompression on 8-bit micros.
• pigz - Parallel implementation of gzip for modern multi-processor, multi-core machines. (HN)
• Precomp - Command line precompressor that can be used to further compress files that are already compressed.
• pako - High speed zlib port to javascript, works in browser & node.js.
• A System to Transparently Compress Hundreds of Petabytes of Image
• Kanzi - Modern, modular, expendable and efficient lossless data compressor implemented in Go.
• Data Compression With Arithmetic Coding (2014)
• Snappy - Snappy compression format in the Go programming language.
• Semantic Compression (2014)
• fflate - High performance (de)compression in an 8kB package.
• etcpak - Extremely fast Ericsson Texture Compression and S3 Texture Compression (DXT1/DXT5) utility.
• Zopfli Compression Algorithm - Compression library programmed in C to perform very good, but slow, deflate or zlib compression.
• python-blosc - Python wrapper for the extremely fast Blosc compression library.
• zip - Portable, simple zip library written in C.
• Compressing data with Parquet (2021)
• Lossless Text Compression (HN)
• CompressAI - PyTorch library and evaluation platform for end-to-end compression research.
• zipindex - Package for indexing zip files and storing a compressed index.
• COIN: COmpression with Implicit Neural representations (2021) (Code)
• Zippy - Pure Nim implementation of deflate, zlib, gzip and zip.
• ezip2dmg - Simple wrapper utility to decompress an encrypted ZIP into a newly mounted encrypted DMG.
• Text Classification by Data Compression (2021) (HN)
• JPEG compression algorithm explained
• NeuralCompression - Python repository dedicated to research of neural networks that compress data.
• Compressing Multisets with Large Alphabets (2021) (Code)
• Ouch - Easy and painless way of compressing and decompressing files in the terminal.
• TinyJPG - Filesystem watcher and image compress.
• Faster Compression with Snappy's S2 Extension (2021)
• Faster ZIP Decompression (2019)
• Minimalist Guide to Lossless Compression (2019) (HN)
• FreeArc'Next
• Experiments with compression-related algorithms
• Imager - Automated image compression for efficiently distributing images on the web. (Web)
• Heatshrink - Data compression library for embedded/real-time systems. (HN)
• Neural Network Compression Framework (NNCF) - Provides a suite of advanced algorithms for Neural Networks inference optimization in OpenVINO with minimal accuracy drop.
• libdeflate - Heavily optimized library for DEFLATE/zlib/gzip compression and decompression.
• sltar - Minimal implementation of tar.
• Box - Open Standard Archive Format, akin to zip or 7z but without the legacy baggage.
• Box in D - Simple archive format with UTF-8 filenames.
• Bitbottle - Modern archive format. (Lobsters)
• libarchive - Multi-format archive and compression library. (Web)
• zlib by Cloudflare - Cloudflare fork of zlib with massive performance improvements.
• Inflate.jl - Julia implementation of zlib decompression.
• miniz_oxide - Rust replacement for the miniz deflate/zlib encoder/decoder.
• Hop - 25x faster than unzip and 10x faster than tar at reading individual files. (HN)
• Buzon - Writing large ZIP archives without memory inflation. (HN)
• The QOI File Format Specification (2021) (HN)
• QOI: Lossless Image Compression in O(n) Time (2021) (Tweet) (HN) (Code) (Lobsters) (HN)
• qoiview - QOI image viewer on top of the Sokol headers.
• qoi_rs - Rust port of the Quite Okay Image format.
• QOI - Rust implementation of the “Quite OK Image” format for fast, lossless image compression.
• zig-qoi - Implementation of the Quite-OK-Image format for Zig.
• QOI Go - Pure Go encoder/decoder of the QOI image format.
• QOI Rust - Fast implementation of QOI format in Rust.
• Implementing zip archiving in Golang: unzipping (2021) (Lobsters)
• lz-fear - Fast pure-rust no-unsafe implementation of LZ4 compression and decompression.
• Decompressing a Gzip File by Hand (HN)
• librsync - Remote delta-compression library.
• Quantile Compression - Lossless compressor and decompressor for numerical data using quantiles. (Reddit) (Article)
• fzstd - High performance Zstandard decompression in a pure JavaScript, 8kB package.
• snap - Snappy compression implemented in Rust (including the Snappy frame format).
• Image Compression Algorithm in Rust - New lossless image compression algorithm.
• mjpeg2http - Pure C Mjpeg-over-HTTP server. (HN)
• go-tsz - Time series compression algorithm from Facebook's Gorilla paper.
• Leanify - Lightweight lossless file minifier/optimizer. It removes unnecessary data (debug information, comments, metadata, etc.) and recompress the file to reduce file size.
• Brotli - Generic-purpose lossless compression algorithm that compresses data using a combination of a modern variant of the LZ77 algorithm. (Web) (Visualizing Brotli Decompression)
• Brotli JS - Google brotli binding to Node.js via Rust and napi-rs.
• lz-string - LZ-based compression algorithm for JavaScript.
• divANS - Crate meant to be used for generic data compression.
• Dictionary Compression (2022) (HN)
• Compressing and embedding a Wordle word list (2022) (Lobsters)
• zlib - Compression and decompression in the gzip and zlib format.
• QOI — The Quite OK Image Format (HN)
• littleZip - Memory-friendly basic zip compression, decompression, and single file extraction for deno framework.
• Libarchivejs - Archive tool for browser which can extract various types of compression.
• lz4dec - LZ4 block decompressor written in arm64 assembly.
• Packing Bits For Compression (2022)
• How LCEVC works – A new approach to video compression (HN)
• zstd-wasm - Zstandard for browser, Node.js and Deno.
• Crash Course on Data Compression
• Silly Image Compression Idea (2022) (HN)
• go-unarr - Go bindings for unarr (decompression library for RAR, TAR, ZIP and 7z archives).
• libzip - C library for reading, creating, and modifying zip archives.
• The smallest 256x256 single-color PNG file, and where you've seen it
• Efficient Compression Tool - C++ file optimizer. It supports PNG, JPEG, GZIP and ZIP files.
• WASM PNG Image compressor
• libimagequant - Palette quantization library that powers pngquant and other PNG optimizers.
• StreamVByte - Fast integer compression in C using the StreamVByte codec.
• BZip3 - Better, faster and stronger spiritual successor to BZip2.
• Time-series compression algorithms, explained (2020)
• Zstandard Worked Example (HN)
• Silly Lossy Text Compression Idea (2022) (HN)
• Lizard - Efficient compression with fast decompression. (HN)
• lzbase62 - LZ77(LZSS) based compression algorithm in base62 for JavaScript.
• Halve the size of images by optimising for high density displays (2021)
• What’s the best lossless image format? (2021) (HN)
• Survey of 3D Compression and Streaming
• compress.js - Simple JavaScript based client-side image compression algorithm.
• Python-LZ4 - LZ4 bindings for Python.
• The tar archive format, its extensions, and why GNU tar extracts in quadratic time (2022)
• unblob - Accurate and fast extraction suite for 30+ formats. (Docs) (Lobsters)
• Brotli-WASM - Reliable compressor and decompressor for Brotli, supporting node & browsers via WASM.
• ZSTD seekable compression format implementation in Go
• zstd compressed debug sections (2022)
• Vaar - Tar archive tool & library optimized for lots of small files.
• Binary Refinery - High Octane Triage Analysis.
• Dissecting the GZIP format
• Better than JPEG? Researcher discovers that Stable Diffusion can compress images (2022)
• Using AI to compress audio files for quick and easy sharing (2022) (HN)
• EnCodec: High Fidelity Neural Audio Compression - State-of-the-art deep learning based audio codec supporting both mono 24 kHz audio and stereo 48 kHz audio.
• Ratarmount - Random Access Tar Mount.
• Brotli-G - GPU compression/decompression standard for digital assets.
• Conflux - Zip/unzip files of any size in the browser using streams.
• QOIR lossless image file format was explained (2022)
• ripunzip - Tool to unzip files in parallel.
• Tinier - Make your videos, images and audio files tinier.
• Netflix: 30% bitrate saving using Film Grain Synthesis (2022) (HN)
• XCompress - Free file archiver utility on Linux, providing multi-format archiving to and extracting from ZIP, Z, GZIP, BZIP2, LZ, XZ, LZMA, 7ZIP, TAR, RAR and ZSTD.
• SOZip Spec - Specification of seek-optimized zip file profile.
• lzwCompress.js - Lossless LZW compression/decompression implemented in JavaScript for strings/JSON/JS objects.
• zx - Make extracting archive plain and simple.
• ndzip - High-Throughput Parallel Lossless Compressor for Scientific Data.
• Zap - Extremely fast alternative to zip which is written in rust.
• WASM compression benchmarks and the cost of missing compression APIs (2023) (HN)
• Video Compression for Mere Mortals (2023)
• Image Compression with Singular Value Decomposition (HN)
• JBIG2 - JBIG2 Encoder.
• GZ Heatmap - Upload a text file to see how GZip compresses your file. (Code)
• pio - Optimize images while maintaining the same perceived quality.
• Tight Model Format - Lossy 3D model format focused on reducing file size as much as posible without decreasing visual quality of the viewed model or read speeds.
• DataCompression - Swift libcompression wrapper as an extension for the Data type.
• The History of LZ Compression (2023)
• Squash - Compression Abstraction Library.
• ARM ASTC Encoder - Compressor for the Adaptive Scalable Texture Compression data format.
• ICER Image Compression Algorithm
• The Near-Lossless Image Formats Using Ultra-Fast LZ Codecs (2023) (HN)
• sevenzip - Go library for dealing with 7-zip archives.
• Faster LZ is not the answer to 150-250 GB video game downloads (2023) (HN)
• 7-Zip-zstd - 7-Zip with support for Brotli, Fast-LZMA2, Lizard, LZ4, LZ5 and Zstandard.
• Pigz - Parallel gzip for modern multi-processor, multi-core machines. (HN)
• ncompress - Fast, simple LZW file compressor.
• How Lossless Data Compression Works (2023)
• HDT - Library for the Header Dictionary Triples (HDT) compression file format for RDF data.
• Compression algorithms are a super cool way to do a lot of very useful AI/ML, and can be more scalable (2023) (Lobsters)
• Gzip and KNN Outperforms Transformers on Text Classification (2023) (HN)
• Minify and Gzip (2022) (HN)
• LZAV - Fast In-Memory Data Compression Algorithm (In C). (HN)
• Data compression (2023)
• compress - Compress and uncompress for Deno.
• Compressor - Implementation of a modern compressor.