|
| 1 | + |
| 2 | +# FUIF: Free Universal Image Format |
| 3 | + |
| 4 | + |
| 5 | +FUIF is a new image format, combining ideas from JPEG, lossless WebP, and FLIF. |
| 6 | +It is a 'universal' image format in several senses: |
| 7 | + |
| 8 | +1. it works well for any kind of image (photographic and non-photographic) |
| 9 | +2. it can do both lossless and lossy (with the same underlying algorithm) |
| 10 | +3. it is Responsive By Design: one file can be used, instead of needing |
| 11 | + separate images for the low-quality image placeholder, thumbnail, preview, dpr 1, dpr 2, etc |
| 12 | +4. it is backwards compatible with JPEG, in the sense that existing JPEG images can be |
| 13 | + transcoded to FUIF losslessly (no generation loss) and effectively (smaller files) |
| 14 | +5. it can achieve various trade-offs between encode/decode complexity and compression density |
| 15 | + |
| 16 | + |
| 17 | +In more detail: |
| 18 | + |
| 19 | +## 1. Any kind of image |
| 20 | + |
| 21 | +FUIF supports several methods for image preprocessing and entropy coding, so it can use |
| 22 | +the methods that work best for a particular image (or a particular part of an image). |
| 23 | + |
| 24 | +For non-photographic images with few colors, a palette (color index) can be used. |
| 25 | +There is no limit on the palette size. |
| 26 | + |
| 27 | +For images with repetition (e.g. images that include text, where the same letter shapes |
| 28 | +appear in multiple locations), an optional transformation can be used to replace repetition |
| 29 | +with references (somewhat similar to JBIG2). |
| 30 | + |
| 31 | +For photographic images, the DCT transformation can be used. |
| 32 | + |
| 33 | +For raw sensor data, the format supports an arbitrary number of channels, that do not need |
| 34 | +to have the same dimensions, and that do not need to have the same value ranges either. |
| 35 | +The actual ranges can be used effectively (not just 8-bit or 16-bit like in PNG, but also |
| 36 | +things like 14-bit with a black level and white level range of 1023..15600). |
| 37 | + |
| 38 | +FUIF supports bit depths up to 28-bit per channel, unsigned or signed. |
| 39 | +Only integer values are supported (no floating point), but in principle, floating point |
| 40 | +numbers can be represented as integers plus a suitable transfer function. |
| 41 | + |
| 42 | + |
| 43 | +## 2. Lossless and lossy |
| 44 | + |
| 45 | +FUIF uses reversible transforms (YCoCg, reversible Haar-like squeezing); |
| 46 | +optional quantization is the only source of loss. |
| 47 | + |
| 48 | +Unlike FLIF, FUIF was designed with lossy compression in mind. |
| 49 | +Unlike JPEG, FUIF can obtain fully lossless compression. |
| 50 | +Unlike WebP and JPEG 2000, FUIF uses the same algorithm for both lossy and lossless. |
| 51 | + |
| 52 | +FUIF attempts to offer state-of-the-art compression for both lossy and lossless compression. |
| 53 | + |
| 54 | + |
| 55 | +## 3. Responsive By Design |
| 56 | + |
| 57 | + |
| 58 | +The FUIF bitstream is designed in such a way that truncating a file results in a |
| 59 | +lower resolution downscaled image. It is similar to progressive JPEG or JPEG 2000 |
| 60 | +in this respect. Image formats (like WebP, HEIC/BPG, AVIF) that are derived from a video codec |
| 61 | +(VP8, HEVC, AV1) intra-frame encoding, do not support progressive decoding, since |
| 62 | +in a video codec it does not make much sense to decode a single frame progressively. |
| 63 | + |
| 64 | +FUIF was designed specifically with web delivery in mind. Instead of having to produce, |
| 65 | +store and deliver many variants of an image, scaled to various dimensions, the idea is |
| 66 | +to use a single file and truncate it depending on the dimensions it gets rendered on. |
| 67 | +This approach has several advantages: |
| 68 | + |
| 69 | +* Less storage needed (no redundancy) |
| 70 | +* Better for CDNs (cache hit is more likely) |
| 71 | +* Less bandwidth needed (e.g. if you are currently sending a LQIP, then a thumbnail, then when it gets clicked a larger image) |
| 72 | +* Smart browsers could make choices themselves, e.g. load lower resolution when on a slow or expensive connection; when the local browser cache |
| 73 | + gets too full, instead of deleting whole files, it could trim off bytes at the end, keeping a preview in cache |
| 74 | + |
| 75 | +The header of a FUIF file contains a mandatory list of truncation offsets, which makes |
| 76 | +it easy to know how many bytes should be requested or served. The following offsets are |
| 77 | +included in the header: |
| 78 | + |
| 79 | +* LQIP: the first very low-quality preview of an image (typically at 100-200 bytes) |
| 80 | +* scale 1/16 |
| 81 | +* scale 1/8 |
| 82 | +* scale 1/4 |
| 83 | +* scale 1/2 |
| 84 | + |
| 85 | +These power-of-two downscales are exact, in the sense that if you truncate the file at the given offset, |
| 86 | +the resulting image is the same as decoding the whole full-resolution image and then downscaling it. |
| 87 | +This cannot really be done in an efficient way with progressive JPEG. |
| 88 | + |
| 89 | +FUIF has a minimalistic, compact header layout, so the first bits of actual image data appear as |
| 90 | +early as possible. This makes it possible to get a LQIP within a small byte budget, while it is still |
| 91 | +the beginning of the actual full image, so you also get the actual image dimensions, truncation offsets |
| 92 | +etc. |
| 93 | + |
| 94 | + |
| 95 | +## 4. Backwards compatible with JPEG |
| 96 | + |
| 97 | +There are a LOT of existing JPEG images out there, as well as devices that produce new JPEG images. |
| 98 | +If the only way to transcode a JPEG image to a new image format, is to decode the JPEG to pixels and |
| 99 | +encode that to the new format, then there is a problem. Either you get significant generation loss, |
| 100 | +or you get new files that are actually larger than the original. |
| 101 | + |
| 102 | +FUIF supports the 8x8 DCT transform (which is inherently lossy due to rounding errors) and the |
| 103 | +YCbCr color transform (which is also inherently lossy for the same reason). This means it can |
| 104 | +losslessly represent the actual information in a JPEG image, while still adding most of the benefits |
| 105 | +of FUIF: |
| 106 | + |
| 107 | +* LQIP and scale 1/16 (progressive JPEG starts at scale 1/8) |
| 108 | +* Minimal header overhead |
| 109 | +* Better compression |
| 110 | + |
| 111 | +Comparing FUIF to [Dropbox's Lepton](https://github.com/dropbox/lepton), which also does lossless JPEG recompression: |
| 112 | + |
| 113 | +* Lepton can obtain slightly better compression density |
| 114 | +* Lepton is faster |
| 115 | +* Lepton is bit-exact (not just image-exact) |
| 116 | +* FUIF can be decoded progressively (Lepton is anti-progressive since it encodes the AC before the DC) |
| 117 | +* FUIF can do more than just what JPEG can, for example you can add an alpha channel to an existing JPEG |
| 118 | + |
| 119 | + |
| 120 | +## 5. Trade-offs between complexity and compression density |
| 121 | + |
| 122 | +The entropy coding of FUIF is based on MANIAC (just like FLIF). |
| 123 | +Different trade-offs between computational complexity and compression density can be obtained. |
| 124 | +Using predefined or restricted MANIAC trees (or even no trees at all), encoding can be made |
| 125 | +faster; if encode time is not an issue, then there are many ways to optimize the encoding. |
| 126 | + |
| 127 | +In principle, subsets of the formats ("profiles") could be defined |
| 128 | +(e.g. by using fixed or restricted transformations and MANIAC trees) |
| 129 | +for which both encoding and decoding can be specialized (or done in hardware), |
| 130 | +making it significantly faster. |
| 131 | +These subsets are not currently defined nor implemented. |
| 132 | + |
| 133 | +FUIF is also well-suited for adaptive compression (i.e. having different qualities in |
| 134 | +different regions of the image). The MANIAC tree can represent an arbitrary segmentation |
| 135 | +of the image; there is no notion of (having to align with) macroblocks. This makes it easy |
| 136 | +to effectively use different context models for different regions of the image. |
| 137 | + |
| 138 | + |
| 139 | +## FUIF Design Principles |
| 140 | + |
| 141 | +The following goals or principles have guided the design of FUIF: |
| 142 | + |
| 143 | +* FUIF is an __image format, not a container__. FUIF stores pixels and the metadata needed to |
| 144 | +render the image, but that's it. Anything else, like Exif metadata, comments, |
| 145 | +rotation, crop coordinates, layers, tiling, image sequences, and so on is a task |
| 146 | +for the container format (e.g. HEIF), not for the image format. |
| 147 | +FUIF does support simple 'filmstrip' animation, so it can be used to recompress GIF and APNG, |
| 148 | +but it is not a video codec. |
| 149 | +_Rationale: this is a matter of separation of concerns, as well as avoiding duplication of functionality. |
| 150 | +For simple animations, FUIF might be suitable, but in general we think that most animations |
| 151 | +should be encoded as a short (looping) video |
| 152 | +(taking advantage of inter-frame prediction, which is out of the scope of an image format)._ |
| 153 | + |
| 154 | +* FUIF is optimized for __delivery__, not storage. It is progressive / "responsive by design". One single file |
| 155 | +can be used instead of having to downscale a high-resolution original to various target resolutions. |
| 156 | +The bitstream does not require seeking and a meaningful placeholder/preview can be shown from the first few |
| 157 | +hundred bytes. Perhaps it is possible to achieve better compression density and/or faster encoding by not |
| 158 | +optimizing for delivery (e.g. by predicting DC from AC like Lepton does). |
| 159 | +If the use case is (essentially) write-only archival then maybe it is worth it to sacrifice progressive decoding, |
| 160 | +and FUIF can in principle be used in such a way, but that's not the primary target. |
| 161 | +_Rationale: the variety of display devices (from smart watches to huge 8K screens) and network conditions |
| 162 | +(from 2G on the road in rural areas to very high speed at home) is a huge challenge. |
| 163 | +Making many files for the same image is not the best approach, e.g. in terms of CDN cache hits. |
| 164 | +Moreover browsers get few opportunities to be smart and adjust the image resolution and quality |
| 165 | +automatically based on the device and network conditions. Non-progressive image formats result |
| 166 | +in a bad user experience when bandwidth is an issue._ |
| 167 | + |
| 168 | +* __'Honest' compression artifacts__. |
| 169 | +There are various approaches to how to do lossy compression. Different techniques lead to different artifacts. |
| 170 | +Some artifacts are less "annoying" than other artifacts. For example, blurring, smearing and mild ringing are probably |
| 171 | +not very annoying (or even desireable to some, because it might eliminate noise and increase perceived sharpness), |
| 172 | +while pixelation, blockiness and color banding are annoying and obvious compression artifacts. |
| 173 | +Also, some artifacts are not very "honest", in the sense that the image looks deceptively better than |
| 174 | +it actually is. For example, JBIG2 in lossy mode or HEIC at low bitrates can produce images that look like they |
| 175 | +are high-quality (e.g. they have sharp details at the pixel level), |
| 176 | +but they are actually very different from the uncompressed image. |
| 177 | +For example, JPEG artifacts are "honest" and "annoying", while WebP and HEIC artifacts are "not honest" and "not annoying". |
| 178 | +FUIF aims for compression artifacts that are "honest" and "not annoying". At low bitrates, pixelation will become |
| 179 | +obvious at a 1:1 scale, but the overall image fidelity will still be as high as possible |
| 180 | +(e.g. comparing a downscaled lossy FUIF image to a downscaled original). |
| 181 | +_Rationale: this is a matter of preference, but we think that image fidelity is more important than hiding the fact |
| 182 | +that lossy compression was used. An image format should not act as an artistic filter that modifies an image more than |
| 183 | +necessary. At least that's our opinion._ |
| 184 | + |
| 185 | +* FUIF is royalty __free__ and it has a reference implementation that is free and open source software. |
| 186 | +_Rationale: we don't want no patent mess, please._ |
| 187 | + |
| 188 | +* FUIF is __legacy-friendly__ but not backwards compatible. |
| 189 | +Unlike JPEG XT, the FUIF bitstream is completely different from legacy JPEG, so it is not backwards compatible |
| 190 | +(cannot be read by existing JPEG decoders). But unlike most other recent image formats (e.g. WebP, HEIC, AVIF), |
| 191 | +existing JPEG images can be converted losslessly and effectively to FUIF, |
| 192 | +potentially with added information like alpha or depth channels. |
| 193 | +_Rationale: backwards compatibility is obviously convenient, but it has serious downsides. |
| 194 | +It implies that the compression density cannot be improved compared to legacy JPEG. |
| 195 | +It also means that none of the new features (like e.g. alpha) are guaranteed to be rendered correctly, |
| 196 | +since (at least initially) most decoders will ignore any extensions of the legacy JPEG bitstream. |
| 197 | +So for these reasons, not being backwards compatible is a better choice. |
| 198 | +However it is nice to be able to transcode existing JPEG images to FUIF without generation loss and while saving bytes._ |
| 199 | + |
| 200 | + |
| 201 | +## TL;DR feature summary |
| 202 | + |
| 203 | +* Lossless and lossy, works well for photographic and non-photographic |
| 204 | +* Arbitrary number of channels (grayscale, RGB, CMYK, RGBA, CMYKA, CMYKA+depth, etc) |
| 205 | +* Bit depth up to 28-bit, arbitrary actual ranges of pixel values |
| 206 | +* Progressive/Responsive by design: single-file responsive images |
| 207 | +* Minimal header overhead, so LQIP in say a 200-byte budget is possible |
| 208 | +* Backwards compatible with JPEG (can losslessly and efficiently represent legacy JPEG files) |
| 209 | +* Adjustable trade-off between encode/decode complexity and compression density |
0 commit comments