Logo (Image): supports transparent background of sixel logos

src/3rdparty/sixel/README.md

@@ -1,30 +1,71 @@
 # libsixel (encoder subset, vendored)
 
 Encoder-only subset of [libsixel](https://github.com/saitoha/libsixel) `1.8.7-r2`,
-used by `src/logo/image/` to render the `sixel` logo type **on Windows only**.
+used by `src/logo/image/` to render the `sixel` logo type **on Windows and macOS**.
 
 ## Why this is vendored
 
-Neither MSYS2 nor vcpkg packages libsixel, so there is nothing to `dlopen()` or to
-link against. The encoder is small and has no external dependencies, so the
-necessary files are embedded instead. The intent is to upstream a libsixel package
-to MSYS2 later and switch to it; until then this directory is the source of truth.
+Neither MSYS2 nor vcpkg packages libsixel, so on Windows there is nothing to
+`dlopen()` or to link against. The encoder is small and has no external
+dependencies, so the necessary files are embedded instead. The intent is to upstream
+a libsixel package to MSYS2 later and switch to it; until then this directory is the
+source of truth.
 
-## Why Windows only
+macOS *does* have a package — Homebrew's `libsixel` is this very version, `1.8.7-r2`
+(`brew info libsixel`) — but it is deliberately not used. It drags in `jpeg-turbo` /
+`libpng` for the decoder side we never call, and linking it would mean a different
+encoder on each platform. Since Windows needs an embedded copy regardless, one shared
+copy is simpler than two.
+
+## Why only Windows and macOS
 
 Windows loses ImageMagick entirely — its image backend moves to WIC, which decodes
 and scales but cannot *encode* sixel — so it needs an encoder of its own.
 
+macOS does not get ImageMagick either: `ENABLE_IMAGEMAGICK6` / `ENABLE_IMAGEMAGICK7`
+are gated on `LINUX OR FreeBSD OR OpenBSD OR NetBSD OR ANDROID OR SunOS OR Haiku OR
+GNU`, with no `APPLE`, and its image backend is ImageIO. So it shares the same
+requirement.
+
 On every other platform ImageMagick stays, and it already produces sixel through
 its own SIXEL coder. Adding libsixel there would mean a second encoder for a
 capability that already works: one more dependency for zero gain. So `ENABLE_SIXEL`
-is gated on `WIN32`, and these sources are **not compiled at all** on Linux / macOS /
-BSD.
+is gated on `WIN32 OR APPLE`, and these sources are **not compiled at all** on
+Linux / the BSDs.
 
-Consequence: sixel bytes differ between Windows (libsixel) and the rest
+Consequence: sixel bytes differ between Windows / macOS (libsixel) and the rest
 (ImageMagick). That is deliberate and accepted — see
 `doc/windows-image-backend.md` decision 9.
 
+## Transparency
+
+The subset is used for its keycolor support. `sixel_dither_set_transparent()` marks a
+palette index as the transparent one; `output_rgb_palette_definition()` then leaves that
+entry out of the palette definition, and `sixel_encode_body()` never puts a keycolor
+pixel into the node map, so nothing is drawn there and the terminal background shows
+through. `src/logo/image/sixel.c` drives this for sources that contain fully transparent
+pixels — it maps those pixels onto a dedicated index one past the quantized palette,
+because keying a *content* colour would also drop every opaque pixel that shares it.
+
+Two things have to line up for the background to actually be transparent: the caller has to
+mark a keycolor, so the keyed pixels are never painted (above), **and** the encoder has to ask
+for a transparent background, so the terminal leaves those unpainted pixels alone instead of
+filling them. The second half is the DCS `P2` parameter — see
+[Local modifications](#local-modifications) item 3.
+
+Note what this is and is not: it is a **palette-index** keycolor, not alpha support. A
+partially transparent pixel has to be drawn in its unblended colour, since a sixel has
+no way to blend with whatever the terminal has behind it.
+
+⚠️ Do not vendor a *newer* libsixel in the belief that this version cannot do
+transparency. It can, and it was measured: the official `1.8.7-r2` omits the keycolor
+entry and leaves the keyed pixels unpainted, with the remaining pixels correctly placed.
+When a transparent background is not showing up, check the caller first — historically
+the caller simply never called `sixel_dither_set_transparent()`, and an image logo only
+renders correctly once it does. If the caller *is* setting a keycolor and the background is
+still solid, check `P2` instead: the keyed pixels are being left unpainted correctly, but the
+terminal is filling them.
+
 ## What was kept
 
 Starting from `sixel_dither_new` / `sixel_dither_initialize` / `sixel_output_new` /
@@ -48,8 +89,8 @@ Everything else in the upstream tree, in particular:
 
 - **Decoding**: `decoder.c`, `fromsixel.c`, `frompnm.c`, `fromgif.c`, `frame.c`,
   `loader.c`, `stb_image.h`, `stb_image_write.*`. Image decoding is done by the
-  platform backend (WIC on Windows, ImageMagick elsewhere), so libsixel only ever
-  sees an RGBA8 buffer here.
+  platform backend (WIC on Windows, ImageIO on macOS, ImageMagick elsewhere), so
+  libsixel only ever sees an RGBA8 buffer here.
 - **The `sixel_encoder_*` / `sixel_decoder_*` high-level API**: `encoder.c`,
   `writer.c`, `tty.c`, `scale.c`. These pull in file I/O, terminal probing and the
   loader registry. We call the dither/output/encode trio directly.
@@ -59,20 +100,21 @@ Everything else in the upstream tree, in particular:
   **`tests/`**, **`images/`** (216 test images), and the packaging templates
   (`package.json.in.in`, `libsixel.pc.in`).
 - **The autotools machinery**: `configure`, `Makefile.in`, `aclocal.m4`, `m4/`,
-  `ltmain.sh`, `config.h.in`, … See `config.h` below.
+  `ltmain.sh`, `config.h.in`, … See item 2 under
+  [Local modifications](#local-modifications).
 - **Six of the nine `LICENSE.*` files**: `LICENSE.images` / `.mesa` / `.pngsuite` /
   `.sdump` / `.stb` cover only material that was dropped (test images, the OpenGL
   example, the `sdump` tool, stb). The three that remain are the complete set
   required by the files above — verified against each file's own header, see
   [License](#license).
 
-Result: **20 files / 255 KB**, down from 319 files / 11.3 MB. Note that 8.0 MB of
+Result: **20 files / 256 KB**, down from 319 files / 11.3 MB. Note that 8.0 MB of
 that 11.3 MB was `images/` fixtures, so the meaningful comparison is code:
-`src/` + `include/` went from 50 files / 1.17 MB to 20 files / 255 KB.
+`src/` + `include/` went from 50 files / 1.17 MB to 20 files / 256 KB.
 
 ## Local modifications
 
-Two deliberate divergences from upstream. Re-apply both when re-syncing.
+Three deliberate divergences from upstream. Re-apply all three when re-syncing.
 
 1. **`sixel.h` — `SIXELAPI` is empty.** Upstream defines it as
    `__declspec(dllexport)` on Windows. This subset is compiled into `libfastfetch`
@@ -92,6 +134,24 @@ Two deliberate divergences from upstream. Re-apply both when re-syncing.
      `-Werror=implicit-function-declaration`, so a missing `<memory.h>` /
      `<string.h>` include in `pixelformat.c` / `status.c` is a hard error.
 
+3. **`tosixel.c` — the DCS `P2` parameter is initialised to 1.** `sixel_encode_header()`
+   declares `int p[3] = {0, 0, 0}`, and the trailing-zero trimming right below it then drops
+   all three, so the envelope is a bare `ESC P q`. That means `P2 = 0`, which tells the
+   terminal to fill every unset pixel with **colour-table entry 0** — and that entry is
+   defined by the image's own palette, so a keycolor image comes out with a solid rectangle
+   behind it rather than transparency. Setting `p[1] = 1` leaves unset pixels untouched.
+
+   The trimming still drops the trailing aspect-ratio parameter, so the envelope is
+   `ESC P 0;1 q`. `P3` defaults to 0 and the two spellings are equivalent — verified by
+   decoding both with libsixel's own decoder and with ImageMagick's. ImageMagick's SIXEL
+   coder and chafa both write `0;1;0` explicitly.
+
+   This only shows up on a terminal that implements the VT340 background rule. Windows
+   Terminal does (`_backgroundFillRequired = (_conformanceLevel == 1 || !transparent)` in
+   `src/terminal/adapter/SixelParser.cpp`), which is where it was reproduced; WezTerm does
+   not fill, so the identical bytes look correct there. It is also why Linux never showed
+   the problem: that path goes through ImageMagick, which already sends `P2 = 1`.
+
 The vendored sources are also exempted from `-Wconversion` in `CMakeLists.txt`:
 upstream has ~160 implicit int→`unsigned char` narrowing warnings. Everything else
 (`-Wall -Wextra` and fastfetch's `-Werror=` set) applies unchanged, and the subset
@@ -111,14 +171,17 @@ done
 cp /tmp/libsixel/include/sixel.h .
 cp /tmp/libsixel/LICENSE /tmp/libsixel/LICENSE.sixel /tmp/libsixel/LICENSE.pnmcolormap .
 
-# 2. re-apply the two local modifications listed above
+# 2. re-apply the three local modifications listed above
 # 3. verify
 ```
 
-Verification (must be 0 diagnostics, and the encoder must emit a DCS-wrapped stream):
+Verification (must be 0 diagnostics, and the encoder must emit a DCS-wrapped stream).
+Both platforms compile this subset, so run it on both:
 
 ```sh
-CC=/c/msys64/clang64/bin/cc.exe
+# Windows: CC=/c/msys64/clang64/bin/cc.exe
+# macOS:   CC=cc   (verified with Apple clang 21: 0 diagnostics)
+CC=cc
 FLAGS="-I. -Wall -Wextra -Wconversion -Wno-conversion -Werror=uninitialized \
  -Werror=return-type -Werror=vla -Werror=incompatible-pointer-types \
  -Werror=implicit-function-declaration -Werror=int-conversion -std=gnu23"

src/3rdparty/sixel/repo.json

@@ -2,5 +2,10 @@
     "home": "https://github.com/saitoha/libsixel",
     "license": "MIT ( embed in source )",
     "version": "1.8.7-r2",
-    "author": "Hayaki Saito"
+    "author": "Hayaki Saito",
+    "modifications": [
+        "sixel.h: SIXELAPI is empty, so the statically linked subset exports nothing",
+        "config.h: hand-written, because only the vendored files read it",
+        "tosixel.c: DCS P2 initialised to 1, so unset pixels are not filled with colour-table entry 0"
+    ]
 }

src/3rdparty/sixel/tosixel.c

@@ -307,7 +307,15 @@ sixel_encode_header(int width, int height, sixel_output_t *output)
 {
     SIXELSTATUS status = SIXEL_FALSE;
     int nwrite;
-    int p[3] = {0, 0, 0};
+
+    /* fastfetch: P2 = 1 asks for a transparent background, so a pixel we leave unset keeps
+     * whatever the terminal already had there. Upstream leaves it at 0, which tells the terminal
+     * to fill unset pixels with colour-table entry 0 instead -- and that entry is defined by the
+     * image's own palette, so a keycolor image comes out with a solid rectangle behind it rather
+     * than transparency. Measured on Windows Terminal, which implements the VT340 semantics;
+     * ImageMagick's SIXEL coder emits "0;1;0" here and chafa emits the same.
+     * The trimming below drops the trailing zero, so the envelope is "0;1" -- P3 defaults to 0. */
+    int p[3] = {0, 1, 0};
     int pcount = 3;
     int use_raster_attributes = 1;
 

src/logo/image/sixel.c

@@ -2,16 +2,89 @@
 
 #ifdef FF_HAVE_SIXEL
 
-    #include <sixel.h> // src/3rdparty/sixel/sixel.h
+    #include <string.h>
+    #include <sixel.h>
+    // The transparent path needs two things the public header does not declare: the palette mapping
+    // -- which is where libsixel applies the Floyd-Steinberg dithering, so doing it by hand would
+    // band the gradients -- and the palette size the encoder's keycolor handling is built around.
+    // The vendored copy is pinned to the exact revision this build compiles, so there is no
+    // version skew to guard against.
+    #include "dither.h"
 
 static int sixelWriteCallback(char* data, int size, void* priv) {
     ffStrbufAppendNS((FFstrbuf*) priv, (uint32_t) size, data);
     return 1; // non-zero means "keep going"
 }
 
+// Only a fully transparent pixel is left out. A partially transparent one can not be represented --
+// a sixel has no way to blend it with whatever the terminal has behind it -- and painting it in its
+// unblended colour is closer to the source than dropping it would be.
+static bool hasTransparentPixels(const FFImageBuffer* buffer) {
+    const uint8_t* pixels = buffer->data;
+    const uint32_t pixelCount = buffer->width * buffer->height;
+    for (uint32_t i = 0; i < pixelCount; ++i) {
+        if (pixels[i * 4 + 3] == 0) {
+            return true;
+        }
+    }
+
+    return false;
+}
+
+// Encodes an image with transparent pixels as an indexed image whose transparent pixels carry a
+// keycolor: that entry is left out of the palette definition and its pixels are left out of the
+// pixel data, so nothing is drawn there and the terminal background shows through.
+static SIXELSTATUS sixelEncodeTransparent(sixel_dither_t* dither, const FFImageBuffer* buffer, sixel_output_t* output) {
+    const int width = (int) buffer->width;
+    const int height = (int) buffer->height;
+
+    // Map the pixels onto the palette through libsixel's own lookup, then hand the result back as
+    // an indexed image. This is the same mapping the opaque path gets from sixel_encode.
+    sixel_index_t* indices = sixel_dither_apply_palette(dither, buffer->data, width, height);
+    if (indices == nullptr) {
+        return SIXEL_RUNTIME_ERROR;
+    }
+
+    // The keycolor is the first index past the quantized palette, so it is never a colour that a
+    // pixel was mapped to. Asking for one colour less than the maximum is what guarantees the
+    // index exists.
+    const int keycolor = sixel_dither_get_num_of_palette_colors(dither);
+    const uint8_t* pixels = buffer->data;
+    for (uint32_t i = 0; i < (uint32_t) width * (uint32_t) height; ++i) {
+        if (pixels[i * 4 + 3] == 0) {
+            indices[i] = (sixel_index_t) keycolor;
+        }
+    }
+
+    // The encoder sizes its per-colour map from ncolors and skips the keycolor inside that range,
+    // so the keycolor has to be counted in it. Its own palette entry is never read: leaving the
+    // definition out is exactly what the keycolor means.
+    int ncolors = keycolor + 1;
+    if (ncolors < 3) {
+        // Two colours mean "the terminal's default black and white palette, no definitions needed"
+        // to the encoder, which would be wrong here: one of the two is the keycolor, not a colour.
+        // The padding entry is written out so no uninitialised palette byte reaches the output.
+        ncolors = 3;
+        memset(dither->palette + keycolor * 3, 0, (size_t) (ncolors - keycolor) * 3);
+    }
+    dither->ncolors = ncolors;
+
+    sixel_dither_set_transparent(dither, keycolor);
+    // PAL8 hands the indices over as they are, instead of mapping the pixels a second time
+    sixel_dither_set_pixelformat(dither, SIXEL_PIXELFORMAT_PAL8);
+
+    SIXELSTATUS status = sixel_encode(indices, width, height, 1, dither, output);
+
+    sixel_allocator_free(dither->allocator, indices);
+    return status;
+}
+
 bool ffSixelEncode(const FFImageBuffer* buffer, FFstrbuf* result, const char** error) {
+    // A source without fully transparent pixels takes the opaque path, unchanged
+    const bool transparent = hasTransparentPixels(buffer);
+
     sixel_dither_t* dither = nullptr;
-    if (sixel_dither_new(&dither, 256, nullptr) != SIXEL_OK) {
+    if (sixel_dither_new(&dither, transparent ? SIXEL_PALETTE_MAX - 1 : SIXEL_PALETTE_MAX, nullptr) != SIXEL_OK) {
         if (error) {
             *error = "sixel_dither_new() failed";
         }
@@ -37,7 +110,9 @@ bool ffSixelEncode(const FFImageBuffer* buffer, FFstrbuf* result, const char** e
     }
 
     // The depth parameter is unused by libsixel; the DCS envelope is emitted by default
-    SIXELSTATUS status = sixel_encode(buffer->data, (int) buffer->width, (int) buffer->height, 4, dither, output);
+    SIXELSTATUS status = transparent
+        ? sixelEncodeTransparent(dither, buffer, output)
+        : sixel_encode(buffer->data, (int) buffer->width, (int) buffer->height, 4, dither, output);
 
     sixel_output_unref(output);
     sixel_dither_unref(dither);