Zstandard for WASI

v1.5.7WASI

Zstandard 1.5.7 for command-line programs under wasmtime, precompiled for wasm32-wasip3, single-threaded as @crossbind/port-zstd-wasi.

npm install @crossbind/port-zstd-wasi@beta

npm ↗Port recipe and licence files ↗

Install

shell

npm install @crossbind/port-zstd-wasi@beta crossbind@beta

A zstd command-line tool

WASI has no JavaScript bindings: the program is a main() that links the library, built into one .wasm and run with wasmtime. This one streams files through zstd in 128 KB steps.

crossbind.config.js

import zstdWasi from '@crossbind/port-zstd-wasi/crossbind.config.js';
 
export default {
    general: { name: 'zstd-tool' },
    dependencies: [zstdWasi],
    paths: { config: import.meta.url },
};

src/native/main.cpp

// A command-line zstd for WASI:
//   zstd-tool compress <input> <output> [level]
//   zstd-tool decompress <input> <output>
#include <zstd.h>
 
#include <cstdio>
#include <cstdlib>
#include <cstring>
#include <memory>
#include <vector>
 
namespace {
 
using File = std::unique_ptr<FILE, int (*)(FILE*)>;
 
File open(const char* path, const char* mode) {
    File file(std::fopen(path, mode), std::fclose);
    if (!file) std::fprintf(stderr, "cannot open %s\n", path);
    return file;
}
 
bool failed(size_t code) {
    if (!ZSTD_isError(code)) return false;
    std::fprintf(stderr, "zstd: %s\n", ZSTD_getErrorName(code));
    return true;
}
 
long long compress(FILE* in, FILE* out, int level) {
    std::unique_ptr<ZSTD_CCtx, size_t (*)(ZSTD_CCtx*)> cctx(ZSTD_createCCtx(), ZSTD_freeCCtx);
    if (failed(ZSTD_CCtx_setParameter(cctx.get(), ZSTD_c_compressionLevel, level))) return -1;
    if (failed(ZSTD_CCtx_setParameter(cctx.get(), ZSTD_c_checksumFlag, 1))) return -1;
    std::vector<char> input(ZSTD_CStreamInSize());
    std::vector<char> output(ZSTD_CStreamOutSize());
    long long written = 0;
    for (;;) {
        const size_t read = std::fread(input.data(), 1, input.size(), in);
        const bool last = read < input.size();
        ZSTD_inBuffer source = {input.data(), read, 0};
        bool finished = false;
        while (!finished) {
            ZSTD_outBuffer target = {output.data(), output.size(), 0};
            const size_t remaining = ZSTD_compressStream2(cctx.get(), &target, &source, last ? ZSTD_e_end : ZSTD_e_continue);
            if (failed(remaining)) return -1;
            written += static_cast<long long>(std::fwrite(output.data(), 1, target.pos, out));
            finished = last ? remaining == 0 : source.pos == source.size;
        }
        if (last) return written;
    }
}
 
long long decompress(FILE* in, FILE* out) {
    std::unique_ptr<ZSTD_DCtx, size_t (*)(ZSTD_DCtx*)> dctx(ZSTD_createDCtx(), ZSTD_freeDCtx);
    std::vector<char> input(ZSTD_DStreamInSize());
    std::vector<char> output(ZSTD_DStreamOutSize());
    long long written = 0;
    size_t pending = 0;
    size_t read = 0;
    while ((read = std::fread(input.data(), 1, input.size(), in)) > 0) {
        ZSTD_inBuffer source = {input.data(), read, 0};
        while (source.pos < source.size) {
            ZSTD_outBuffer target = {output.data(), output.size(), 0};
            pending = ZSTD_decompressStream(dctx.get(), &target, &source);
            if (failed(pending)) return -1;
            written += static_cast<long long>(std::fwrite(output.data(), 1, target.pos, out));
        }
    }
    if (pending != 0) {
        std::fprintf(stderr, "the input ends in the middle of a zstd frame\n");
        return -1;
    }
    return written;
}
 
}  // namespace
 
int main(int argc, char** argv) {
    const bool packing = argc >= 4 && std::strcmp(argv[1], "compress") == 0;
    const bool unpacking = argc >= 4 && std::strcmp(argv[1], "decompress") == 0;
    if (!packing && !unpacking) {
        std::fprintf(stderr, "usage: zstd-tool compress <input> <output> [level]\n       zstd-tool decompress <input> <output>\n");
        return 2;
    }
    File in = open(argv[2], "rb");
    File out = open(argv[3], "wb");
    if (!in || !out) return 1;
    const long long written = packing ? compress(in.get(), out.get(), argc >= 5 ? std::atoi(argv[4]) : 3) : decompress(in.get(), out.get());
    if (written < 0) return 1;
    std::printf("zstd %s %s: %s -> %s, %lld B written\n", ZSTD_versionString(), argv[1], argv[2], argv[3], written);
    return 0;
}

shell

npx crossbind build -p wasi -b release
wasmtime run --dir=. .crossbind/build/zstd-tool-wasi-wasm32-st-release.wasm compress access.log access.log.zst 3
wasmtime run --dir=. .crossbind/build/zstd-tool-wasi-wasm32-st-release.wasm decompress access.log.zst access.copy.log

output

zstd 1.5.7 compress: access.log -> access.log.zst, 378699 B written
zstd 1.5.7 decompress: access.log.zst -> access.copy.log, 2537578 B written

What is different on WASI

There are no JavaScript bindings: src/native provides main(int, char**), and the build is a single .wasm.
Files come from the host through --dir preopens; --dir=. gives the program the current directory.
crossbind build -p wasi -e wasi -b release writes the program to .crossbind/build/<name>-wasi-wasm32-st-release.wasm.
WASI 0.3's wasi:cli/exit carries success or failure only, so any non-zero return from main reaches the shell as exit code 1.
The build runs in the crossbind Docker image, or under RUNNER=LOCAL with wasi-sdk 34 or newer (WASI_SDK_PATH); running needs wasmtime 47 or newer. See WASI.

Other platforms

Zstandard overview: the apps, every platform's setup and the packages.
Zstandard for WebAssembly: browsers, Node.js and edge runtimes.
Zstandard for Android: React Native apps on Android.
Zstandard for iOS: React Native apps on iOS.
Zstandard for macOS: native Node.js addons and Electron on macOS.
Zstandard for Linux: native Node.js addons on Linux.
Zstandard for Windows: native Node.js addons on Windows.

Facts on this page come from the port manifests in the repository and from what npm served on beta when the site was built. See the Libraries guide for the full consumer flow.

MORE LIBRARIES

cURL Expat GDAL GEOS GeoTIFF iconv LERC libjpeg-turbo libTIFF OpenSSL PROJ SpatiaLite SQLite WebP zlib