The C library ============= The reader and writer live in a C++ core, ``pvzstd``, and the whole of its public surface is a C ABI declared in a single header, ``cpp/include/pvzstd/pvzstd.h``. The Python package is one consumer of that ABI, bound with :mod:`ctypes`; nothing in the ABI is Python-specific, and a C or C++ consumer can link the same library directly. Two properties are what make that worth doing: * **The interface is pure C.** No C++ type, no zstd type, and no VTK type appears in the header, so it binds from C, from C++, from ``ctypes``, or from any FFI that can call a C function. Turning a dataset into arrays stays with the caller, which is what keeps VTK out of the library and lets it build as WebAssembly. * **The only dependency is zstd**, and it is linked privately. The header never names a zstd type, so for the default shared build zstd is an implementation detail of ``libpvzstd`` rather than something a consumer has to find. (See :ref:`c_api_static_caveat` for the one case where that does not hold.) The on-disk format the library reads and writes is specified in ``doc/format/container-v2.md``. Building -------- The core is an ordinary CMake project rooted at ``cpp/``: .. code:: bash cmake -S cpp -B build -DCMAKE_BUILD_TYPE=Release cmake --build build -j cmake --install build --prefix /where/you/want/it The install tree carries the library, the header under ``include/pvzstd/``, and a CMake package under ``lib/cmake/pvzstd/``. Options ~~~~~~~ .. list-table:: :header-rows: 1 :widths: 26 12 62 * - Option - Default - Meaning * - ``PVZSTD_BUILD_SHARED`` - ``ON`` - Build a shared library. The Python ``ctypes`` binding needs one. A static build is possible but exports zstd resolution to the consumer; see :ref:`c_api_static_caveat`. * - ``PVZSTD_BUILD_TOOLS`` - ``ON`` - Build the command-line tools ``pvzstd_dump``, ``pvzstd_rewrite``, ``pvzstd_append`` and ``pvzstd_stream``. They are conformance and inspection aids, not part of the ABI; turn them off for a build that only ships the library. * - ``PVZSTD_THREADS`` - ``ON`` - Use threads for batch decompression and for zstd's own workers. Turn it off for a target with no thread runtime. Reads are unaffected. Writes lose byte-identity with the reference writer above its 2 MiB threshold, and say so: a zstd built without multithreading rejects a non-zero worker count, so ``pvzstd_writer_write`` returns ``PVZSTD_E_ZSTD`` rather than quietly emitting frames that would not match. * - ``PVZSTD_ZSTD_PROVIDER`` - ``auto`` - Where zstd comes from. ``auto`` prefers an installed zstd and falls back to the pinned source build; ``system`` requires an installed one and fails configuration if there is none; ``vendored`` always builds the pinned source, ignoring anything installed. * - ``PVZSTD_VENDOR_ZSTD`` - ``ON`` - Whether the pinned source build is available as a fallback at all. With it ``OFF`` and no zstd installed, configuration fails rather than downloading anything. ``vendored`` is not paranoia about versions in general: different zstd releases emit different, equally valid frames for the same input, so the conformance suite has to pin the same source the reference writer carries. Linking a system zstd instead was measured to fail the byte-identity tests on multi-frame data that platforms without a system zstd passed. A distribution build that cares more about receiving zstd's security updates than about byte-identity should prefer ``system``. The pinned source build is also what makes Windows and WebAssembly possible at all, since neither has a system zstd to find. Consuming it ------------ Both paths yield the same target name, ``pvzstd::pvzstd``, so the rest of a consumer's CMake does not have to know which one was taken. Against an install tree ~~~~~~~~~~~~~~~~~~~~~~~ .. code:: cmake find_package(pvzstd CONFIG REQUIRED) target_link_libraries(my_app PRIVATE pvzstd::pvzstd) Point ``CMAKE_PREFIX_PATH`` at the install prefix if it is not already on the search path. The package's version is the released version line (``find_package(pvzstd 0.3)`` means what a reader expects), and it is compatible within a major version. It is *not* the compatibility contract a consumer should rely on at runtime: that is ``PVZSTD_ABI_VERSION``, below. Against the source ~~~~~~~~~~~~~~~~~~ As a subdirectory -- a submodule, a vendored copy, or a sibling checkout: .. code:: cmake add_subdirectory(third_party/pyvista-zstd/cpp pvzstd) target_link_libraries(my_app PRIVATE pvzstd::pvzstd) or fetched at configure time: .. code:: cmake include(FetchContent) FetchContent_Declare(pvzstd GIT_REPOSITORY https://github.com/pyvista/pyvista-zstd.git GIT_TAG main SOURCE_SUBDIR cpp) FetchContent_MakeAvailable(pvzstd) target_link_libraries(my_app PRIVATE pvzstd::pvzstd) ``pvzstd::pvzstd`` is an ``ALIAS`` of the in-tree target in these two cases and an imported target in the ``find_package`` case, which is the point of having the alias: one link line serves all three. .. _c_api_static_caveat: The static build does not carry zstd ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ zstd is linked ``PRIVATE``. For the default shared build that folds zstd's object code into ``libpvzstd`` itself, so the installed library is self-contained whether zstd was vendored or found on the system. It does **not** hold for ``-DPVZSTD_BUILD_SHARED=OFF``. CMake does not fold a private static dependency's objects into the resulting archive, so ``libpvzstd.a`` still needs zstd's symbols resolved at the consumer's link step, and the exported package does not declare that dependency -- zstd is treated as this build's implementation detail, not a re-exported one. With a system zstd a consumer can supply it themselves. With ``PVZSTD_ZSTD_PROVIDER=vendored`` the vendored zstd happens to install its own findable package alongside this one, as a side effect of ``FetchContent`` adding it as a subdirectory rather than anything this project arranges, so it is not strictly unfindable -- but that is fragile and not worth designing around. Prefer the shared build for anything that will be installed and linked downstream. The ABI version --------------- .. code:: c #define PVZSTD_ABI_VERSION 11u PVZSTD_API uint32_t pvzstd_abi_version(void); The macro is the version the caller compiled against; the function is the version the library it actually loaded was built with. **Compare them for equality, not as a floor.** The number is bumped on any change to the header, additions included, because a caller that binds every symbol up front -- which is what an FFI binding does -- is broken by an addition just as surely as by a removal, if it looks for a symbol the loaded library does not have. .. code:: c if (pvzstd_abi_version() != PVZSTD_ABI_VERSION) { /* wrong library; do not call anything else */ } Calling the function is also the cheapest real proof that a library linked and loaded at all. A header parses whether or not anything is behind it, and CMake reporting a successful link says only that the symbols resolved at build time, not that the shared object the loader picks up at runtime is the right one -- or that it is present. ``pvzstd_abi_version()`` is a call into that object with no arguments and no failure mode, so a correct answer from it means the library is there, is loadable, and is the one the header describes. The Python binding does exactly this at import time, so that a bad install fails at import naming what it tried rather than at the first read. A second, separate number describes the *files* rather than the library: .. code:: c #define PVZSTD_FILE_VERSION_MAX 2u PVZSTD_API uint32_t pvzstd_max_file_version(void); This is the highest container ``file_version`` the build can decode. A container stamped higher is refused with ``PVZSTD_E_VERSION`` rather than read, because a newer format may transform payloads in a way this build cannot invert, and reading one would hand back plausible-looking corrupt values instead of failing. The ceiling lives beside the decoder it describes so that every caller of the ABI gets the same answer; a binding keeping its own copy would refuse files the library can read, or accept files it cannot. Reading a container ------------------- A reader is opened, queried, read from, and closed. Opening parses only the trailer and the array headers; payloads are decompressed on demand, one array at a time. .. code:: c pvzstd_status pvzstd_open(const char *path, pvzstd_reader **out); pvzstd_status pvzstd_open_versioned(const char *path, pvzstd_reader **out, uint32_t *file_version); pvzstd_status pvzstd_open_memory(const void *data, uint64_t size, pvzstd_reader **out); pvzstd_status pvzstd_open_memory_versioned(const void *data, uint64_t size, pvzstd_reader **out, uint32_t *file_version); void pvzstd_close(pvzstd_reader *reader); On success ``*out`` receives a reader that must be released with ``pvzstd_close``. On failure ``*out`` is set to ``NULL``. ``pvzstd_close`` accepts ``NULL``. The ``_versioned`` variants additionally report the container's own ``file_version``, **including when the open was refused for being too new**, which is precisely the case in which a caller needs the number -- to say what it found against ``pvzstd_max_file_version()``. The out-parameter may be ``NULL``, and is left untouched when the container carried no readable version. Opening from memory ~~~~~~~~~~~~~~~~~~~ ``pvzstd_open_memory`` takes ``size`` contiguous bytes of a whole container that the caller already holds: an archive member, an HTTP response body, or a build with no filesystem to open a path on. It refuses a crafted buffer wherever ``pvzstd_open`` would refuse a crafted file. **The bytes are borrowed, never copied.** The caller owns the buffer and must keep it allocated and unmodified until ``pvzstd_close``, which is what makes this cheaper than staging the container somewhere the path entry point can reach. ``pvzstd_close`` releases only what the reader itself acquired; the buffer is left alone and is the caller's to free afterwards. Modifying the buffer while a reader is open **is not detected**. Offsets and sizes were read at open time, so arrays read afterwards hold undefined contents and no status reports it. One difference between the two doors is deliberate. A ``NULL`` pointer, or a ``size`` of zero, is ``PVZSTD_E_INVALID``: a zero size is a bad argument. A zero-length *file* is a property of the thing being opened and so reaches ``pvzstd_open`` as ``PVZSTD_E_IO``. A caller writing one error handler over both entry points should expect that. Describing arrays ~~~~~~~~~~~~~~~~~ .. code:: c typedef struct pvzstd_array_info { const char *name; /* NUL-terminated, UTF-8, includes the UID prefix */ const uint64_t *shape; /* ndim entries; NULL when ndim == 0 */ uint32_t ndim; uint8_t filter_id; char dtype[PVZSTD_DTYPE_LEN + 1]; /* e.g. ".append.lock``, beside the container, created exclusively. A second append meanwhile returns ``PVZSTD_E_BUSY`` **immediately, rather than waiting.** It does not block, because a library has no business choosing how long its caller waits; retrying is the caller's decision, and a retry succeeds as soon as the other append finishes. Exclusive file creation is the primitive rather than ``flock`` because it is the one form of mutual exclusion every target this builds for implements the same way. On the WebAssembly target ``flock`` returns success without locking anything, so a second holder would be handed a lock the first one believes it has. .. warning:: **A killed append leaves its lock behind, and a human has to delete it.** The lock file is removed when the call returns, however it returns -- but a process that dies between taking it and finishing never gets to. Every later append to that container then returns ``PVZSTD_E_BUSY`` until the file is removed, and no retry clears it. Recovery is deleting ``.append.lock``. That is safe whenever no append is actually running against the container; the file carries no state, and the container itself was never touched by the append that died. The trade is deliberate: this is a visible, named, recoverable failure, where what it replaces was one writer's arrays disappearing with nothing reported. The lock binds appends and nothing else. A writer that does not take it -- another tool, or a plain ``mv`` onto the path -- is caught separately: each append stages into a file the operating system names, so no two callers are ever handed the same staging file, and before committing, a call checks that its path still names the container it read. One replaced during the staging returns ``PVZSTD_E_CHANGED`` having written nothing, and running it again picks up the other writer's result. That one is a check and not a lock: it covers the staging, which is the slow part, and not the microseconds between the check and the rename. Streams take no lock and are covered by none. ``pvzstd_stream_*`` writes into the container in place, so it has neither a commit point at which to notice another writer nor a staging file to hold back, and it returns neither ``PVZSTD_E_BUSY`` nor ``PVZSTD_E_CHANGED``. One stream at a time, and no append against the same container while a stream is open, is the caller's to arrange. On WebAssembly ^^^^^^^^^^^^^^ The lock is real on that target, not inert: exclusive creation works under both Emscripten's own filesystems and ``NODERAWFS``, and a stale lock refuses a later append there exactly as it does natively. What it cannot do is protect a consumer against itself. A single-threaded module with no other writer never contends, so the lock costs it two filesystem operations per append and nothing else; two module instances sharing one real file through ``NODERAWFS`` are two writers and are excluded properly. Two instances over separate in-memory filesystems are not sharing a container at all, so there is nothing to exclude -- and if such a build ever loses a lock file to a discarded filesystem, the container is discarded with it. A worked example ---------------- Open a container, read its first array, close. This is complete: it compiles as C, links against ``pvzstd::pvzstd``, and runs. .. code:: c #include #include #include int main(int argc, char **argv) { if (argc != 2) { fprintf(stderr, "usage: %s CONTAINER.pv\n", argv[0]); return 2; } /* Proves a library actually linked, and that it is the one this header describes: the two are compared for equality, not as a floor. */ if (pvzstd_abi_version() != PVZSTD_ABI_VERSION) { fprintf(stderr, "pvzstd ABI mismatch: built against %u, linked %u\n", (unsigned)PVZSTD_ABI_VERSION, pvzstd_abi_version()); return 1; } pvzstd_reader *reader = NULL; uint32_t file_version = 0; pvzstd_status status = pvzstd_open_versioned(argv[1], &reader, &file_version); if (status != PVZSTD_OK) { fprintf(stderr, "open: %s (container version %u, this build decodes %u)\n", pvzstd_status_message(status), file_version, pvzstd_max_file_version()); return 1; } if (pvzstd_array_count(reader) == 0) { fprintf(stderr, "container holds no arrays\n"); pvzstd_close(reader); return 1; } pvzstd_array_info info; status = pvzstd_array_info_at(reader, 0, &info); if (status != PVZSTD_OK) { fprintf(stderr, "info: %s\n", pvzstd_status_message(status)); pvzstd_close(reader); return 1; } void *dst = malloc((size_t)info.nbytes); if (dst == NULL && info.nbytes != 0) { pvzstd_close(reader); return 1; } status = pvzstd_read_array_at(reader, 0, dst, info.nbytes); if (status != PVZSTD_OK) { fprintf(stderr, "read: %s\n", pvzstd_status_message(status)); free(dst); pvzstd_close(reader); return 1; } printf("%s: dtype %s, ndim %u, %llu bytes\n", info.name, info.dtype, (unsigned)info.ndim, (unsigned long long)info.nbytes); free(dst); /* the destination is the caller's */ pvzstd_close(reader); /* info.name and info.shape die here */ return 0; } with .. code:: cmake cmake_minimum_required(VERSION 3.16) project(pvzstd_example LANGUAGES C) find_package(pvzstd CONFIG REQUIRED) add_executable(read_one read_one.c) target_link_libraries(read_one PRIVATE pvzstd::pvzstd) Against a container written by ``pyvista_zstd.write``, it prints something like:: 00007c31d6209960points: dtype