Installation¶

PyCyBase is a Cython extension package — it cannot be used with pip install -e . (editable mode breaks .pxd resolution for downstream Cython projects that cimport from it). Extensions must be compiled in-place before installation.

Quick Build (POSIX)¶

git clone https://github.com/BolunHan/PyCyBase.git
cd PyCyBase

# Build + install (recommended)
./build.sh -i

# Or via Makefile:
make build && pip install -U . --no-build-isolation

# Or step by step:
python setup.py build_ext --inplace --verbose --force
pip install -U . --no-build-isolation

Build Script Reference¶

`build.sh`¶

The primary build script for POSIX (Linux/macOS). Supports venv activation, clean/rebuild/all-clean modes, optional pip install, and compile-time macro introspection.

./build.sh [options]

Options:

`-v`	Path to virtual environment to activate before building
`-i`	`pip install .` after build
`-r`	Force-reinstall (uninstall + `pip install --force-reinstall`)
`-c`	Clean build artifacts only (no build)
`-a`	Deep clean — remove `.c` and `.so` files, then exit
`-l`	List all compile-time macros and their default values
`-h`	Show help

`Makefile`¶

Convenience targets wrapping build.sh:

Target	Effect
`build`	Clean + `build_ext --inplace --verbose --force`
`dev`	Alias for `build`
`install`	Build + `pip install .`
`reinstall`	Build + force-reinstall (uninstall first)
`clean`	Remove `build/`, `*.egg-info`, `includes/`
`clean-all`	`clean` + delete all `.c` and `.so` files
`list-args`	List compile-time macros (delegates to `build.sh -l`)

`build.ps1` (Windows)¶

PowerShell build script for Windows NT. Activates a specified venv, cleans artifacts, and runs build_ext --inplace --verbose --force.

.\build.ps1 -VenvPath "C:\Users\...\venv_313"

`nt_build.py` (Cross-Compile from Linux)¶

Paramiko-based orchestrator that syncs the local source tree to a Windows VM via SSH, then triggers build.ps1 and runs tests. Configured via nt_config.json.

python nt_build.py                # sync → build → test
python nt_build.py --host Win11-CN

Compile-Time Macros¶

PyCyBase exposes numerous #define macros that control allocation behaviour, page sizes, thread safety, and debugging. Override any macro at compile time by setting an environment variable of the same name:

DEBUG=1 ./build.sh            # Enable debug mode
AP_ALLOC_VIGILANT=0 make build # Disable vigilant/canary checks

Listing available macros:

./build.sh -l        # Reads macros.json; auto-generates via probe.py if missing
make list-args        # Same, via Makefile
python probe.py       # Run the probe manually

probe.py scans Cython .pxd files for cdef extern from headers, extracts every #define macro from those headers, and writes a JSON inventory to macros.json.

Key macros (see macros.json for the full list):

Macro	Default	Description
`AP_ALLOC_VIGILANT`	`1`	Enable bounds/canary validation
`AP_ALLOC_MAGIC`	`0xCFBBBBFCULL`	Magic sentinel for live allocations
`AP_DEALLOC_MAGIC`	`0xDEADDEADULL`	Magic sentinel for freed memory
`AP_DECREF_AUTOFREE`	`1`	Auto-free on refcount reaching zero
`AP_ALLOC_WITH_LOCK`	`1`	Enable pthread mutex locking
`AP_ALLOC_WITH_SHM`	`0`	Default to SHM (off = heap default)
`AP_ALLOC_WITH_FREELIST`	`1`	Enable free-list reuse
`AP_HEAP_AUTOPAGE_CAPACITY`	`64 KiB`	Default heap page size
`AP_HEAP_AUTOPAGE_CAPACITY_MAX`	`16 MiB`	Max heap page size
`AP_HEAP_AUTOPAGE_ALIGNMENT`	`4 KiB`	Heap page alignment
`AP_SHM_AUTOPAGE_CAPACITY`	`64 KiB`	Default SHM page size
`AP_SHM_AUTOPAGE_CAPACITY_MAX`	`16 MiB`	Max SHM page size
`AP_SHM_AUTOPAGE_ALIGNMENT`	`4 KiB`	SHM page alignment
`AP_SHM_ALLOCATOR_PREFIX`	`/c_cbase_shm`	SHM name prefix
`AP_SHM_ALLOCATOR_DEFAULT_REGION_SIZE`	`128 GiB`	Virtual region size
`AP_SHM_NAME_LEN`	`256`	Max SHM name length
`AP_SHM_PREFIX_MAX`	`64`	Max custom prefix length

Prerequisites¶

Python: 3.12 or later
Build: Cython ≥ 3.0, C11 compiler (GCC/Clang on Linux, MSVC on Windows)
Runtime: No external Python dependencies (stdlib only for the Python layer)
Docs (optional): sphinx + furo

Linux¶

# Ubuntu/Debian
sudo apt-get install build-essential python3-dev

# Arch/Manjaro
sudo pacman -S base-devel

Windows¶

Visual C++ Build Tools (download)
Select “Desktop development with C++” and Windows SDK during install

Verifying the Build¶

import cbase
print(cbase.__version__)

from cbase.allocator_protocol import AllocatorProtocol, AP_SHARED
from cbase.bytemap import ByteMap
from cbase.intern_string import POOL

Using `get_include()` in Downstream Projects¶

PyCyBase provides cbase.get_include() to help downstream Cython extensions find .pxd and .h files:

from setuptools import setup, Extension
from Cython.Build import cythonize
import cbase

ext = Extension(
    "my_module",
    sources=["my_module.pyx"],
    include_dirs=cbase.get_include(),
)
setup(ext_modules=cythonize([ext]))

Important

pip install -e . is not supported for PyCyBase. Editable installs break .pxd resolution because cimport searches Cython/Includes/ for the package name, not the editable egg-link. Always use setup.py build_ext --inplace followed by pip install -U . --no-build-isolation.

Platform-Specific Notes¶

SHM Allocator¶

POSIX (Linux/macOS): Full SharedMemoryAllocator with shm_open/mmap, cross-process pointer stability.
Windows: NtSharedMemoryAllocator compat layer (limited functionality, no cross-process sharing).

NB: the top-level cbase.allocator_protocol package selects the correct backend automatically based on sys.platform.

NT Cross-Compilation¶

For cross-compiling to Windows from a Linux host, use the NT build orchestrator:

pip install paramiko              # (in the host venv)
python nt_build.py                # sync → build → test on remote VM

Configuration is in nt_config.json. See the memory files for detailed VM connection and environment setup.

Troubleshooting¶

Compilation fails with missing Python.h

sudo apt-get install python3-dev

Cython version mismatch

pip install -U cython

Binary incompatibility after rebuild

If you see ValueError: ... size changed, may indicate binary incompatibility, do a deep clean and rebuild:

./build.sh -a            # Remove all .c and .so files
./build.sh -i            # Rebuild and install

Multiprocessing forkserver errors (Python 3.14 on Arch)

Set the multiprocessing start method before building:

export N_THREADS=0
python setup.py build_ext --inplace --verbose --force

Building Documentation¶

pip install sphinx furo
cd docs
sphinx-build -M html . _build

Open docs/_build/html/index.html in a browser.