Intern String¶

Module: cbase.intern_string

A thread-safe string interning system backed by the allocator protocol. Strings are stored as NUL-terminated UTF-8 in memory, with FNV-1a hashing and cached hash values for fast comparisons.

Key benefits over Python str:

String deduplication — same pointer for the same key
Stable C pointer — const char* usable as a hash key
Cached 64-bit FNV-1a hash — no re-computation on hash()
Cross-process sharing — SHM-backed pools are visible after fork()

Benchmarks show 1.7–2.2× speedup over Python str creation with typical miss rates (1/1,000 to 1/1,000,000).

class cbase.intern_string.c_intern_string.InternString¶

Bases: object

__reduce_cython__(self)¶

__setstate_cython__(self, __pyx_state)¶

address¶

Type:: str | None

hash_value¶

Type:: int

pool¶

Type:: InternStringPool

string¶

Type:: str

class cbase.intern_string.c_intern_string.InternStringPool¶

Bases: object

__init__(*args, **kwargs)¶

address¶

Type:: str | None

internalized(self)¶

Return type:: Generator[InternString, None, None]

istr(self, str string) → InternString¶

Parameters:: string (str)
Return type:: InternString

size¶

Type:: int

class cbase.intern_string.c_intern_string.IstrTestToolkit¶

Bases: object

Performance benchmark toolkit for InternString vs Python str.

Generates a large shared character buffer, segments it into random-length strings, then benchmarks C-level hash / intern / lookup / equality against equivalent Python str operations.

Parameters:

buf_size (size_t) – Size of the character buffer in bytes (default 2^30 ≈ 1 GiB).
n_seg (size_t) – Number of segments to generate (default 100_000).
max_seg_len (size_t) – Maximum length of each random segment in bytes (default 64).
n_iters (size_t) – Number of benchmark iterations (default 10).

buf_size¶

Type:: int

istr_eq_routine(self) → double¶

Return type:: float

istr_hash_routine(self) → double¶

Return type:: float

istr_intern_routine(self) → double¶

Return type:: float

istr_intern_synced_routine(self) → double¶

Return type:: float

istr_limited_pool_routine(self) → double¶

Repeatedly intern from a limited pool of n_unique strings.

Each iteration picks n_ops random strings (with replacement) from the first n_unique segments and interns them. Since the pool is small relative to n_ops, most operations are hits (already-interned keys) — this mirrors real-world usage where a finite set of strings (e.g. ticker symbols) is interned repeatedly.

Return type:: float

istr_limited_pool_synced_routine(self) → double¶

Same as istr_limited_pool_routine but with c_istr_synced.

Return type:: float

istr_lookup_routine(self) → double¶

Return type:: float

istr_lookup_synced_routine(self) → double¶

Return type:: float

istr_miss_rate_routine(self, double miss_rate) → double¶

Benchmark intern with a controlled miss rate.

miss_rate=0.001 → 1/1K, miss_rate=0.0001 → 1/10K, etc.

Parameters:: miss_rate (float)
Return type:: float

max_seg_len¶

Type:: int

n_iters¶

Type:: int

n_ops¶

Type:: int

n_seg¶

Type:: int

n_unique¶

Type:: int

pool¶

Type:: InternStringPool

py_eq_routine(self) → double¶

Return type:: float

py_hash_routine(self) → double¶

Return type:: float

py_unicode_limited_routine(self) → double¶

Create Python str objects from random limited-pool picks.

Return type:: float

py_unicode_miss_rate_routine(self, double miss_rate) → double¶

Python str creation with the same miss-rate pattern.

Parameters:: miss_rate (float)
Return type:: float

py_unicode_routine(self) → double¶

Return type:: float

run_test(self) → dict¶

Return type:: dict[str, Any]

cbase.intern_string.c_intern_string.__reduce_cython__(self)¶

cbase.intern_string.c_intern_string.__setstate_cython__(self, __pyx_state)¶

Module Singletons¶

from cbase.intern_string import POOL, INTRA_POOL, C_POOL, C_INTRA_POOL

POOL — SHM-backed pool for cross-process sharing. After fork(), the child process sees all entries regardless of when they were interned.
INTRA_POOL — Heap-backed pool for intra-process use. After fork(), the child sees a COW copy of pre-fork entries, but post-fork insertions are isolated to each process.
C_POOL / C_INTRA_POOL — Raw uintptr_t pointers to the underlying C istr_map structures.

Usage Examples¶

Basic interning:

from cbase.intern_string import POOL

# Intern a string (returns an InternString view)
aapl = POOL.istr("AAPL")
tsla = POOL.istr("TSLA")

# Same string → same view
aapl_again = POOL.istr("AAPL")
assert aapl is aapl_again  # Identity!

# Comparison
assert aapl != tsla
assert aapl == "AAPL"  # Compares to plain str

# Hash for dict/set use
tickers = {aapl, tsla}

# Access properties
print(aapl.string)      # "AAPL"
print(aapl.hash_value)  # 64-bit FNV-1a hash
print(aapl.address)     # Hex C pointer, e.g. "0x7f..."

Pool membership:

pool = POOL

# Check membership (no side effects)
assert "AAPL" in pool
assert "UNKNOWN" not in pool

# Direct lookup (raises KeyError if missing)
view = pool["AAPL"]

# Pool size
print(len(pool))  # Number of interned strings

# Iterate all entries (reverse insertion order)
for istring in pool.internalized():
    print(istring.string)

Custom pool:

from cbase.intern_string import InternStringPool

# Creates a new pool with default allocator (SHM-backed)
local = InternStringPool()
local.istr("hello")

Benchmark Toolkit¶

Run comprehensive benchmarks:

from cbase.intern_string import IstrTestToolkit

toolkit = IstrTestToolkit(buf_size=2**30, n_seg=100_000)
results = toolkit.run_test()

for k, v in results.items():
    if k.endswith("_ns"):
        print(f"{k}: {v:.1f} ns")

Cross-Process Usage¶

import os
from cbase.intern_string import POOL

POOL.istr("shared_data")
print(len(POOL))  # e.g., 1

pid = os.fork()
if pid == 0:
    # Child: sees all pre-fork entries
    print(len(POOL))         # 1
    print("shared_data" in POOL)  # True

    # Insertions in child are visible to parent (SHM-backed)
    POOL.istr("child_data")
else:
    os.waitpid(pid, 0)
    print("child_data" in POOL)  # True — SHM-backed