C API Reference

This page documents the low-level C API exposed by PyEventEngine’s Cython extensions. These APIs are used internally by the Cython layer and are generally not needed for typical Python usage.

Warning

The C API is considered advanced/internal. Most users should use the Python API Reference (CAPI) instead. Direct C API usage requires Cython or C extension knowledge.

Overview

PyEventEngine’s C layer provides the following components:

  • Topic structures: Fast topic parsing and matching

  • Message payloads: Zero-copy message passing

  • Event hooks: Low-overhead handler dispatch

  • Queue primitives: Lock-free/hybrid message queues

  • Memory allocators: Pool-based allocation for payloads

All C structures are exposed to Python via Cython cdef classes in the .pyx files.

Header Files

The C API is declared in the following headers under event_engine/capi/:

c_topic.h

Topic data structures and parsing:

typedef enum {
    TOPIC_PART_EXACT = 0,
    TOPIC_PART_ANY = 1,
    TOPIC_PART_RANGE = 2,
    TOPIC_PART_PATTERN = 3
} TopicType;

typedef struct TopicPart {
    TopicType type;
    char* data;
    size_t data_len;
    struct TopicPart* next;
} TopicPart;

typedef struct Topic {
    char* literal;
    size_t literal_len;
    uint64_t hash;
    TopicPart* parts;
    int is_exact;
} Topic;

Key functions:

  • Topic* topic_parse(const char* topic_str, size_t len) - Parse a topic string

  • int topic_match(Topic* pattern, Topic* target) - Check if target matches pattern

  • void topic_free(Topic* topic) - Free topic memory

c_event.h

Event structures for message passing:

typedef struct MessagePayload {
    Topic* topic;
    void* args;        // PyObject* (tuple)
    void* kwargs;      // PyObject* (dict)
    uint64_t seq_id;
    struct MemoryAllocator* allocator;
} MessagePayload;

Note

args and kwargs are opaque void* in C but point to Python objects (PyObject*). Reference counting is managed by the Cython layer.

c_allocator.h

Custom memory allocator for payload objects:

typedef struct MemoryAllocator {
    void* pool;
    size_t capacity;
    size_t block_size;
    int active;
} MemoryAllocator;

MemoryAllocator* c_allocator_new(size_t capacity, size_t block_size);
void* c_heap_alloc(MemoryAllocator* alloc, size_t size);
void c_heap_recycle(MemoryAllocator* alloc, void* ptr);
void c_allocator_free(MemoryAllocator* alloc);

c_bytemap.h

Hash map for topic-to-hook lookups:

typedef struct MapEntry {
    const char* key;
    void* value;
    uint64_t hash;
    struct MapEntry* next;
} MapEntry;

typedef struct ByteMap {
    MapEntry** buckets;
    size_t capacity;
    size_t size;
} ByteMap;

ByteMap* c_bytemap_new(size_t capacity);
void c_bytemap_set(ByteMap* map, const char* key, void* value);
void* c_bytemap_get(ByteMap* map, const char* key);
void* c_bytemap_pop(ByteMap* map, const char* key);
void c_bytemap_free(ByteMap* map);

xxhash.h

Fast hashing (xxHash3) for topic strings:

uint64_t XXH3_64bits(const void* data, size_t len);

This is a vendored copy of the xxHash library for consistent, fast hashing.

Memory Management

Ownership Rules

  1. Topics:

    • Topics created via topic_parse() must be freed with topic_free()

    • Topics from the internal pool (c_bytemap) are managed by the pool

    • Python PyTopic wrappers manage C Topic* via owner flag

  2. MessagePayload:

    • Payloads allocated via MemoryAllocator are recycled to the pool

    • args and kwargs (Python objects) use CPython reference counting

    • The Cython layer calls Py_INCREF/Py_DECREF as needed

  3. ByteMap:

    • The map owns keys (copied strings) but not values (just pointers)

    • Caller must manage value lifetime (e.g., hooks)

Allocator Usage

Example from c_engine.pyx:

cdef MemoryAllocator* allocator = c_allocator_new(4096, sizeof(MessagePayload))

# Allocate a payload
cdef MessagePayload* payload = <MessagePayload*>c_heap_alloc(allocator, sizeof(MessagePayload))
payload.topic = some_topic
payload.args = <void*>args_tuple
Py_INCREF(args_tuple)

# ... use payload ...

# Recycle when done
Py_XDECREF(<object>payload.args)
c_heap_recycle(allocator, payload)

# Clean up allocator on shutdown
c_allocator_free(allocator)

Thread Safety

  • Topic parsing: Thread-safe (no shared state)

  • ByteMap: NOT thread-safe (must be externally locked)

  • MemoryAllocator: NOT thread-safe (designed for single-threaded use or per-thread pools)

  • MessagePayload: Thread-safe if reference counting is correct

The Cython layer (c_engine.pyx) uses Python’s GIL and threading primitives to ensure thread safety at the Python API level.

Using from Cython

To use the C API in your own Cython extensions:

  1. Add event_engine.capi to your include_path

  2. cimport the relevant .pxd files

Example:

# my_extension.pyx
from event_engine.capi.c_topic cimport Topic, topic_parse, topic_free

cdef Topic* t = topic_parse(b"My.Topic.String", 15)
try:
    print(f"Parsed topic with hash: {t.hash}")
finally:
    topic_free(t)

See event_engine/capi/*.pxd for full cimport declarations.

Performance Considerations

  • Topic parsing: O(n) in string length; results are cached in ByteMap

  • Topic matching: O(k) where k = number of parts (~3-5 typically)

  • ByteMap lookups: O(1) expected, O(n) worst-case (hash collisions)

  • Allocator: O(1) alloc/recycle when pool is not exhausted

For maximum performance:

  • Pre-parse and intern topics (init_internal_map())

  • Use exact topics where possible (faster than wildcards)

  • Size ByteMap capacity to ~2x expected topics to minimize collisions

Debugging

Enable debug output in Cython:

import os
os.environ['CYTHON_TRACE'] = '1'

# Rebuild and run with profiling/tracing enabled

Use gdb/lldb for C-level debugging:

# Build with debug symbols
CFLAGS="-g -O0" pip install -e .

# Run under gdb
gdb --args python my_test.py
(gdb) break c_topic.c:topic_parse
(gdb) run

For memory issues, use valgrind:

valgrind --leak-check=full python -m pytest demo/

See Also