System Architecture

Qinematos implements a clean three-layer architecture that separates concerns between agents, core services, and the underlying data plane.

Architecture Overview

Layer 1: Agent Layer

Python agents (PyArrow mmap) | Java agents (FFM) | Rust agents (Arrow2) | Any MCP-compliant client

v

Layer 2: Core Services

Context Orchestrator | Zero-Copy IPC Engine | Arrow Serializer | Lineage Tracker | Xodus Store

v

Layer 3: Data Base Plane

REST API :8080 | gRPC Service :9000 | MCP Server /mcp | Shared Memory /dev/shm

Core Components

Context Orchestrator

The central coordination service managing agent registration, topic subscriptions, and context routing.

  • Agent lifecycle management (registration, heartbeat, deregistration)
  • Topic-based publish/subscribe with wildcard pattern matching
  • Subscription updates via gRPC streaming
  • Context version management with replay capability

Zero-Copy IPC Engine

The performance core of Qinematos, using Java 25 Foreign Function & Memory API.

  • Memory-mapped files in shared memory (/dev/shm on Linux, temp directory on Windows)
  • Arena-based memory management for deterministic cleanup
  • Atomic writes with proper memory barriers
  • File path handles instead of inline data transfer
IPC Flow 
1. Publisher serializes data to Arrow format
2. IPC Engine writes Arrow bytes to memory-mapped file
3. Engine returns IPCHandle with file path and metadata
4. Subscribers receive handle via gRPC stream
5. Subscribers mmap the file directly (zero-copy read)

Arrow Context Serializer

Handles conversion between domain objects and Apache Arrow format.

  • Pre-defined schemas: agent.context, lineage.trace, finance.market_data, generic.keyvalue
  • Schema versioning per topic
  • Cross-language compatibility (Java, Python, Rust, Go)
  • Efficient columnar storage for analytical queries

Topic Manager

Manages the topic hierarchy and subscription state.

  • Hierarchical topic naming: finance.trades.realtime
  • Wildcard subscriptions: finance.*, finance.**
  • Schema binding per topic
  • Retention policies and access control

Lineage Tracker

Provides complete data provenance for audit and compliance.

  • Content-addressable trace IDs (SHA-256 hash of payload)
  • Transformation chain tracking
  • eBPF kernel-level audit capability
  • OpenTelemetry integration for distributed tracing

Xodus Persistence Store

ACID storage for topic metadata, schemas, and configuration.

  • JetBrains Xodus embedded key-value store
  • Pure Java implementation - no external dependencies
  • Transactional guarantees for metadata operations
  • Configurable memory usage (default 50%)

Data Flow

Publishing Context

Publish Flow 
Agent -> REST/gRPC API
     -> Context Orchestrator
     -> Arrow Serializer (convert to Arrow format)
     -> Lineage Tracker (generate trace ID)
     -> IPC Engine (write to shared memory)
     -> Topic Manager (notify subscribers)
     -> gRPC Stream (send IPCHandle to subscribers)

Subscribing to Context

Subscribe Flow 
Agent -> gRPC Watch API
     -> Context Orchestrator (register subscription)
     -> Topic Manager (match patterns)
     -> ... (wait for publications)
     -> Receive IPCHandle via stream
     -> Memory-map file path (zero-copy)
     -> Read Arrow data directly from shared memory

Delivery Methods

Qinematos supports multiple delivery strategies based on agent location and capability:

Method Use Case Latency
DELIVERY_METHOD_IPC Local agents with mmap support <5ms
DELIVERY_METHOD_ARROW_FLIGHT Remote agents, distributed clusters Network RTT + minimal serialization
DELIVERY_METHOD_INLINE Fallback for agents without mmap Depends on payload size

Technology Choices

Why Java 25?

The Foreign Function & Memory (FFM) API in Java 25 provides safe, efficient access to native memory without JNI overhead. This enables true zero-copy operations while maintaining Java's type safety and garbage collection benefits.

Why Apache Arrow?

Arrow provides a language-agnostic columnar format that enables zero-copy reads across process boundaries. Any language with an Arrow implementation can read Qinematos context data without serialization overhead.

Why Quarkus?

Quarkus offers cloud-native features (fast startup, low memory), reactive programming model (Mutiny), and excellent GraalVM native image support for production deployments.

Why gRPC?

gRPC provides efficient binary serialization, HTTP/2 multiplexing, and native streaming support - essential for the Watch API's server-streaming subscriptions.

Configuration

Key configuration properties in application.properties:

application.properties 
# Server ports
quarkus.http.port=8080
quarkus.grpc.server.port=9000

# IPC Engine
qinematos.ipc.base-path=/dev/shm/qinematos
qinematos.ipc.max-segment-size=1073741824
qinematos.ipc.enable-lineage=true

# Persistence
qinematos.xodus.data-dir=./data/xodus
qinematos.xodus.memory-usage-percentage=50

# Observability
quarkus.micrometer.enabled=true
quarkus.otel.enabled=true