Orchestration

When entities change in Nebula, the platform orchestrates which systems need to respond and ensures reliable, scalable processing. You don't configure message brokers or set up consumer groups — Nebula manages the entire pipeline from commit to acknowledgement.

The Commit-to-Processing Pipeline

Every change in Nebula follows a deterministic path from commit to system execution:

1. Commit — A batch of entity changes is committed — from a system completing work, a user action in the frontend, or an external API call. Each commit is atomic.

2. Checkpoints — Nebula creates individual checkpoints for each entity changed in the commit. Each checkpoint captures the delta (what changed), link changes (parent/child modifications), and ownership changes.

3. Matching — A shared pool of Nebula instances evaluates each checkpoint against all registered system queries. This matching determines which systems are interested in each entity's new state.

4. Enqueueing — For every system that matches, the checkpoint is enqueued to that system's dedicated stream. Each system has its own independent queue — systems can't block each other.

5. Processing — Multiple instances of a system can connect to Nebula and share the queue of work. Each instance pulls tasks, processes them, and commits any resulting entity changes (which may trigger the pipeline again).

6. Acknowledgement — Systems send acknowledgement upon task commit. If a system doesn't acknowledge within the timeout, the task is retried and delivered to another available instance.

7. Deduplication — A unique task key is generated for each commit + entity combination. Nebula rejects duplicate commits with the same key, preventing double-processing even during retries.

Scalability Model

Nebula's orchestration scales horizontally at every layer:

Matching pool — The shared Nebula instances that evaluate checkpoints against system queries scale independently. Adding more instances increases checkpoint throughput without affecting any individual system.

Per-system streams — Each system has its own dedicated queue. A slow or backlogged system doesn't affect others. If one system is processing thousands of entities while another handles a trickle, they operate independently.

System instance pools — Multiple instances of the same system share its queue. If a system needs more throughput, you add instances. Each instance is stateless — any instance can process any task from the queue.

This architecture means you can scale specific bottlenecks without over-provisioning the entire system. A high-volume data ingestion system can run 10 instances while a low-frequency alerting system runs 1.

Reliability Guarantees

Nebula provides at-least-once delivery with deduplication at the commit level:

Every checkpoint is delivered to every matching system at least once
Task key uniqueness at commit time prevents duplicate writes even if the same task is processed twice
Unacknowledged tasks are retried after a configurable timeout
Systems are stateless by design — any instance can pick up any task, and a task that fails on one instance can succeed on another

The deduplication guarantee is important: a system might process the same entity twice during a retry, but because the commit includes a unique task key, Nebula will reject the second commit. This means your system code doesn't need to be idempotent at the write level — Nebula handles it.

The pipeline is self-propagating. When a system commits changes as part of processing, those changes flow through the same pipeline — creating checkpoints, matching against other systems, and potentially triggering further processing. This is how complex workflows emerge from simple, independent systems.

Orchestration ensures that Behaviors execute reliably and at scale. For controlling who can trigger and observe these changes, see Access Control.