Phoenix

Overview

Production-grade, fail-closed financial intelligence and execution system for Arbitrum — retired after measurement, with the negative result kept public.

Most on-chain trading systems are described as strategy code. In production they must coordinate real-time event ingestion, on-chain state, independent RPC evidence, exact economics, signer and nonce safety, single-transaction authority, receipts, reconciliation, monitoring, and incident recovery. Phoenix treats all of those as one product and one control system.

What I built

• A multi-language production system across Rust (engine, executor, RPC gateway, supervisor), Go (feed ingestor, atlas observer, recorder), Python (release controller, monitoring, reporting), and Solidity (on-chain executor with minimum-profit guards).
• Three independently armed revenue lanes: Aave V3 liquidation, Atlas auction solver, and origin-aware DEX arbitrage research.
• A fail-closed decision pipeline: ingest → normalize → prefilter → exact validation → dual-provider agreement → conservative economic gate → global submission lock → single-transaction execution → receipt reconciliation → realized PnL.
• A protected release model with immutable image build, manifest, preflight, candidate rehearsal, evidence mode, burn-in, and explicit owner activation before any revenue lane becomes live.
• Production observability across release SHA, container health, provider identity, signers (without keys), controls, candidate counts, expected vs realized PnL, loss-cause ledger, and protected rollback.

Why it matters

The system explicitly refuses to count submission as profit. Realized net PnL after every actual cost — gas, L1 data, flash premium, ordering, slippage, model reserve — is the only authority. When measurement showed that no opportunity would ever clear the economic gates on the targeted pools, the system was retired rather than loosened. That honesty is the design.

Problem

Most on-chain trading systems are described as strategy code. In production they must coordinate real-time event ingestion, on-chain state, independent RPC evidence, exact economics, signer and nonce safety, single-transaction authority, receipts, reconciliation, monitoring, and incident recovery. Phoenix treats all of those as one product and one control system.

Architecture

External Inputs              Data & State Layer           Decision / Economic Engine          Operations
─────────────────             ──────────────────           ──────────────────────────           ───────────
Arbitrum Nitro Feed    ──▶    nitro-feed-relay      ──▶    prefilter                    ──▶    recorder
Atlas Auction Stream   ──▶    feed-ingestor         ──▶    exact-state validation       ──▶    shadow-dispatcher
Aave V3 State          ──▶    atlas-observer        ──▶    dual-provider agreement      ──▶    prometheus
Primary RPC            ──▶    aave-liquidation-hunter──▶   conservative economic gate  ──▶    dashboard
Secondary RPC          ──▶    rpc-gateway           ──▶    candidate emission          ──▶    release-controller
                                NATS JetStream             route & revenue-lane controls           GitHub Actions
                                PostgreSQL                 global submission lock                  Docker Compose

Decision pipeline

1. Ingest event / borrower / auction / route
   ↓
2. Normalize identity, dedupe, reject malformed
   ↓
3. Low-cost prefilter (chain, freshness, debt, route policy)
   ↓
4. Bind opportunity to exact finalized block
   ↓
5. Two independent RPC providers — block, account, prices
   ↓
6. Require exact agreement on chain / block / state
   ↓
7. Estimate full economic outcome (gross − every cost)
   ↓
8. Reject unless conservative_net_pnl > retained_floor
   ↓
9. Revalidate route / lane / signer / contract / nonce / lock
   ↓
10. Materialize one typed execution request
   ↓
11. Submit at most one revenue transaction at a time
   ↓
12. Reconcile receipt / events / balances / fees / nonce
   ↓
13. Record realized net PnL (only after reconciliation)

Safety model

The system is designed around explicit fail-closed controls. These are the invariants that must hold before any execution authority is granted.

• Two independent providers required for exact authority.
• Provider disagreement closes execution authority until fresh agreement returns.
• One global submission lock prevents conflicting revenue transactions.
• One transaction at a time across all lanes.
• Signer material is file-mounted and never stored in CI.
• Each lane has independent armed state and kill switch.
• Unknown submission state blocks new authority.
• On-chain executor includes minimum-profit protection.
• No-alpha is not an error; it never forces a bad trade.
• Release failures roll back through protected, version-matched workflows.
• Realized PnL is recorded only after receipt and balance reconciliation.

Engineering decisions

• Strict separation of intelligence (signals) and authority (execution) — the engine computes; only the executor may sign.
• Conservative economics by default: every cost is modeled, including L1 data, flash premium, ordering cost, and an explicit risk reserve.
• Block-pinned dual-provider agreement — never trust one RPC source for a state-dependent decision.
• Independent revenue lanes with their own armed / kill-switch / loss-limit / floor / epoch.
• Integer arithmetic for token amounts, sqrt prices, ticks, liquidity, gas, and PnL.
• No floating-point on any on-chain quantity or opportunity accounting.
• Submit at most one revenue transaction at a time, regardless of lane.
• Protected release model: immutable build, manifest, preflight, rehearsal, evidence mode, burn-in, then explicit owner activation.
• Realized PnL only after receipt and balance reconciliation — never after submission.
• Public defaults remain safe; production secrets and activation authority live server-side.

What changed during development

• Early prototypes signed and broadcast directly when a modelled spread crossed a threshold — model, state, pool identity, fee tier, and signer boundary were not independent. That surface had to be rebuilt from the evidence layer up.
• Wide initial scope (triangular routes, CEX strategies, sandwiching, blind scanning) created too much surface and too little truth. Scope was narrowed deliberately; breadth is being re-added lane-by-lane only after evidence.
• Long-tail residual observer was activated for LINK/WETH and similar pairs, but upstream providers in the current slot do not support debug_traceTransaction — that lane remains research until the supporting sensor is available.
• A healthy production system can sit in FULL_LIVE_NO_ALPHA for extended periods. That is not failure; it is the system refusing to act without authority. Recognizing this state took time.

Technical implementation

• Rust core (phoenix-engine, live-executor, rpc-gateway, economic-supervisor, recorder, replay, fork-sandbox) — latency-critical path contains no Python.
• Go services (feed-ingestor, atlas-observer, aave-liquidation-hunter, recorder, shadow-dispatcher) for ordered event normalization and observation.
• Python release controller, validation, monitoring, and reporting tooling.
• Solidity PhoenixExecutor contract with minimum-profit guards.
• PostgreSQL as durable control and economic truth; NATS JetStream as internal transport; Prometheus for metrics; Streamlit dashboard for bounded operator visibility.
• Docker Compose orchestration across ingestion, decision, execution, recorder, shadow, database, messaging, metrics, dashboard.
• GitHub Actions + protected release controller + immutable artifact delivery.
• Arbitrum One as the execution network; Aave V3 as the liquidation market and flash provider; Atlas as the auction / solver opportunity stream.

Product implications

• A live production system may legitimately not trade. The product must communicate this honestly to operators, partners, and reviewers.
• Every external metric a dashboard shows — gas, latency, RPC quality — is downstream. The product is realized net PnL after reconciliation.
• Risk controls are not an overlay; they are the architecture. Lane isolation means a researcher lane cannot accidentally take capital from a revenue lane.
• A protected release pipeline is a product capability, not a CI concern. Without it, "live" is not a meaningful word.

Lessons learned

• Reliability beats cleverness. A profitable-looking opportunity is irrelevant if the underlying state is uncertain.
• Economic correctness matters more than technical correctness. A strategy can be technically correct and economically wrong.
• Production is part of the product. Deployment, rollback, observability, and reconciliation are product capabilities — not afterthoughts.
• Separate intelligence from authority. The engine should not hold the keys, and the executor should not pick the opportunity.
• Fail-closed is different from fast. When execution has real consequences, uncertainty must reduce authority rather than trigger action.
• Honest state naming matters. FULL_LIVE_NO_ALPHA is a valid terminal state. Pretending otherwise is how systems become fragile.

Current state

Production live in observation mode. The system is running, dual-provider authority is available, and no currently observed opportunity passes every profitability and safety gate. The next legitimate terminal state is FIRST_POSITIVE_REALIZED_PNL — and that label is only applied after a real transaction is submitted, confirmed, balance-reconciled, and produces positive realized net PnL.