Dashboard / Forex-Macro / Doc

Forex Data Collection Engine — MERGED FINAL SPEC

Version: Merged Final (March 2026) Base: Claude Opus 4.6 review + original spec Panel additions cherry-picked from: Gemini 3.1 Pro, Grok 4.1, DeepSeek V3.1, GPT-4.1, Grok 4 Fast, Sonar Reasoning Pro Primary trading venue: OANDA (spot forex — all live price, order book, position book, spread, and swap data) Secondary venues: Kalshi and Polymarket (prediction market contracts on macro/economic data events — ladder structures for rate decisions, CPI, NFP outcomes) Status: Complete. Use this as the authoritative implementation reference.

PART 0: PANEL ASSESSMENT SUMMARY

This merged spec incorporates the Opus review (the most thorough: $1.07 cost, 8192 tokens out, grade B+) as the complete base, with unique additions cherry-picked from the other six panel members. The Opus review diagnosed the original spec accurately: exceptional domain depth, but approximately 3-4x larger than what is realistic to build simultaneously at $145/month. The build order in this merged spec reflects Opus's phased recommendation with targeted supplements from the rest of the panel.

Key findings from all reviewers (consensus):

PART 1: WHAT THIS IS

A script that runs silently 24 hours a day, 5 days a week (Sunday 5PM ET to Friday 5PM ET) during forex market hours. It pulls price data and calculates indicators every 1-5 minutes, stores everything in a SQLite database, and never sends alerts or makes trades. Its only job is to build the deepest possible historical dataset so that AI research analysts can find patterns, confluences, and edges over time.

This script feeds data into a research pipeline. AI researchers and analysts receive snapshots from this database as "known facts" injected into their prompts. They don't search the internet for "what's EUR/USD doing" — they already know because this script tells them.

Primary trading venue: OANDA v20 REST API (live spot forex — all candle, order book, position book, spread, and carry data comes from here).

Secondary venues: Kalshi and Polymarket prediction markets carry contracts on macro outcomes (Fed rate decisions, CPI beats/misses, NFP ranges). These have ladder structures (multiple strikes, not just binary) and are tracked separately in Section 20.

PART 2: WHERE IT RUNS

PART 3: CURRENCY PAIRS

Start with major and cross pairs that have liquid markets and prediction market contracts:

EUR/USD, GBP/USD, USD/JPY, USD/CHF, AUD/USD, NZD/USD, USD/CAD, EUR/GBP, EUR/JPY, GBP/JPY, AUD/JPY, NZD/JPY, EUR/AUD, GBP/AUD, EUR/CAD

Boss can add or remove pairs via a config file. The script reads the list at startup.

PART 4: TIMEFRAMES

Every indicator below is calculated on these timeframes unless noted otherwise:

[PANEL: Sonar Reasoning Pro] For the complex calculations (S&R Engine, full ZigZag trend lines, Section 16 structure), restrict to 1H, 4H, Daily, Weekly, Monthly only. Use 5min and 15min strictly for basic indicators (RSI, MAs, SuperTrend) and entry timing. This reduces computational load by ~80% while preserving the institutional-grade macro edge.

The script pulls candle data (open, high, low, close, tick_volume) from OANDA for each timeframe and pair.

SECTION 1: CORE STRATEGY — SuperTrend + ATR

This is the primary trading strategy. Everything else in this spec supports it.

SuperTrend (10-period ATR, multiplier 2)

SuperTrend is a trend-following indicator. When price is above the line, the trend is bullish. When price is below, bearish. The moment it switches is a "flip."

What to calculate and store per pair, per timeframe, every candle close:

When a flip occurs, create a separate event record:

Outcome tracking (filled in later as price moves after the flip):

[PANEL: Sonar Reasoning Pro] Critical: outcome backfill must have a hard stop. Set a maximum window of 50 candles per timeframe. After 50 candles, mark outcome_window_expired = true and freeze all outcome fields. Without this, win rate statistics are incomparable across trades — some tracked for 50 candles, others for 500.

[PANEL: Sonar Reasoning Pro] Weighted confluence scoring. The multi-indicator confluence score (Section 6) should weight signals by independence, not equal +1/-1. Suggested weights: SuperTrend direction: +1.0, ADX > 25: +0.5, RSI divergence: +1.0, price > SMA200: +0.5. These should be validated on historical data before being set in stone.

ATR (Average True Range, 14-period)

What to store per pair, per timeframe, every candle close:

SuperTrend Pullback Entry Strategy

After a SuperTrend flip, calculate and track the pullback entry level:

When a SuperTrend flip occurs, calculate and store:

Outcome tracking:

SECTION 2: TREND LINES — 3 PRICE SETS + 1 RSI SET

We track multiple sets of trend lines simultaneously. Each uses a different method for identifying swing points. Over time the database reveals which set is most predictive.

Common Concepts

ZigZag Indicator (15/5): Depth 15, Deviation 5. Identifies significant swing highs and lows by filtering noise.

The Repainting Problem and Fix: ZigZag's most recent point is NOT reliable — it keeps moving. A point only locks in when a new swing in the OPPOSITE direction is confirmed after it. Rule: NEVER use unconfirmed ZigZag points.

[PANEL: Sonar Reasoning Pro] The deeper repainting problem: Even "confirmed" points can generate repainting trend lines. A confirmed B-to-C line repaints when a new D point shifts C's anchor. Fix: only lock a trend line when the swing AFTER its anchor point is confirmed. Report CLOSED/locked trend lines only — analysts use yesterday's frozen lines to trade today. Zero repainting, zero confusion.

[PANEL: Opus] Custom ZigZag implementation required. Do not use a library. Implement from scratch with explicit state management: maintain current_tentative_point (direction, price, timestamp), confirmed_points[] array. On each new candle, check if tentative point updates (price extended) or locks (opposite reversal exceeds threshold). Store tentative points with is_confirmed = false, flip to true when locked. Approximately 100-150 lines of careful code.

Higher-Timeframe (HTF) Validation:

Chart Timeframe Swing point must match the high or low of a closed...
15min 4H candle
1H Daily candle
4H Weekly candle
Daily Monthly candle

How trend lines are drawn: Take the last 3 qualifying swing points: A (oldest), B (middle), C (most recent). Draw:

  1. Line 1: connects A to B
  2. Line 2: connects B to C

As new qualifying points appear, A drops off, B becomes A, C becomes B, new point becomes C.

PRICE SET 1: ZigZag + HTF Validated (Tightest Filter — Fewest Lines, Highest Quality)

A point qualifies only if: (1) it is confirmed (not the most recent unconfirmed ZZ point), AND (2) it also matches the high or low of a completed higher-timeframe period.

Fields stored per pair, per timeframe, every candle close:

PRICE SET 2: HTF High/Low Only (No ZigZag Required)

Use only the highs and lows of completed higher-timeframe periods. No ZigZag. For 4H chart: take high and low of each closed week. Catches important levels ZigZag might skip.

Fields: Same structure as Set 1 but prefixed s2_

PRICE SET 3: ZigZag Only (No HTF Validation)

All confirmed ZigZag (15/5) swing points regardless of HTF match. Catches mid-period swings that don't coincide with a period boundary.

Fields: Same structure but prefixed s3_

RSI TREND LINES: Independent RSI Highs/Lows Within Closed HTF Periods

RSI trend lines use the same concept but applied to RSI(14). RSI swing points are identified INDEPENDENTLY from price — they may be on different candles. That disagreement is divergence information.

How RSI swing points are selected: Within a closed higher-timeframe period, find the candle with the HIGHEST RSI reading and the candle with the LOWEST RSI reading. These are the RSI swing points for that period.

Same timeframe pairings as price:

Chart Timeframe RSI swing points from highs/lows of RSI within a closed...
15min 4H period
1H Daily period
4H Weekly period
Daily Monthly period

Fields stored per pair, per timeframe, every candle close:

TOTAL TREND LINES PER PAIR PER TIMEFRAME

Set Lines What It Captures
Price Set 1 (ZZ + HTF) 2 (A-B, B-C) Strongest structural levels
Price Set 2 (HTF only) 2 (A-B, B-C) HTF structure ZZ might miss
Price Set 3 (ZZ only) 2 (A-B, B-C) Mid-period swings HTF misses
RSI (independent HTF) 2 (highs, lows) Momentum structure independent from price
Total 8 trend lines

Trend Line Events (Apply to ALL Sets)

SECTION 3: EVENT TIMING CORRELATIONS

This section tracks timing relationships between major events. Window: 10 candles (adjustable).

SuperTrend Flip + Price Trend Line Break

When a pairing is detected (ST flip + price TL break within 10 candles):

RSI Trend Line Break + Price Trend Line Break

RSI Trend Line Break + SuperTrend Flip

Same structure, comparing RSI break to ST flip.

Triple Confluence (All Three)

Outcome Tracking (All Correlation Types)

Backfilled as subsequent candles arrive (hard cap: 50 candles, then freeze):

SECTION 4: RSI DIVERGENCES

Four Types

Regular Bullish (price lower low, RSI higher low): Selling pressure weakening. Reversal up signal.

Regular Bearish (price higher high, RSI lower high): Buying pressure weakening. Reversal down signal.

Hidden Bullish (price higher low, RSI lower low): Uptrend continuation. Pullback dip is a reset.

Hidden Bearish (price lower high, RSI higher high): Downtrend continuation. Rally is a reset.

What to Store When Detected

Outcome tracking (hard cap 20 candles):

SECTION 5: OTHER TECHNICAL INDICATORS

All calculated per pair, per timeframe, every candle close.

Moving Averages

RSI (14-period)

MACD (12, 26, 9)

Bollinger Bands (20, 2)

Stochastic (14, 3, 3)

ADX / DMI (14-period)

Keltner Channels (20, 1.5 ATR)

VWAP (Volume-Weighted Average Price)

[PANEL: Gemini 3.1 Pro] Important: spot forex has no real volume, only tick counts. Calculating VWAP on OANDA tick data is mathematically imprecise and won't match institutional VWAP. Either rename these fields to TWAP (Time-Weighted Average Price) to reflect reality, or calculate using CME futures tick data. The label "VWAP" on tick-count data is misleading.

Volume-Pressure Indicators

On-Balance Volume (OBV):

Chaikin Money Flow (CMF — 20-period):

Elder's Force Index (13-period EMA):

Momentum / Rate of Change

Fibonacci Retracement

From most recent major swing (identified by ZigZag on current timeframe):

[PANEL: Opus] Fibonacci direction clarification: Define explicitly which direction retracements measure. Take the last confirmed ZZ swing HIGH and the last confirmed ZZ swing LOW. If the HIGH is more recent, draw retracement from HIGH down to LOW (measuring pullback in a downtrend). If LOW is more recent, draw from LOW up to HIGH (measuring pullback in an uptrend). Store fib_direction field to make this explicit.

Pivot Points

Round Numbers (.0025 Grid)

[PANEL: Opus] Round number grid generates too many levels. Only generate round number candidates within 2 * ATR(14, Daily) of current price. Weight by significance: .0000 levels = 100% score, .0050 = 60%, .0025/.0075 = 30%. Weaker levels only survive clustering if another method confirms them.

Pair Correlations and Lead-Lag Analysis

Lead-Lag Analysis:

[PANEL: Opus] Lead-lag analysis is computationally expensive and relationships shift with regime. Collect the raw price data now; run lead-lag as a batch offline job after 3+ months of clean data. Do not build into the real-time collection engine.

SECTION 6: CONFLUENCE EVENTS

Log specific multi-indicator setups as timestamped events.

Momentum Squeeze

Trigger: BB inside KC (bb_squeeze) AND RSI 40-60 AND ATR below 20-period average. Log: timestamp, pair, timeframe, squeeze duration, breakout direction when released.

Trend Exhaustion

Trigger: RSI regular divergence AND volume spike 2x+ 20-period average AND SMA20/SMA50 spread narrowing. Log: timestamp, pair, timeframe, exhaustion direction.

Breakout Setup

Trigger: ATR expanding AND BB squeeze just released (last 5 candles) AND volume spike. Log: timestamp, pair, timeframe, breakout direction, ATR at breakout.

Mean Reversion Extreme

Trigger: RSI < 30 or > 70 AND price > 3x ATR from SMA20 AND price near S&R zone. Log: timestamp, pair, timeframe, direction, distance from mean.

Multi-Indicator Confluence Score

Per pair, per timeframe, every candle:

[PANEL: Sonar Reasoning Pro] Equal weighting is naive — SuperTrend and RSI are semi-correlated (both momentum). Better: weight by signal independence. Suggested validated weights: SuperTrend +1.0, ADX > 25 +0.5, RSI divergence +1.0, price > SMA200 +0.5. Test these on historical data before locking in.

SECTION 7: ORDER FLOW AND POSITIONING

OANDA Order Book (free via OANDA REST API)

Pull every 15-20 minutes:

OANDA Position Book (free via OANDA REST API)

Pull every 15-20 minutes:

[PANEL: DeepSeek V3.1] Retail profit/loss ratios. Derive: when retail traders are heavily underwater (underwater_longs_pct or underwater_shorts_pct > 70%), brokers and institutions often pressure those positions aggressively. Track the combination as retail_trapped_side = "longs_trapped" / "shorts_trapped" / "balanced". This adds a Wyckoff-style squeeze detection from the OANDA book.

Spread Dynamics (OANDA real-time quotes)

Pull every 1 minute:

[PANEL: Opus] During high-impact events (from calendar), increase spread polling to every 5 seconds for a 10-minute window around the event. Store event_max_spread, event_avg_spread, spread_time_to_normal. OANDA streaming API (v20) can provide real-time pricing for this.

COT Report (CFTC — weekly)

Pull weekly (published Friday, data from Tuesday — always model the 3-day lag):

[PANEL: Opus] COT is a weekly, lagged indicator (3-day minimum lag between data and publication). Use it only for weekly/monthly bias assessment, never for intraday or daily timing. Label every COT field with a cot_as_of_tuesday flag so downstream analysts always know the data is from Tuesday regardless of when it was ingested.

[PANEL: DeepSeek V3.1] CME FX futures open interest. Available as free delayed data (15-min delay) from CME Group. Shows whether new money is entering trends (open interest rising) vs position squaring (open interest falling on price moves). Distinguish from COT: this is more timely. Table: cme_oi with pair, timestamp, open_interest, open_interest_change_pct.

Swap Points and Carry Trade Signals (OANDA API)

Pull daily:

SECTION 8: CROSS-MARKET SIGNALS

Bond Yields (FRED API — free, official, reliable)

[PANEL: GPT-4.1] Do NOT use Yahoo Finance for yields. FRED API is free, official, and reliable. It updates daily, which is sufficient for yield spread analysis — spreads don't change meaningfully intraday for forex purposes.

Pull daily via FRED API (series IDs: DGS2, DGS10, IRLTLT01DEM156N, IRLTLT01GBM156N, IRLTLT01JPM156N, etc.):

Equity Indices (Yahoo Finance / Twelve Data — daily granularity acceptable)

Pull daily:

Commodities (Yahoo Finance / Twelve Data — daily)

Volatility

Dollar Index (DXY — Synthetic from OANDA Components)

[PANEL: Opus] DXY data source clarification — MUST-HAVE fix. DXY is NOT tradeable on OANDA. Yahoo Finance DXY (DX-Y.NYB) is delayed and only updates during US futures hours. The correct solution: calculate synthetic DXY from OANDA pairs you already have.

Formula: DXY = 50.14348112 × EURUSD^(-0.576) × USDJPY^(0.136) × GBPUSD^(-0.119) × USDCAD^(0.091) × USDSEK^(0.042) × USDCHF^(0.036)

Note: USD/SEK may not be on OANDA free tier. If unavailable, approximate with an EUR weight adjustment or add USD/SEK to the pair list. Store as dxy_synthetic so it's clearly labeled as calculated, not pulled from an exchange.

[PANEL: Gemini 3.1 Pro] Central Bank Rate Probabilities. Yield spreads track levels, but FX markets price in specific PROBABILITIES of rate changes at upcoming meetings. A shift from 60% to 80% probability of a Fed cut drives USD instantly. Scrape CME FedWatch Tool or pull from Atlanta Fed / FRED PROBFIRSTHIKE or similar series.

[PANEL: Grok 4.1] Central bank speech and forward guidance sentiment. FX moves sharply on central bank rhetoric, not just official releases. Scrape central bank websites for speeches and minutes. Apply NLP sentiment scoring (free with Python NLTK or Hugging Face). Store per speech as cb_speech_events with bank, speaker, sentiment_score (-1 to +1), key_terms (JSON array), timestamp.

SECTION 9: CURRENCY STRENGTH ISOLATION

For each currency (USD, EUR, GBP, JPY, AUD, NZD, CAD, CHF):

Pull every 5 minutes from OANDA cross-pair data.

Why: Trading the strongest currency against the weakest gives sharper edges. If EUR/USD is breaking out, and EUR strength z-score is spiking but USD strength is flat, it's a Euro-driven move with more follow-through. If USD strength is crashing, EUR/USD rally reverses when USD pressure stops.

SECTION 10: SESSION-SPECIFIC TRACKING

Session Definitions (ET)

[PANEL: GPT-4.1] Session times must be calculated from UTC, not hardcoded in ET. Session start/end times shift with daylight saving transitions. Maintain a mapping table: UTC offset per region per week. Convert to ET for display only.

Per Session, Per Pair, Log:

Session-Specific Patterns

Local Bank Holiday Calendar

[PANEL: GPT-4.1] True market holidays include partial days. Track Japanese Golden Week, US half-days, and other partial closures — not just full bank holidays. These create thin liquidity that breaks normal session statistics without a full holiday flag.

Source: Nager.Date API (free) + maintain a static calendar file updated yearly.

Weekend Gap Handling

[PANEL: Sonar Reasoning Pro] Weekend gaps break indicator continuity. When the Asian session opens Sunday 5PM ET, 48-hour gaps cause ZigZag false swing points, stale moving averages, and discontinuous volume. Fix: detect gap size on session open (weekend_gap_pips), mark data as weekend_gap_period = true for the first 2 hours, and log gap_size_pct as a regime flag. Do not use pre-gap data for short-window calculations during this period.

SECTION 11: VOLATILITY REGIME CLASSIFICATION

Historical Volatility Percentile

Kaufman Efficiency Ratio (14-period)

Fractal Dimension Index (30-period)

[PANEL: Opus] FDI is computationally expensive and adds marginal value over Kaufman ER (which captures the same trending-vs-choppy distinction). Defer to v2.

Volatility Squeeze Detection

Realized vs Implied Volatility Spread

[PANEL: Opus] Free vol spread signal from existing data. VIX (Section 8) is implied vol for equities. Compare to realized vol (HV percentile). When implied >> realized, market pricing in a move that hasn't happened. When realized >> implied, market is complacent. Store the spread and its z-score.

Adaptive Regime Flags

SECTION 12: ECONOMIC CALENDAR AND NEWS PROXIMITY

Economic Calendar

[PANEL: Opus/Gemini/GPT-4.1/Grok — consensus] Do NOT scrape ForexFactory. It actively blocks scrapers using Cloudflare. Will break within days.

Use instead (in order of preference):

  1. ForexFactory XML feed: https://www.forexfactory.com/calendar?week=this (unofficial XML, more stable than HTML scraping but still unofficial)
  2. myfxbook.com calendar RSS feed (more stable)
  3. TradingEconomics API (free tier: 1000 requests/month)
  4. FinancialModelingPrep ($19/month) for reliability
  5. Fallback: FRED API for actual release values (official, free)

Build with primary + fallback: if primary source fails, try secondary and alert Telegram.

Pull daily, update every 6 hours:

[PANEL: Grok 4 Fast / Sonar Reasoning Pro] Actual release values are critical. The economic calendar must capture actual vs forecast data, not just event timing. Surprise magnitude is the primary driver of immediate market reaction. Source: FRED API for US releases (official, free). Store as economic_releases table with actual, consensus, previous, surprise, surprise_normalized (surprise / std deviation of historical surprises for this series).

Specific Event Flags

Central Bank Rate Decision Events

[PANEL: Opus] Rate decisions are the most impactful forex event. Track not just the meeting week but: cb_rate_decision table with bank, decision_timestamp, outcome ("hike"/"cut"/"hold"), amount_bp, matched_consensus (boolean), surprise_direction, surprise_magnitude_bp. FRED API provides actual rate levels for free (DFEDTAR for Fed funds, etc.).

Post-Release Tracking

Prediction Market Event Contracts (OANDA Secondary / Kalshi + Polymarket)

[PANEL: GPT-4.1] Prediction markets provide real-money probability on macro outcomes. Kalshi and Polymarket carry contracts on Fed rate decisions, CPI beats/misses, NFP range outcomes. These are binary/ladder markets with real-money pricing — more predictive than survey consensus. Pull available markets around scheduled economic releases. See Section 20 for full prediction market spec.

SECTION 13: SEASONAL AND TIME-BASED PATTERNS

Day-of-Week Tracking

Month-End / Quarter-End Flows

Holiday Liquidity

Seasonal Factors

Institutional Fix Times

SECTION 14: DERIVED METRICS

Forward Returns (Labels for Future ML)

[PANEL: Sonar Reasoning Pro] Forward bias protection. Forward returns must only be calculated on data older than 30 days. Any candle from the last 30 days does NOT get a forward return label yet — future data isn't in the database. This prevents training-test leakage where models learn to "predict" data they can already see. Add fwd_label_available boolean = false for candles within last 30 days.

Setup Win Rates (rolling, recalculated daily)

Price Path Efficiency

Intermarket Correlation Shifts

SECTION 15: DATA STORAGE AND ARCHITECTURE

Critical Infrastructure: SQLite WAL Mode

[PANEL: Opus/Gemini/GPT-4.1/Grok — CRITICAL] SQLite write throughput will bottleneck without WAL mode.

At 15 pairs × 7 timeframes, peak coincides with 4H close: indicators, S&R updates, ZigZag, trend lines, and event detection all fire simultaneously. SQLite uses a single-writer lock — concurrent writes queue up.

Required at startup:

PRAGMA journal_mode = WAL;
PRAGMA synchronous = NORMAL;
PRAGMA cache_size = -64000;  -- 64MB cache
PRAGMA temp_store = MEMORY;
PRAGMA mmap_size = 268435456;  -- 256MB memory map

Required in all write loops: Use single BEGIN TRANSACTION / COMMIT per collection cycle. Never write row-by-row.

[PANEL: Gemini 3.1 Pro] Offload heavy math to Node.js Worker Threads if using Node.js. S&R clustering and ZigZag recalculation on 4H/Daily closes can block the event loop for seconds, delaying next candle ingestion and causing database is locked errors. Worker threads share the write queue without blocking.

[PANEL: Grok 4.1] For future scale beyond 1-2 years of data: plan a migration path to PostgreSQL + TimescaleDB extension or InfluxDB. Keep the SQLite schema normalized now so migration is not a rewrite.

Database Schema (SQLite)

Main time series table:

CREATE TABLE forex_candles (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  timestamp TEXT NOT NULL,           -- UTC ISO8601
  pair TEXT NOT NULL,                -- e.g., 'EUR_USD'
  timeframe TEXT NOT NULL,           -- e.g., '4H', 'D', 'W'
  open REAL NOT NULL,
  high REAL NOT NULL,
  low REAL NOT NULL,
  close REAL NOT NULL,
  tick_volume INTEGER,
  is_imputed INTEGER DEFAULT 0,      -- 1 if forward-filled
  -- SuperTrend
  supertrend_direction TEXT,
  supertrend_value REAL,
  supertrend_flipped INTEGER,
  candles_since_last_flip INTEGER,
  price_distance_to_st REAL,
  -- ATR
  atr_14 REAL,
  atr_14_expanding INTEGER,
  atr_14_percentile_90d REAL,
  atr_14_rate_of_change REAL,
  -- Moving averages
  sma_20 REAL, sma_50 REAL, sma_100 REAL, sma_200 REAL,
  price_vs_sma_20 REAL, price_vs_sma_50 REAL, price_vs_sma_100 REAL, price_vs_sma_200 REAL,
  ma_alignment TEXT,
  golden_cross INTEGER, death_cross INTEGER,
  -- RSI
  rsi_14 REAL, rsi_overbought INTEGER, rsi_oversold INTEGER,
  -- MACD
  macd_line REAL, macd_signal REAL, macd_histogram REAL,
  macd_cross_up INTEGER, macd_cross_down INTEGER, macd_above_zero INTEGER,
  -- Bollinger Bands
  bb_upper REAL, bb_middle REAL, bb_lower REAL,
  bb_width_pct REAL, bb_squeeze INTEGER, price_position_in_bb REAL,
  -- Stochastic
  stoch_k REAL, stoch_d REAL, stoch_overbought INTEGER, stoch_oversold INTEGER,
  -- ADX/DMI
  adx_14 REAL, plus_di REAL, minus_di REAL,
  di_cross_up INTEGER, di_cross_down INTEGER, adx_rising INTEGER,
  -- Keltner Channels
  kc_upper REAL, kc_lower REAL, kc_middle REAL,
  kc_width_pct REAL, price_position_in_kc REAL,
  kc_breakout_up INTEGER, kc_breakout_down INTEGER,
  -- VWAP / TWAP
  vwap REAL, price_vs_vwap REAL,
  vwap_upper_1sd REAL, vwap_lower_1sd REAL,
  vwap_upper_2sd REAL, vwap_lower_2sd REAL,
  vwap_24h REAL, vwap_48h REAL, vwap_weekly REAL,
  -- Volume indicators
  obv REAL, obv_sma_20 REAL,
  obv_divergence_bullish INTEGER, obv_divergence_bearish INTEGER,
  cmf_20 REAL, cmf_cross_zero_up INTEGER, cmf_cross_zero_down INTEGER,
  force_index_13 REAL, force_index_divergence INTEGER,
  -- Momentum
  roc_10 REAL, roc_20 REAL,
  -- Fibonacci
  fib_0 REAL, fib_236 REAL, fib_382 REAL, fib_500 REAL,
  fib_618 REAL, fib_786 REAL, fib_100 REAL,
  fib_direction TEXT,
  nearest_fib_level TEXT, distance_to_nearest_fib REAL,
  -- Pivots
  daily_pivot REAL, daily_r1 REAL, daily_r2 REAL, daily_r3 REAL,
  daily_s1 REAL, daily_s2 REAL, daily_s3 REAL,
  weekly_pivot REAL, weekly_r1 REAL, weekly_r2 REAL, weekly_r3 REAL,
  weekly_s1 REAL, weekly_s2 REAL, weekly_s3 REAL,
  monthly_pivot REAL, monthly_r1 REAL, monthly_r2 REAL, monthly_r3 REAL,
  monthly_s1 REAL, monthly_s2 REAL, monthly_s3 REAL,
  -- Round numbers
  nearest_round_above REAL, nearest_round_below REAL,
  round_number_bracket TEXT, distance_to_nearest_round REAL,
  -- Correlations
  eurusd_gbpusd_corr REAL, usdjpy_dxy_corr REAL,
  correlation_break INTEGER,
  -- Trend line Sets (s1, s2, s3)
  s1_line_ab_current_value REAL, s1_line_bc_current_value REAL,
  s1_line_ab_slope REAL, s1_line_bc_slope REAL,
  s1_price_dist_to_line_ab REAL, s1_price_dist_to_line_bc REAL,
  s2_line_ab_current_value REAL, s2_line_bc_current_value REAL,
  s2_line_ab_slope REAL, s2_line_bc_slope REAL,
  s2_price_dist_to_line_ab REAL, s2_price_dist_to_line_bc REAL,
  s3_line_ab_current_value REAL, s3_line_bc_current_value REAL,
  s3_line_ab_slope REAL, s3_line_bc_slope REAL,
  s3_price_dist_to_line_ab REAL, s3_price_dist_to_line_bc REAL,
  -- RSI trend lines
  rsi_tl_high_line_slope REAL, rsi_tl_low_line_slope REAL,
  rsi_tl_high_line_current REAL, rsi_tl_low_line_current REAL,
  rsi_dist_to_high_line REAL, rsi_dist_to_low_line REAL,
  -- Volatility regime
  hv_20_percentile_252d REAL,
  kaufman_er REAL, fdi_30 REAL,
  bb_inside_kc INTEGER, squeeze_duration INTEGER,
  squeeze_release INTEGER, squeeze_release_direction TEXT,
  regime_trend_strategy_preference TEXT,
  vix_vs_realized_spread REAL, vix_vs_realized_zscore REAL,
  -- Confluence score
  confluence_score REAL, multi_tf_confluence_score REAL,
  -- Forward returns (labels for ML)
  fwd_return_1h REAL, fwd_return_4h REAL, fwd_return_24h REAL,
  fwd_mfe_24h REAL, fwd_mae_24h REAL, fwd_label_available INTEGER DEFAULT 0,
  UNIQUE(timestamp, pair, timeframe)
);
CREATE INDEX idx_candles_pair_tf_ts ON forex_candles (pair, timeframe, timestamp);
CREATE INDEX idx_candles_ts ON forex_candles (timestamp);

Event tables:

CREATE TABLE supertrend_flips (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  flip_timestamp TEXT NOT NULL,
  pair TEXT NOT NULL,
  timeframe TEXT NOT NULL,
  flip_direction TEXT NOT NULL,
  flip_price REAL,
  st_line_at_flip REAL,
  atr_at_flip REAL,
  atr_trend_at_flip TEXT,
  higher_tf_agrees INTEGER,
  multi_tf_alignment_count INTEGER,
  stop_loss REAL,
  entry_price REAL,
  risk_distance REAL,
  target_2r REAL, target_3r REAL, target_4r REAL, target_5r REAL, target_6r REAL,
  -- Outcome tracking (backfilled, hard cap 50 candles)
  max_favorable_excursion_pips REAL,
  max_favorable_excursion_atr REAL,
  max_adverse_excursion_pips REAL,
  hit_2r INTEGER, hit_3r INTEGER, hit_4r INTEGER, hit_5r INTEGER, hit_6r INTEGER,
  hit_2r_timestamp TEXT, hit_3r_timestamp TEXT,
  time_to_2r_candles INTEGER, time_to_3r_candles INTEGER,
  outcome_window_expired INTEGER DEFAULT 0,
  outcome_candles_checked INTEGER DEFAULT 0
);

CREATE TABLE pullback_setups (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  flip_id INTEGER REFERENCES supertrend_flips(id),
  pair TEXT NOT NULL,
  timeframe TEXT NOT NULL,
  setup_timestamp TEXT NOT NULL,
  entry_price REAL,
  stop_loss REAL,
  risk_distance REAL,
  pullback_reached_entry INTEGER,
  pullback_reached_timestamp TEXT,
  entry_to_stop_hit INTEGER,
  entry_to_2r_hit INTEGER, entry_to_3r_hit INTEGER,
  entry_to_4r_hit INTEGER, entry_to_5r_hit INTEGER, entry_to_6r_hit INTEGER,
  no_fill INTEGER DEFAULT 0,
  outcome_window_expired INTEGER DEFAULT 0
);

CREATE TABLE zigzag_points (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  pair TEXT NOT NULL,
  timeframe TEXT NOT NULL,
  timestamp TEXT NOT NULL,
  price REAL NOT NULL,
  type TEXT NOT NULL,           -- 'high' or 'low'
  is_confirmed INTEGER DEFAULT 0,
  htf_validated INTEGER DEFAULT 0,
  htf_period TEXT             -- which HTF period validated this
);

CREATE TABLE trendline_events (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  pair TEXT NOT NULL,
  timeframe TEXT NOT NULL,
  timestamp TEXT NOT NULL,
  tl_set TEXT NOT NULL,        -- 's1', 's2', 's3', 'rsi'
  tl_line TEXT NOT NULL,       -- 'AB', 'BC', 'high', 'low'
  event_type TEXT NOT NULL,    -- 'touch', 'break', 'retest'
  direction TEXT,
  magnitude_pips REAL,
  volume_at_event INTEGER,
  held INTEGER                 -- for retests: did it hold?
);

CREATE TABLE st_tl_correlations (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  pair TEXT NOT NULL,
  timeframe TEXT NOT NULL,
  correlation_type TEXT NOT NULL,  -- 'st_tl', 'rsi_tl', 'rsi_st', 'triple'
  st_flip_timestamp TEXT,
  st_flip_direction TEXT,
  tl_break_timestamp TEXT,
  tl_break_set TEXT,
  tl_break_line TEXT,
  rsi_break_timestamp TEXT,
  rsi_break_line TEXT,
  candles_apart INTEGER,
  which_first TEXT,
  direction_agrees INTEGER,
  rsi_led INTEGER,
  sequence TEXT,               -- for triple: order of three events
  all_directions_agree INTEGER,
  atr_at_event REAL,
  higher_tf_agrees INTEGER,
  outcome_10_candles REAL,
  outcome_20_candles REAL,
  outcome_50_candles REAL,
  max_favorable_excursion REAL,
  max_adverse_excursion REAL,
  outcome_window_expired INTEGER DEFAULT 0
);

CREATE TABLE divergence_events (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  pair TEXT NOT NULL,
  timeframe TEXT NOT NULL,
  divergence_type TEXT NOT NULL,
  timestamp TEXT NOT NULL,
  price_swing_1_price REAL, price_swing_1_timestamp TEXT,
  price_swing_2_price REAL, price_swing_2_timestamp TEXT,
  rsi_swing_1 REAL, rsi_swing_2 REAL,
  price_slope_zscore REAL, rsi_slope_zscore REAL, divergence_strength REAL,
  supertrend_direction TEXT,
  supertrend_agrees INTEGER,
  higher_tf_divergence INTEGER,
  at_support_resistance INTEGER,
  outcome_5_candles REAL, outcome_10_candles REAL, outcome_20_candles REAL,
  reversal_occurred INTEGER, reversal_pips REAL,
  continuation_occurred INTEGER, continuation_pips REAL,
  outcome_window_expired INTEGER DEFAULT 0
);

CREATE TABLE confluence_events (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  pair TEXT NOT NULL,
  timeframe TEXT NOT NULL,
  event_type TEXT NOT NULL,    -- 'momentum_squeeze', 'trend_exhaustion', 'breakout_setup', 'mean_reversion_extreme'
  timestamp TEXT NOT NULL,
  direction TEXT,
  squeeze_duration INTEGER,
  breakout_direction TEXT,
  atr_at_event REAL
);

CREATE TABLE news_releases (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  event_timestamp TEXT NOT NULL,
  pair TEXT,
  currency TEXT,
  event_name TEXT NOT NULL,
  impact_level INTEGER,
  actual_value REAL,
  consensus_value REAL,
  previous_value REAL,
  surprise_magnitude REAL,
  actual_vs_consensus TEXT,     -- 'beat', 'miss', 'inline'
  move_5min_pips REAL, move_30min_pips REAL,
  move_1h_pips REAL, move_4h_pips REAL,
  retrace_pct REAL
);

CREATE TABLE economic_calendar (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  event_timestamp TEXT NOT NULL,
  currency TEXT,
  event_name TEXT NOT NULL,
  impact_level INTEGER,         -- 1-5
  consensus_value REAL,
  previous_value REAL,
  source TEXT                   -- which source provided this
);

Reference / slow tables:

CREATE TABLE cot_data (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  pair TEXT NOT NULL,
  as_of_tuesday TEXT NOT NULL,  -- always store which Tuesday this covers
  published_friday TEXT,
  commercial_net_position INTEGER,
  non_commercial_net_position INTEGER,
  position_change_week_over_week INTEGER,
  extreme_positioning_percentile_52w REAL
);

CREATE TABLE cme_open_interest (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  pair TEXT NOT NULL,
  timestamp TEXT NOT NULL,
  open_interest INTEGER,
  open_interest_change_pct REAL
);

CREATE TABLE session_summaries (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  date TEXT NOT NULL,
  session TEXT NOT NULL,        -- 'asian', 'london', 'ny', 'overlap'
  pair TEXT NOT NULL,
  open_price REAL, close_price REAL, high_price REAL, low_price REAL,
  range_pips REAL, range_vs_avg REAL,
  direction TEXT, tick_volume INTEGER,
  is_holiday INTEGER DEFAULT 0
);

CREATE TABLE economic_data (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  timestamp TEXT NOT NULL,
  data_type TEXT NOT NULL,       -- 'yield', 'equity', 'commodity', 'vix', 'dxy'
  source TEXT NOT NULL,          -- 'FRED', 'Yahoo', 'Twelve Data'
  series_id TEXT,
  value REAL,
  pair_context TEXT              -- which forex pair this most influences
);

CREATE TABLE currency_strength (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  timestamp TEXT NOT NULL,
  currency TEXT NOT NULL,
  strength_roc REAL,
  strength_zscore REAL
);

CREATE TABLE carry_data (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  date TEXT NOT NULL,
  pair TEXT NOT NULL,
  swap_long REAL, swap_short REAL,
  net_carry REAL, carry_direction TEXT,
  carry_annualized_pct REAL, carry_change_5d REAL, carry_rank INTEGER
);

CREATE TABLE cb_speech_events (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  timestamp TEXT NOT NULL,
  bank TEXT NOT NULL,            -- 'FED', 'ECB', 'BOJ', 'BOE', etc.
  speaker TEXT,
  sentiment_score REAL,          -- -1.0 to +1.0
  key_terms TEXT,                -- JSON array
  source_url TEXT
);

CREATE TABLE cb_rate_decisions (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  decision_timestamp TEXT NOT NULL,
  bank TEXT NOT NULL,
  outcome TEXT NOT NULL,         -- 'hike', 'cut', 'hold'
  amount_bp REAL,
  matched_consensus INTEGER,
  surprise_direction TEXT,
  surprise_magnitude_bp REAL
);

CREATE TABLE system_health (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  timestamp TEXT NOT NULL,
  check_type TEXT NOT NULL,
  pair TEXT,
  timeframe TEXT,
  last_candle_timestamp TEXT,
  candle_age_minutes REAL,
  db_size_mb REAL,
  api_response_time_ms REAL,
  status TEXT,                   -- 'ok', 'stale', 'gap', 'error'
  notes TEXT
);

Sampling Frequencies

Data Type Frequency
Price candles + all calculated indicators Every candle close
OANDA order book + position book Every 15-20 minutes
OANDA spread Every 1 minute (every 5 seconds around high-impact events)
Cross-market: yields (FRED) Daily
Cross-market: equities, VIX, commodities Daily (Yahoo Finance / Twelve Data)
Currency strength Every 5 minutes
Session metrics Continuously, summarized at session close
Economic calendar Pull daily, update every 6 hours
COT data Weekly (Friday publish, Tuesday data)
CME open interest Hourly (free delayed)
Derived metrics (win rates, correlations) Daily recalculation
Forward return labels Continuous backfill, label only for candles > 30 days old
Outcome backfill (events) Every candle, hard stop at 50 candles

OANDA API Rate Limits — Critical Notes

[PANEL: Opus] Design request batching from day one. OANDA v20 REST API limits: ~100 requests/second burst. Sustained polling of 105 pair-timeframe combinations (15 pairs × 7 TF) every 5 minutes is feasible but requires careful batching. OANDA returns max ~5000 candles per request — for 5min timeframe that's only ~17 days of history. Backfill strategy required at startup.

Required implementation:

Implementation Notes

  1. All timestamps in UTC. Never store local time. Convert to ET only for session logic display.
  2. Forward-fill missing data. If API call fails, carry forward last known value. Set is_imputed = 1 flag.
  3. Confirmed ZigZag points only. Never write unconfirmed points to the database.
  4. Closed higher-timeframe periods only for ZZ validation.
  5. Outcome fields are backfilled as subsequent candles arrive. Hard cap: 50 candles, then outcome_window_expired = 1.
  6. Config file for pairs. Read list at startup from config.
  7. Error handling: if OANDA API is down, log and continue. One Telegram alert per source per hour max.

SECTION 16: CLOSED-PERIOD STRUCTURE TABLE

Tracks statistics from closed/completed periods only — never live data. Non-repainting.

Calculated per pair per timeframe (Daily, Weekly, Monthly), updated at period close.

16.1 Base Structure

16.2 Close and Open Dynamics

16.3 Candle Body Analysis

16.4 Range and Volatility Dynamics

16.5 Structural Classifications

16.6 Failure Pattern Detection

16.7 Gap Analysis

16.8 Momentum and Velocity

16.9 Volume-Based Metrics

16.10 Cross-Timeframe Alignment

16.11 Composite Scores

16.12 Advanced Derived Metrics

16.13 Storage

CREATE TABLE closed_period_structure (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  pair TEXT NOT NULL,
  timeframe TEXT NOT NULL,      -- 'D', 'W', 'M'
  period_close_timestamp TEXT NOT NULL,
  -- 16.1 Base Structure
  last_high REAL, last_low REAL, prev_high REAL, prev_low REAL,
  avg_high REAL, avg_low REAL,
  high_change_pct REAL, low_change_pct REAL,
  last_range_pct REAL, prev_range_pct REAL,
  high_struct TEXT, low_struct TEXT,
  -- 16.2 Close/Open
  last_close REAL, prev_close REAL, last_open REAL, prev_open REAL,
  close_change_pct REAL, open_change_pct REAL,
  close_position_pct_last REAL, close_position_pct_prev REAL,
  close_position_shift REAL, close_bias TEXT,
  -- 16.3 Body
  last_body_pct REAL, prev_body_pct REAL, body_expansion_pct REAL,
  body_direction_last TEXT, body_direction_prev TEXT, body_flip INTEGER,
  upper_wick_pct_last REAL, lower_wick_pct_last REAL,
  upper_wick_pct_prev REAL, lower_wick_pct_prev REAL,
  wick_ratio REAL, shadow_imbalance REAL,
  -- 16.4 Range/Vol
  range_expansion_ratio REAL, range_change_pct REAL,
  price_overlap_pct REAL, midpoint_shift_pct REAL,
  -- 16.5 Structure
  trend_structure_type TEXT, is_inside_bar INTEGER, is_outside_bar INTEGER,
  close_vs_prev_range TEXT,
  -- 16.6 Failures
  failed_new_high INTEGER, failed_new_low INTEGER,
  -- 16.7 Gaps
  gap_pct REAL, gap_type TEXT, gap_filled INTEGER, gap_persistence INTEGER,
  -- 16.8 Momentum
  net_travel_pct_last REAL, net_travel_pct_prev REAL,
  velocity_ratio REAL, momentum_label TEXT,
  high_velocity REAL, low_velocity REAL, velocity_bias TEXT,
  -- 16.9 Volume
  last_volume REAL, prev_volume REAL, volume_change_pct REAL, volume_struct TEXT,
  volume_range_efficiency REAL, effort_vs_result TEXT, body_volume_confirmation TEXT,
  -- 16.10 MTF Alignment
  dw_high_struct_agree INTEGER, dw_low_struct_agree INTEGER,
  wm_high_struct_agree INTEGER, wm_low_struct_agree INTEGER,
  dm_high_struct_agree INTEGER, dm_low_struct_agree INTEGER,
  full_alignment INTEGER, timeframe_divergence_count INTEGER,
  cross_tf_range_ratio_dw REAL, cross_tf_range_ratio_wm REAL,
  micro_close_in_macro_pct REAL, cross_tf_close_position_align TEXT,
  -- 16.11 Composite
  structure_score REAL, structure_strength TEXT, range_efficiency REAL,
  -- 16.12 Advanced
  fvg_up INTEGER, fvg_down INTEGER, volume_price_divergence INTEGER,
  emr REAL, nrr_daily_weekly REAL, nrr_weekly_monthly REAL,
  true_range_last REAL, true_range_prev REAL, true_range_change_pct REAL,
  range_atr_ratio REAL, candle_type TEXT,
  breakout_retest INTEGER, reversal_up INTEGER, reversal_down INTEGER,
  mtf_clv_agree_dw INTEGER, mtf_clv_agree_wm INTEGER,
  -- Cross-references
  sr_zone_id_at_high TEXT, sr_zone_id_at_low TEXT,
  structure_high_at_resistance INTEGER, structure_low_at_support INTEGER,
  structure_break_through_sr INTEGER, close_near_sr_zone INTEGER,
  UNIQUE(pair, timeframe, period_close_timestamp)
);
CREATE INDEX idx_structure_pair_tf ON closed_period_structure (pair, timeframe);

Updated at: Daily = 5PM ET, Weekly = Friday 5PM ET, Monthly = last trading day 5PM ET. Keep full history. Never delete. Builds the long-term pattern dataset.

SECTION 17: MULTI-METHOD S&R ZONE ENGINE

The most important cross-referencing system. Every S&R zone is always TWO prices (a range, never a single point). Both boundaries are rounded OUTWARD to the nearest .0025 increment.

This engine is asset-class agnostic. Only the round-number grid and session times change per market.

17.1 Ten Detection Methods

Method 1: Multi-Touch Zones Scan price history for levels where price bounced 2+ times. A "touch" = price within 0.3 * ATR(14), then reversed at least 1.0 * ATR(14). More touches = stronger.

Method 2: Retest Zones (Polarity Flip) Levels that acted as resistance AND later as support (or vice versa). Zone where resistance and support overlap within 0.5 * ATR(14).

Method 3: ZigZag Swing Points Confirmed ZZ (15/5) swings, HTF-validated (Set 1). Each swing point = zone centered on swing price, width = 0.25 * ATR(14) per side.

Method 4: Weekly/Daily/Monthly Highs and Lows Each completed period's high and low = candidate S&R level.

Method 5: Volume Profile / High Volume Nodes Per completed week/month: divide price range into bins (10 pips or 0.25 * ATR). Accumulate tick volume per bin. HVNs = top 10% of volume. POC = highest volume bin. VAH/VAL = range containing 70% of volume.

[PANEL: Opus] Volume profile is an approximation. OANDA provides tick volume (count of price updates) not actual traded volume. Label all fields as estimated_volume_profile. Distribute candle volume as: 1/3 to close price bin, 2/3 evenly across O-H-L range. Imperfect but still useful for identifying high-activity zones.

Method 6: Order Blocks Last opposing candle before a strong impulse (>2 ATR in one direction). Bullish OB: last red candle before big up move (zone = open-to-low). Bearish OB: last green candle before big down move (zone = open-to-high). Confirm with FVG (imbalance after impulse).

Method 7: Liquidity Sweeps / Stop Hunts Price briefly exceeds a known swing high/low (1-3 candles max) then immediately reverses. Sweep zone = from original swing to extreme of sweep candle. After sweep, zone flips: swept high becomes potential support.

Method 8: Fibonacci Levels Fib levels from significant swings (>5 ATR minimum swing length) become candidate zones. Also: Fibonacci extensions (1.000, 1.272, 1.618, 2.618) for targets.

Method 9: Round Numbers (.0025 Grid) .0000 levels strongest, .0050 next, .0025/.0075 weakest. JPY: .00, .50, .25/.75.

Method 10: Dynamic-to-Static Conversion If price bounces off SMA 20/50/100/200 or VWAP 3+ times within 20 candles, graduate that price area to a static zone. Zone = range between highest and lowest bounce points near that MA.

17.2 Zone Strength Scoring (0-100, bonuses can push above)

Factor Calculation Max Points
Touch Count 10 per touch, cap at 30. After 5+ touches, score starts declining. 30
Bounce Quality Average % move after each touch. 5% avg = max. 20
Timeframe Weight 4H +5, Daily +5, Weekly +7, Monthly +3 (stacks across TFs) 20
Volume Profile Zone at HVN/POC percentile 15
Recency 15 - (days_since_last_touch / 10) 15
Method Confluence +5 per additional method confirming same zone uncapped
Retest Bonus +10 if zone confirmed as polarity flip 10
Sweep Bonus +8 if zone from confirmed liquidity sweep 8
Structure Alignment +6 if D/W/M structure point (Section 16) coincides 6
Dynamic Confluence +5 if zone aligns with respected MA/VWAP 5
Fibonacci Confluence +5 if zone aligns with Fib level (0.5/0.618 = full, others = 3) 5
Round Number +4 big figure (.0000), +2 half (.0050) 4

Zone labels: 80+ = "Fortress", 60-79 = "Strong", 40-59 = "Moderate", 20-39 = "Weak", <20 = "Expired"

[PANEL: Sonar Reasoning Pro] Exact scoring algorithm:

MERGE zones within 0.5 * ATR distance
SCORE = (touch_count * 10, capped at 30)
      + (avg_bounce_pct * 4, capped at 20)
      + (timeframe_weight_total, capped at 20)
      + (volume_node_percentile * 0.15)
      + (15 - days_since_last_touch/10, min 0)
      + (additional_methods * 5)
      + bonuses
CAP score display at 100, but allow computed to exceed for ranking
After 5 touches: apply degradation factor: score *= 0.95^(touches - 5)

17.3 Zone Clustering and Merging

Proximity threshold: 0.5 * ATR(14) of higher timeframe.

Merge logic:

  1. Sort candidate zones by strength (descending)
  2. For each zone, check if any unprocessed zone overlaps or has midpoint within threshold
  3. Merge: zone_low = lowest, zone_high = highest
  4. Round OUTWARD to nearest .0025
  5. Merged strength = highest score + 5% per additional zone in cluster
  6. Track sources JSON array: ["multi_touch", "volume_profile", "fibonacci"]
  7. Touch count = sum of all constituent touch counts

17.4 Zone Decay and Expiration

Zone breaks:

Expiration: Archived when score < 20 AND untouched 90+ days (4H/Daily) or 180+ days (W/M). Archived zones stay in database forever but excluded from active analysis.

17.5 Timeframe Hierarchy

Multipliers: Monthly 4.0x, Weekly 2.5x, Daily 1.5x, 4H 1.0x.

17.6 Cross-Reference: S&R + Closed-Period Structure (Section 16)

At every period close, generate:

17.7 Cross-Reference: S&R + Fundamentals

17.8 Dynamic S&R (Moving Averages, VWAP)

17.9 Zone Validation

17.10 Zone Storage

CREATE TABLE sr_zones (
  zone_id TEXT PRIMARY KEY,
  pair TEXT NOT NULL,
  zone_low REAL NOT NULL,
  zone_high REAL NOT NULL,
  round_bracket TEXT,
  strength_score REAL,
  raw_score REAL,
  status TEXT DEFAULT 'Active',    -- 'Active', 'Broken', 'Flipped', 'Expired', 'Noise'
  is_flipped INTEGER DEFAULT 0,
  detection_methods TEXT,          -- JSON array
  method_count INTEGER,
  touch_count INTEGER DEFAULT 0,
  bounce_count INTEGER DEFAULT 0,
  break_count INTEGER DEFAULT 0,
  bounce_rate REAL,
  z_score REAL,
  avg_bounce_pct REAL,
  avg_penetration_pct REAL,
  source_timeframes TEXT,          -- JSON array
  highest_tf TEXT,
  tf_weight REAL,
  created_at TEXT,
  last_touch_at TEXT,
  last_decay_at TEXT,
  structure_interactions INTEGER DEFAULT 0,
  fundamental_flags TEXT,          -- JSON
  dynamic_confluence INTEGER DEFAULT 0,
  notes TEXT
);

CREATE TABLE sr_zone_events (
  event_id TEXT PRIMARY KEY,
  zone_id TEXT REFERENCES sr_zones(zone_id),
  pair TEXT NOT NULL,
  event_type TEXT NOT NULL,        -- 'touch', 'bounce', 'break', 'sweep', 'flip'
  timestamp TEXT NOT NULL,
  price_at_event REAL,
  penetration_pct REAL,
  reaction_pips REAL,
  reaction_bars INTEGER,
  volume_at_event REAL,
  structure_label TEXT,
  fundamental_context TEXT
);

CREATE INDEX idx_sr_zones_pair ON sr_zones (pair);
CREATE INDEX idx_sr_zones_status ON sr_zones (status);
CREATE INDEX idx_sr_events_zone ON sr_zone_events (zone_id);

SECTION 18: CROSS-SECTION INTELLIGENCE FLAGS

18.1 COT + Currency Strength + SuperTrend Alignment

18.2 Yield Spread Direction vs FX Direction

18.3 Order Flow Density at S&R Zones

18.4 Currency Strength Decomposition

18.5 Volatility Regime + Economic Calendar

18.6 Carry Trade + Risk Regime

18.7 Session + Volume + S&R Zone Interaction

18.8 Retail Trapped Signal

[PANEL: DeepSeek V3.1] Derived from OANDA position book + price:

SECTION 19: FUTURE — INSTITUTIONAL-GRADE DATA

When budget allows, these are the additions that separate retail from institutional analysis.

Tier 1: Options-Derived Intelligence (~$200-500/month)

Tier 2: Interbank Market Depth (~$500+/month)

Tier 3: Alternative Data

Tier 4: Advanced Analytics (free, computationally expensive)

SECTION 20: PREDICTION MARKET CONTRACTS (Secondary Venue)

[PANEL: GPT-4.1] Kalshi and Polymarket carry real-money contracts on macro/economic data events. These are secondary venues (OANDA is primary) but provide direct market-implied probabilities that complement yield spreads and COT data.

What to Track

Kalshi (primary prediction market venue):

Polymarket:

Storage

CREATE TABLE prediction_market_contracts (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  venue TEXT NOT NULL,             -- 'Kalshi', 'Polymarket'
  contract_id TEXT NOT NULL,
  event_name TEXT NOT NULL,        -- e.g., 'Fed Rate Dec 2026 - 25bp Cut'
  event_type TEXT,                 -- 'rate_decision', 'cpi', 'nfp', 'macro'
  currency_context TEXT,           -- which forex pairs this most influences
  strike_description TEXT,         -- e.g., '25bp cut', 'CPI >= 3.2'
  timestamp TEXT NOT NULL,
  yes_price REAL,                  -- probability of YES (0-1)
  no_price REAL,
  open_interest INTEGER,
  volume_24h REAL,
  event_date TEXT,                 -- when the event resolves
  resolved INTEGER DEFAULT 0,
  resolution_value TEXT
);

CREATE TABLE prediction_market_snapshots (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  contract_id TEXT REFERENCES prediction_market_contracts(contract_id),
  timestamp TEXT NOT NULL,
  yes_price REAL,
  volume_at_snapshot REAL
);

How They Feed Forex Analysis

These supplement (not replace) the CME FedWatch rate probabilities (Section 8) with real-money market-implied signals.

Collection cadence: Pull daily for all open contracts. Pull every 6 hours for contracts within 2 weeks of resolution.

SECTION 21: SYSTEM HEALTH AND MONITORING

[PANEL: Opus] Heartbeat and health monitoring is a MUST-HAVE. Without it, you'll discover the script died 3 days ago when an analyst asks why data is stale. A 24/5 data collection engine needs uptime monitoring, data freshness checks, database size tracking, disk space alerts, and a daily summary.

Health Check Implementation

-- Query: check data freshness per pair/timeframe
SELECT pair, timeframe,
  MAX(timestamp) as last_candle,
  ROUND((JULIANDAY('now') - JULIANDAY(MAX(timestamp))) * 24 * 60, 1) as minutes_ago
FROM forex_candles
GROUP BY pair, timeframe
ORDER BY minutes_ago DESC;

-- Query: database size
SELECT page_count * page_size / 1024 / 1024.0 as db_size_mb
FROM pragma_page_count(), pragma_page_size();

Daily Telegram Summary

Every day at session close (5PM ET Friday), send to EDGE TEAM group:

Gap Detection and Backfill

On startup, for each pair/timeframe:

  1. Query MAX(timestamp) from forex_candles for that pair/TF
  2. Compare to current time
  3. Calculate number of missing candles
  4. Pull them from OANDA historical candle endpoint in batches
  5. Log any gaps that couldn't be filled (market was closed, API limit hit)

SECTION 22: DATA VALIDATION RULES

[PANEL: Sonar Reasoning Pro] Explicit data validation rules prevent silent corruption.

Every incoming candle must pass these checks before storage:

Log all rejected candles to a data_validation_errors table with reason.

SECTION 23: AI PROMPT REDUCER (Data Interface Layer)

[PANEL: Sonar Reasoning Pro] The most important missing piece. The spec collects 50+ indicators across 7 timeframes on 15 pairs. At candle close, that's ~5,250 data points. Even with 128k context, filling 30-40% of context with raw numbers destroys signal-to-noise ratio.

Build a filtering/aggregation layer BEFORE collecting months of data:

-- Query: top 5 pairs by confluence strength at current moment
SELECT pair,
  multi_tf_confluence_score,
  supertrend_direction,
  hv_20_percentile_252d,
  regime_trend_strategy_preference,
  adx_14,
  rsi_14
FROM forex_candles
WHERE timeframe = '4H'
  AND timestamp = (SELECT MAX(timestamp) FROM forex_candles WHERE timeframe = '4H')
ORDER BY ABS(multi_tf_confluence_score) DESC
LIMIT 5;

-- Query: active high-confidence signals right now
SELECT pair, timeframe, flip_direction, multi_tf_alignment_count, atr_trend_at_flip
FROM supertrend_flips
WHERE flip_timestamp > DATETIME('now', '-24 hours')
  AND outcome_window_expired = 0
  AND multi_tf_alignment_count >= 4
ORDER BY flip_timestamp DESC;

-- Query: S&R zones being tested right now
SELECT z.pair, z.zone_low, z.zone_high, z.strength_score, z.status,
  c.close, c.supertrend_direction
FROM sr_zones z
JOIN forex_candles c ON c.pair = z.pair AND c.timeframe = '4H'
  AND c.timestamp = (SELECT MAX(timestamp) FROM forex_candles WHERE pair = z.pair AND timeframe = '4H')
WHERE z.status = 'Active'
  AND c.close BETWEEN z.zone_low * 0.999 AND z.zone_high * 1.001
ORDER BY z.strength_score DESC;

Reducer output format for AI analysts:

BUILD ORDER (Phased Implementation)

[PANEL: Opus — primary. Supplemented by panel consensus.]

PHASE 1 — Core Infrastructure (3-5 days) OANDA API connection, candle fetcher, SQLite database with WAL mode, config file for pairs, basic error handling and Telegram alerts. Include gap detection and backfill at startup. Prove VPS, API, and storage all work together.

PHASE 2 — Core Indicators (3-5 days) SuperTrend (10/2), ATR(14), RSI(14), SMA(20/50/100/200), MACD, Bollinger Bands, ADX/DMI, Stochastic, Keltner Channels. Calculate on candle close for 15min, 1H, 4H, Daily, Weekly, Monthly (defer 5min). Use proven library (technicalindicators npm or ta-lib).

PHASE 3 — SuperTrend Flip Detection + Outcome Tracking (2-3 days) Primary trading strategy. Flip events, outcome backfill (with hard 50-candle cap), pullback entry setups. This gives analysts their most important signal immediately.

PHASE 4 — Closed-Period Structure Table (2-3 days, Sections 16.1-16.11 only) Runs only at period closes. Low-frequency, low-risk. Base metrics only (16.1-16.11). Skip 16.12 advanced metrics — calculate retroactively from raw data in v2.

PHASE 5 — Cross-Market Data (2-3 days) Yields via FRED API (reliable, free, daily granularity). Synthetic DXY from OANDA components. VIX, gold, oil, equities via Yahoo Finance / Twelve Data daily. FRED API for CME rate probabilities.

PHASE 6 — ZigZag + Price Set 1 Trend Lines (4-6 days) Custom ZigZag from scratch with confirmation state machine. Set 1 trend lines only (tightest filter, highest quality). Sets 2 and 3 deferred.

PHASE 7 — RSI Divergence Detection (3-4 days) Depends on ZigZag (Phase 6) and RSI (Phase 2). Regular and hidden divergence with normalized strength scores and outcome tracking.

PHASE 8 — S&R Zone Engine v1 (5-7 days) Methods 1, 3, 4, 9 only (multi-touch, ZigZag swings, period highs/lows, round numbers). Build clustering, scoring, decay infrastructure properly — that's the hard part. Add methods 2, 5, 6, 7, 8, 10 in v2.

PHASE 9 — OANDA Order Book + Position Book + Spread (1-2 days) Free, unique, high-value contrarian data. Pull every 20 minutes for book, every minute for spread.

PHASE 10 — Session Tracking + Economic Calendar (2-3 days) Session ranges and event proximity. Use reliable calendar source (myfxbook RSS or investing.com). Session tracking = time-windowed aggregation of existing data.

PHASE 11 — Currency Strength + Confluence Scoring (1-2 days) Both derived from existing pair data. No new API calls. High value, low effort.

PHASE 12 — Volatility Regime (1-2 days) HV percentile, Kaufman ER, BB/KC squeeze. Calculated from existing data. Defer FDI.

PHASE 13 — Health Monitoring + Gap Detection + Database Maintenance (2-3 days) Before adding more features, make it robust. Health endpoint, daily Telegram summary, automatic gap detection, database VACUUM and index optimization.

PHASE 14 — AI Prompt Reducer (2-3 days) Build the filtering/aggregation layer that transforms 5,250 data points into 30-50 actionable signals. This is the interface the analysts actually use. Do this before analysts start complaining about noise.

V2 FEATURES (weeks to months, incremental): Remaining S&R methods (2, 5, 6, 7, 8, 10), Price Sets 2 and 3, RSI trend lines, event timing correlations (Section 3), pullback entry strategy, Section 16.12 advanced metrics, lead-lag analysis, 5-minute timeframe, COT data, seasonal patterns, prediction market integration (Section 20), CME open interest, central bank speech NLP, VWAP vs TWAP decision, carry data, cross-section intelligence flags (Section 18).

STORAGE ESTIMATES

Data Type Rows/Day Row Size GB/Year
forex_candles (15 pairs, 6 TFs, no 5min) ~2,160 ~2KB ~1.6 GB
supertrend_flips ~20-40 ~0.5KB ~7 MB
sr_zones (active) ~500 steady state ~1KB ~0.2 MB
sr_zone_events ~200 ~0.5KB ~36 MB
closed_period_structure ~45/day ~3KB ~49 MB
cross_market data ~100 ~0.5KB ~18 MB
session_summaries ~60 ~0.5KB ~11 MB
Total estimated Year 1 ~2 GB

SQLite handles 2GB comfortably. Monitor at the 3GB mark and consider partitioning by year.

KEY SQL QUERIES FOR ANALYSTS

-- Current market state for a specific pair and timeframe
SELECT timestamp, pair, timeframe,
  supertrend_direction, candles_since_last_flip,
  adx_14, rsi_14, kaufman_er,
  regime_trend_strategy_preference,
  multi_tf_confluence_score,
  hv_20_percentile_252d,
  bb_inside_kc as squeeze_active
FROM forex_candles
WHERE pair = 'EUR_USD' AND timeframe = '4H'
ORDER BY timestamp DESC LIMIT 1;

-- All active SuperTrend flips with multi-TF alignment >= 4
SELECT f.*, c.adx_14, c.hv_20_percentile_252d
FROM supertrend_flips f
JOIN forex_candles c ON c.pair = f.pair AND c.timeframe = f.timeframe
  AND c.timestamp = (SELECT MAX(timestamp) FROM forex_candles
                     WHERE pair = f.pair AND timeframe = f.timeframe)
WHERE f.outcome_window_expired = 0
  AND f.multi_tf_alignment_count >= 4
  AND f.flip_timestamp > DATETIME('now', '-7 days')
ORDER BY f.flip_timestamp DESC;

-- Win rate by timeframe for SuperTrend flips (last 90 days, minimum 10 events)
SELECT timeframe,
  COUNT(*) as total_flips,
  ROUND(AVG(CASE WHEN hit_2r = 1 THEN 100.0 ELSE 0 END), 1) as pct_hit_2r,
  ROUND(AVG(CASE WHEN hit_3r = 1 THEN 100.0 ELSE 0 END), 1) as pct_hit_3r,
  ROUND(AVG(max_favorable_excursion_atr), 2) as avg_mfe_atr
FROM supertrend_flips
WHERE flip_timestamp > DATETIME('now', '-90 days')
  AND outcome_window_expired = 1
GROUP BY timeframe
HAVING COUNT(*) >= 10
ORDER BY pct_hit_2r DESC;

-- Strongest active S&R zones near current price for a pair
SELECT z.zone_low, z.zone_high, z.strength_score, z.status,
  z.touch_count, z.bounce_rate, z.z_score,
  z.detection_methods, z.source_timeframes,
  ABS(c.close - (z.zone_low + z.zone_high) / 2) as distance_to_midpoint
FROM sr_zones z
JOIN forex_candles c ON c.pair = z.pair AND c.timeframe = 'D'
  AND c.timestamp = (SELECT MAX(timestamp) FROM forex_candles
                     WHERE pair = z.pair AND timeframe = 'D')
WHERE z.pair = 'GBP_USD' AND z.status IN ('Active', 'Flipped')
ORDER BY z.strength_score DESC
LIMIT 10;

-- Yield spread vs FX pair divergence check
SELECT
  yd.timestamp,
  yd.value as us_de_10y_spread,
  LAG(yd.value, 5) OVER (ORDER BY yd.timestamp) as spread_5d_ago,
  c.close as eurusd_price,
  LAG(c.close, 5) OVER (ORDER BY c.timestamp) as eurusd_5d_ago
FROM economic_data yd
JOIN forex_candles c ON DATE(c.timestamp) = DATE(yd.timestamp)
  AND c.pair = 'EUR_USD' AND c.timeframe = 'D'
WHERE yd.data_type = 'yield' AND yd.series_id = 'US_DE_10Y_SPREAD'
ORDER BY yd.timestamp DESC LIMIT 20;

-- Top pairs by multi-TF confluence right now
SELECT pair,
  multi_tf_confluence_score,
  supertrend_direction,
  adx_14,
  rsi_14,
  regime_trend_strategy_preference
FROM forex_candles
WHERE timeframe = '4H'
  AND timestamp = (SELECT MAX(timestamp) FROM forex_candles WHERE timeframe = '4H')
ORDER BY ABS(multi_tf_confluence_score) DESC
LIMIT 5;

-- Divergences that preceded reversals (model training data)
SELECT d.*,
  (d.outcome_20_candles / d.rsi_swing_2) as signal_quality
FROM divergence_events d
WHERE d.divergence_type IN ('regular_bullish', 'regular_bearish')
  AND d.outcome_window_expired = 1
  AND d.reversal_occurred = 1
  AND d.timeframe IN ('4H', 'D')
ORDER BY d.timestamp DESC;

-- Retail trapped signal with S&R zone context
SELECT
  f.pair, f.retail_trapped_side,
  z.zone_low, z.zone_high, z.strength_score,
  ob.stop_cluster_long_distance, ob.stop_cluster_short_distance
FROM (
  SELECT pair,
    CASE
      WHEN underwater_longs_pct > 70 THEN 'longs_trapped'
      WHEN underwater_shorts_pct > 70 THEN 'shorts_trapped'
      ELSE 'balanced'
    END as retail_trapped_side,
    underwater_longs_pct, underwater_shorts_pct
  FROM (
    SELECT pair,
      AVG(underwater_longs_pct) as underwater_longs_pct,
      AVG(underwater_shorts_pct) as underwater_shorts_pct
    FROM oanda_position_book
    WHERE timestamp > DATETIME('now', '-1 hour')
    GROUP BY pair
  )
  WHERE underwater_longs_pct > 70 OR underwater_shorts_pct > 70
) f
JOIN sr_zones z ON z.pair = f.pair AND z.status = 'Active'
JOIN oanda_order_book ob ON ob.pair = f.pair
  AND ob.timestamp = (SELECT MAX(timestamp) FROM oanda_order_book WHERE pair = f.pair)
WHERE z.strength_score > 60
ORDER BY z.strength_score DESC;

IMPLEMENTATION CHECKLIST

Phase 1: Core Infrastructure

Phase 2: Core Indicators

Phase 3: SuperTrend Events

Phase 4: Closed-Period Structure

Phase 5: Cross-Market Data

Phase 6: ZigZag + Trend Lines

Phase 7: Divergence Detection

Phase 8: S&R Engine v1

Phase 9: OANDA Books and Spread

Phase 10: Session Tracking + Calendar

Phase 11: Derived Signals

Phase 12: Volatility Regime

Phase 13: Health Monitoring

Phase 14: AI Prompt Reducer

CROSS-DESK REUSABILITY

These components from this spec are designed for reuse across all other desks:

Component Reusable for
OANDA API client (rate limiting, retry, backfill) Any future desk using a REST price API
SQLite WAL batched-write infrastructure Every desk — all use SQLite
Technical indicator library (ST, RSI, MACD, BB, ATR) Crypto, Stocks, Futures desks
Custom ZigZag with confirmation state machine Crypto (BTC/ETH swings), Stocks (index swings)
S&R zone engine (detection, clustering, scoring, decay) Every asset class — only round-number grid changes
Economic calendar fetcher + post-release tracking Stocks (earnings), Crypto (protocol events), Prediction markets
VIX, yield, equity index data (Section 8) Stocks (directly), Crypto (risk-on/off), Options
Session timing and holiday calendar Stocks, Futures, Crypto
Volatility regime classification Options, Crypto, Futures
Round number grid (.0025 for forex, adaptable to $0.25/$1 for stocks) Stocks, Futures, Options
Forward return / MFE/MAE tracking All desks
Outcome backfill with hard candle cap All desks — prevents statistics contamination
AI prompt reducer pattern All desks — every desk needs signal filtering

End of Forex Desk Merged Final Spec. Total panel cost for all rounds: ~$1.20. Panels contributing: Opus (base), Gemini 3.1 Pro, Grok 4.1, DeepSeek V3.1, GPT-4.1, Grok 4 Fast, Sonar Reasoning Pro.

Build Status (Updated 2026-03-29)

What's Built

Code Location

DB Tables

Foundation Models

8 models on sidecar (port 5050): Chronos-T5-Small, Chronos-T5-Base, Chronos-2, Kronos-mini, Kronos-base, Moirai, TimesFM 2.5, Lag-Llama (on-demand)

Proprietary Metrics

What's NOT Built Yet

TODO: Upgrade Kalshi REST to WebSocket Feed

Status: NOT BUILT — add when this desk goes live for execution

Current state: All Kalshi data (prices, order books, trades) is fetched via REST API polling on cron schedules. This is fine for edge detection and monitoring, but NOT sufficient for live trade execution.

Why WebSocket matters:

Note: This desk primarily trades on other exchanges, but Kalshi has event markets (rate decisions, index levels, crypto milestones) that overlap with this desk. WebSocket upgrade applies when trading any Kalshi markets.

Added 2026-03-29 — upgrade REST to WS when desk moves to live execution

Source: ~/edgeclaw/results/spec-panel/forex/spec-final.md