Chidi cont work

[Zino-ctrlZ]

QuantTools: EventDriven backtest, risk manager, and datamanager overhaul

Branch: CHIDI-CONT-WORK
Scope: 144 files changed · ~19.5k insertions · ~5k deletions

Summary

This PR is a large integration branch that modernizes the EventDriven backtesting stack, replaces legacy order-selection paths with a scoring-based picker, adds position lifecycle COGs (PnL monitor, Donchian, mean reversion, vectorized sizing), and brings substantial datamanager/cache improvements—including negative caching for option spot gaps. It also adds vectorized option PnL retrieval/runner tooling, refactors position attribution into a dedicated module, and expands developer workflow docs (Copilot skills, docstring standards, MCP/editor config).

---

EventDriven backtest & execution

• OptionSignalBacktest now mixes in BacktestRunMixin, wires a shared LiquidityPolicy, and adds pre/post signal analysis hooks plus finalize-close signal injection on the last backtest day.
• Holiday handling: equity-strategy start dates on US holidays roll back to the prior business day.
• eq_strategy property propagates strategy changes to risk manager and portfolio after initialization.
• SimulatedExecutionHandler refactor:
  • Configurable slippage modes (randomized, fixed, pct_of_spread, none) via ExecutionHandlerConfig.
  • Liquidity-level-2 spread gating can drop orders before fill.
  • Fill tracking (spread, commission, slippage model) with get_fill_tracking_df().
  • Default commission updated to contract-unit semantics (~$0.35/contract).
• New LiquidityPolicy shared across backtest, risk manager, and execution.
• New EventDriven/logging.py for package-wide logger level control.

---

Position attribution

• Replaces attributor.py with new attribution.py (~587 lines).
• Uses xmultiply_attr_v2 payload loading for option PnL decomposition.
• Adds PositionAttributionAnalyzer, trade quantity time series normalization, and trade/signal/portfolio aggregation helpers.
• Fixes direction-aware sign on trade quantity in attribution math.

---

Risk manager & order picker

Legacy removal (breaking)

• Deleted: _order_validator.py, _orders.py, and monolithic market_data.py.
• Legacy schema-driven order paths now hard-fail with compatibility errors; runtime selection is scoring-only.

Scoring-based order selection

• New chain_scoring.py — Gaussian-style moneyness/DTE scoring with tilt options.
• New iv_helper.py and picker/utils.py supporting chain filtering and scoring columns.
• OrderPicker rewritten around _score_chain, preset-order honor path, and flexible filter_contracts API.
• ScoringConfigs added; config export/validation aligned to scoring-only flow.

Position COGs (new / expanded)

Module	Purpose
pnl_monitor.py	Profit lock, roll, optional stop-loss; tick-cash scaling on profitable new orders
donchian_cog.py	Donchian-channel position management
mean_reversion.py	Mean-reversion sizing / management cog
plain_sizing.py	Plain sizing cog with configurable rules
vectorized.py	Vectorized position analysis integration

• PnLMonitorConfig moved to configs.core with YAML export support.
• Business-day arithmetic for cog action effective_date; limits metadata and special dividends loaded from special_dividends.yaml.

Market data & timeseries

• market_timeseries.py expanded to absorb deleted market-data responsibilities; continuous dividend adjustment for chain Greeks.
• Cache instances gain configurable size_limit; temp-cache isolation improved in _vars.py.
• OPTION_TIMESERIES_START_DATE centralized from trade.optionlib.config.defaults.

---

Portfolio & strategy

• new_portfolio.py — trade timeseries lookup, trade aggregation API (multiple granularities), preset-order support, improved PnL consistency and state tracking.
• strategy.py / eventScheduler.py — signal ID propagation, conflict handling, empty-queue semantics, type-hint cleanup.
• BacktestRunMixin on portfolio, risk manager, and backtest timeseries for shared pre-run lifecycle.

---

trade / datamanager

Negative cache (option spot gaps)

• classification.py (new) — classifies fetched vs missing vs checked-missing dates.
• cache.py / option_spot.py / date.py — _CachedData.checked_missing_dates prevents repeat API calls for confirmed empty ranges.
• test_negative_cache.py — manual verification script for the negative-cache flow.

Other datamanager changes

• Dynamic CacheSpec path resolution; global cache toggle and guarded write paths.
• get_div_schedule optional filter_specials; duplicate CACHE_NAME registry check disabled.
• Tighter cached date bounds for queries; date sync improvements.

---

trade / backtester & assets

• New vectorized pipeline:
  • option_vectorized_retrieval.py — contract retrieval for vectorized runs.
  • option_vectorized_pnl_runner.py — retrieval → PnL → trade/signal/portfolio aggregation + equity curve.
  • xmultiply_attr_v2.py — payload-driven option attribution loading.
• Calculate.py major refactor (~1.4k line delta); rates migrated to get_risk_free_rate_helper_v2.
• trade/backtester_/utils/aggregators.py expanded aggregation utilities.
• trade/helpers/helper.py — cache helpers, date utilities, type cleanup in helper_types.py.
• trade/__init__.py — Plotly plotting backend default; minor import hygiene.

---

Config & tooling

• EventDriven/configs/core.py — new config types: LiquidityConfig, ExecutionHandlerConfig, ScoringConfigs, PnlMonitorConfig, VectorizedCogConfig, PlainSizingCogConfig, MeanReversionSizerConfigs, validation mixin updates.
• .github/copilot-instructions.md — commit strategy and strict docstring policy.
• New skills: commit-strategy, docstring-standards, tfp-db.
• .vscode/mcp.json, settings updates; ruff.toml ignores F541, B905.
• EventDriven/Testing/ → EventDriven/tests/ rename with golden regression data (async/process/thread run outputs).

---

Test plan

• Run EventDriven golden tests under EventDriven/tests/ (process, thread, async runners) and compare CSV/txt outputs.
• Run trade/tests/test_option_vectorized_retrieval.py and test_option_vectorized_pnl_runner.py.
• Run module_test/test_trade_pnl_consistency.py, test_position_cache.py, test_xmultiply_dual_mode.py.
• Smoke-test scoring-only OrderPicker with a known strategy config; confirm legacy schema paths raise as expected.
• Verify PnL monitor cog thresholds (profit lock, roll, stop-loss) on a short backtest window.
• Run test_negative_cache.py against ThetaData (or mock) to confirm second call avoids API for checked-missing dates.
• Confirm liquidity level-2 drops wide-spread orders and fill tracking DataFrame is populated.
• Validate attribution output from PositionAttributionAnalyzer against prior attributor baselines on a fixed trade set.

---

Notes / follow-ups

• .live_forward_compatibility.txt tracks live-forward migration items (trade map load, limits DB metadata).
• Binary .DS_Store changes should not be merged; consider stripping before merge.
• chidi_agent.md is an empty placeholder agent file.