Sync market data to the analytics database

This guide assumes you finished Getting Started. After it, two plugins are already running and syncing your broker's market data into the database:

Ticker Universe

Saves every contract your broker gives you access to into the database: stocks, options, and futures. When it is running, the plugin list shows Connected.

The Ticker Universe row in the plugin list with status Connected
Quote Sweep

Once the universe is synced, Quote Sweep loops through every contract and saves its latest live quote to the database. Each pass starts over from the first contract. While it runs, the plugin list shows Fetching and its progress through the current pass.

The Quote Sweep row in the plugin list with status Fetching and its progress through the current pass

Interactive Brokers: set up the Client Portal Gateway

Ticker Universe needs the IBKR Client Portal Gateway to fetch the full contract list.

  1. Install a Java runtime.
  2. Download the Client Portal Gateway from IBKR (Web API quick start).
  3. Unzip it into ~/Library/Application Support/deltaforge/plugins/user.ibkr_provider/data
  4. Open the IBKR Status Panel. Check that Gateway Install shows Discovered and Java JVM shows Found.
  5. Click Open login and sign in. When Session shows Authenticated, the gateway is ready.
The IBKR Status Panel showing Gateway Install, Java JVM, and Session rows with an Open login button

When the gateway signs in with the same IBKR username as a running TWS session, it takes over that username's market data connection, and TWS stops receiving market data. The DeltaForge IBKR plugin gets market data through TWS, not through the gateway, so reclaim the connection in TWS by clicking reconnect. The gateway keeps serving the contract list afterwards. Creating a second IBKR username dedicated to the gateway avoids the takeover.

Charles Schwab

No extra setup. Ticker Universe and Quote Sweep work once the Schwab plugin is connected.

Sync bulk historical data

Plan for disk space. Daily price history for stocks and macro data is relatively small. Intraday data and bulk options data are what make the database large: expect 500 GB or more if you sync them. If your Mac's internal drive doesn't have that much free space, put the database on a fast external SSD or flash drive.

Universe History Sync

Universe History Sync fetches daily price history through your broker for every contract in the ticker universe, plus implied volatility where the contract has listed options. It runs every 12 hours, and an agent can also start it on demand.

It is off by default. To start it:

  1. Open Settings, then Plugins, then Universe History Sync.
  2. Under History, turn on Enabled.
  3. Set Stock Bar Type to match your broker: Split and Dividend Adjusted (IBKR), the default, or Split Adjusted (Schwab).
  4. Optionally, change how far back it goes. History Start (years) is the years of daily history kept for each contract: any whole number of 1 or more, default 20. Dated Futures History (years) is the years kept for each dated futures contract: any whole number of 1 or more, default 3, and never earlier than History Start.
  5. Click Save.
Universe History Sync settings

Schwab users: Schwab's historical stock bars are split adjusted only, not dividend adjusted, and Schwab has no historical bid/ask quotes or short borrow costs. That makes Schwab data close to impossible to build a valid backtest with. For backtesting, get access to a more complete set of historical data through another provider.

Move the database to an external drive

  1. Open Settings, then Plugins, then DuckDB Analytics Store.
  2. Set Database Path to a folder named exactly deltaforge_duckdb_data on the external drive, for example /Volumes/external-drive1/deltaforge_duckdb_data. The macOS sandbox only lets the database live inside a folder with that name.
  3. Click Save, then quit DeltaForge.
  4. Move your existing database file, analytics.duckdb, from ~/Library/Application Support/deltaforge/plugins/user.duckdb_store/data/deltaforge_duckdb_data/ into the new folder. If an analytics.duckdb.wal file sits next to it, move that too.
  5. Start DeltaForge.
DuckDB Analytics Store Database Path setting

Sync free economic and market data

The FRED bundle installs with your broker plugins during Getting Started. Every other free data bundle installs from the Store:

  1. Open the Store tab, then Browse.
  2. Select the bundle, for example BLS.
  3. Click Subscribe, then Install.

Each free data bundle has two plugins:

Sync plugins are off by default. To start one, open Settings, then Plugins, then the Sync plugin. Turn on Enabled, then click Save. Some sources also need something set in the Provider's settings first.

To see a sync's progress, open the Plugin Health tab. Each Sync plugin's row has a status line that says what it is loading right now.

FDIC and FFIEC Provider and Sync plugin rows on Plugin Health, with their status lines
Sync pluginWhat it loadsWhat its Provider needs
BEA SyncNational, industry, regional and international accountsNothing for the bulk files (national, fixed assets, regional, industry). An API Key, free from BEA, is needed for the international and multinational datasets.
BLS SyncWhole surveys: CPI, employment, wages, PPI, JOLTS, productivity, ECI and trade prices, with revision historyA Contact Email (Bulk Downloads). No API key is needed.
Census SyncEconomic Indicator time series (21 programs)A Census API Key, free from Census.
CFTC SyncWeekly Commitments of Traders reports per marketNothing
EIA SyncEnergy bulk datasets: series, observations and categoriesNothing
FDIC SyncBank directory and quarterly bank financials, linked to tickersNothing
FFIEC SyncPer-bank Call Report detail, plus optional UBPR ratiosNothing
Fiscal Data SyncEvery U.S. Treasury Fiscal Data table, with revision historyNothing
FRED SyncCurated FRED series plus full ALFRED vintage historyA FRED API Key, free from FRED.
OCC Market Data ProviderOptions and futures volume, open interest, series, FLEX and position limit reportsNothing
PortWatch Sync18 port and trade data layersNothing
SEC EDGAR SyncFinancial statements, insider data, optional 13F and N-PORT, and company submissionsA Contact (User-Agent). SEC EDGAR Sync also needs Ticker Universe running.

OCC is the exception: its provider and sync are the same plugin, OCC Market Data Provider, so there is no separate Sync plugin.

SEC EDGAR Sync's optional Form 13F and N-PORT data is large: about 460 MB and 23 million rows per quarter. See Plan for disk space above before turning it on.

Once data is syncing, you can explore it with Scanner and Studies, or ask the in-app agent to query the database directly.

Next: explore your data with Scanner and Studies