Market data integration is a product architecture problem
Adding a second or third market-data provider is not simply a matter of changing an API endpoint. Each provider has its own symbol conventions, coverage gaps, adjustment behavior, rate limits, calendars, and response formats. If those differences leak into every scanner and page, the application becomes difficult to reason about and expensive to maintain.
I prefer a provider-neutral application model with explicit adapters at the boundary. The application decides which instruments exist and how users see them. Providers supply mapped prices, fundamentals, corporate actions, and enrichment for those instruments.
Canonical symbols and provider mappings prevent silent errors
A symbol that looks correct in one system may be wrong in another. Borsa Istanbul and Spanish market suffixes, share-class punctuation, exchange codes, and benchmark identifiers are common sources of subtle errors. A direct string pass-through can produce missing prices, broken external links, or results assigned to the wrong instrument.
The safer model stores one canonical symbol identity and a mapping for each provider. Ingestion sends the provider symbol outward; scanner results, account data, URLs, and user interfaces expose the canonical symbol inward. Uniqueness constraints must match the real identity rules so one provider mapping cannot accidentally overwrite another canonical instrument.
Incremental ingestion needs overlap and observability
A naive incremental process starts exactly after the latest stored timestamp. That is fast, but it can miss corrections, late bars, split adjustments, or provider revisions. I use a bounded overlap window so a small recent period is safely upserted while old history remains untouched.
Every run should report what actually happened: requested symbols, successful symbols, failed symbols, inserted or updated rows, affected timeframes, and whether downstream feature calculations were triggered. Operational logs are not secondary; they are part of the data product because they explain why a scanner result may be absent.
- Retry transient network and throttling failures with controlled backoff.
- Continue after symbol-level failures and preserve successful work.
- Refresh features only for symbols and ranges that received new bars.
Corporate actions must be part of the analytical contract
Splits are especially dangerous in scanner performance analysis. If a setup entry price is compared with a post-split price without consistent adjustment, a losing bearish setup can appear profitable or a valid long setup can look like a collapse. Historical bars, setup observations, and forward-performance calculations must use one adjustment convention.
The same principle applies to dividends and symbol changes when they affect the intended analysis. Corporate-action handling cannot be added only to the stock chart while backtests and performance tables continue reading incompatible prices.
Optimize the read path, not only ingestion
Even a fast ingestion job does not guarantee a fast product. Scanner pages become slow when each request recalculates long rolling windows or loads an entire result universe before pagination. Aventra uses latest-feature snapshots for current scans, database-level filtering and pagination, shared connection pools, and bulk result persistence.
The most important lesson is consistency: every supported provider must follow the same contract for universe selection, symbol translation, timeframes, feature refresh, benchmarks, scanning, historical reads, and failure isolation. Otherwise a provider switch changes not only the data source but the meaning of the product.