Case Study
FIP: Financial Insights Pipeline
Ingests ARK ETF holdings and SEC 13F filings, flags NEW/EXIT/INCREASE/DECREASE position changes, and warehouses them to Parquet.
Executive Summary
The Financial Insights Pipeline (FIP) gathers institutional equity holdings from ARK Invest daily ETF CSVs and SEC EDGAR Form 13F filings, normalizes them into one schema, and computes position-change events (NEW, INCREASE, DECREASE, EXIT). Reproducible and open.
Problem & Constraints
Holdings data are inconsistent (tag variants, malformed tables), fragmented (XML/HTML/TXT/CSV), and lagged (13F quarterly). Manual workflows break and don't replicate. FIP removes the manual scraping step for ML feature engineering and academic research.
Architecture
Scheduler → Source Discovery → Downloader → Parsers → Validation → Transformer → Diff Engine → PostgreSQL + Parquet → REST API → Dashboard.
Methodology
- Ingested SEC EDGAR 13F-HR/A via company-submissions JSON and ARK ETF daily CSVs (ARKK, ARKG, ARKQ, ARKW, ARKX)
- Normalized to unified schema with CUSIP/ticker mapping and amendment supersession handling
- Implemented deterministic change-detection: NEW, EXIT, INCREASE, DECREASE with configurable thresholds
- Built compliance-aware ingestion with rate limiting, exponential backoff, and auditable lineage
Results & Metrics
| Metric | Target |
|---|---|
| Parse success | ≥95% on latest 4 quarters |
| Ingest latency | under 30 min from public posting |
| Change types | NEW / EXIT / INCREASE / DECREASE |
| Downstream | CHF institutional-flow features |
Tech Stack
Python, PostgreSQL, Parquet, FastAPI, SQLAlchemy, BeautifulSoup, lxml, Docker, pytest
Future Work
Public GitHub release, Zenodo data snapshot DOI, JOSS software paper, stonyfinlab.org dashboard integration.