Python environment for systematically processing financial data and building pipelines to feed it to models. The aim of the project is to joinly develop robust data pipelines which lead to standadized benchmarks in quantitative finance.
Principles:
- Data Handling Pricinple & Guidelines
- Conding Guidelines [not written yet]
- Model Benchmarking [not written yet]
You will need to
- Install the environment
- Use environment variables to set your caching directories.
- Register your license for each data source.
Code generated docuemntation is still in progress; it can be found here https://hansbuehler.github.io/quant_data.
Once cloned, enter into the quant_data directory and install the environment:
python -m pip install -U pip pip-review
python -m pip install -U -e ".[dev]" For no-frills functionality set
export QUANTDATA_ROOT=~/quantdata_data/
export WRDS_USERNAME=<wrds_username>
export MASSIVE_TOKEN=<massive_token>Run e.g. quantdata/widb/viz_day.ipynb to test WRDS Option Metrics or quantdata/massive/viz_bars.ipynb for a Massive example.
You can set more granular data directories, see below.
This project targets Python 3.12 but should run under 3.13.
Under Linux, if your default Python is not 3.12, you can install a different version with
sudo apt update
sudo apt install python3.12 python3.12-venv Then create a python environment:
python3.12 -m venv .vqdata or
python -m venv .vqdata Then activate and install dependencies:
source .vqdata/bin/activate # activate it
python --version
python -m pip install -U pip pip-review
python -m pip install -U -e ".[dev]" Use .[dev] for the full local development toolchain. It includes testing and documentation tools.
Under windows you can use the Windows Store to install Python. Once Python 3.12 is installed create the environment:
py -3.12 -m venv .vqdata or, if you have only one python installed,
python -m venv .vqdata Then follow up with
.vqdata\Scripts\Activate
python --version
python -m pip install --upgrade pip pip-review
python -m pip install -U -e .[dev] Use .[dev] for the full local development toolchain. It includes testing and documentation tools.
Using Anaconda, the simplest path is a local conda environment pinned to Python 3.12:
conda create --prefix .\.vqdata python=3.12
conda activate .\.vqdata
python --version
python -m pip install -U pip pip-review
python -m pip install -U -e .[dev] Use .[dev] for the full local development toolchain. It includes testing and documentation tools.
If conda plugin warnings interfere with environment creation, retry with:
conda create --prefix .\.vqdata python=3.12If you prefer uv, create the local environment with
uv sync --extra dev
source .venv/bin/activate
python --versionThis will create a local .venv and install the project with the full development toolchain.
If you do not want to activate the environment, you can run commands with uv run, e.g. uv run pytest.
After installation, the package should be importable from the active environment:
python -c "import quantdata; print(quantdata.__version__)"The project environment is organised around two main inputs: data storage/caching locations on one hand side, and access tokens to data sources on the other.
The project is organized around proactive caching of:
- raw data: downloaded from a data source. Such data can be stored in slower drives. The point of the cache is to reduce download overhead and intensity.
- processed data: enriched data. Can also be stored in a slower drive location. This data is cached because processing itself is slow.
- training data: data stored in a format optimized for training a particular model class. This should be stored at a location which allows for fast access during training, i.e. the local SSD drive.
- experiment data generated during experiments, e.g. losses, metric, weights and biases. For example, place WandB data under this root directory.
Please delete all data whose license expires.
Caching in is version-aware. You will note that a number of functions have a @version or @cache decorator. The version of a function often also depends on other
function versions. A change to the version will trigger a refesh of any cache affected by the respected function - so be careful, as this can be slow; also do not change the version of a function or class
without good reason as it may trigger refreshs for others once they check out.
Chaning a core function - e.g. massive's client version - will trigger a
redownload and recalculation of everything you have on disk. (If such hard refresh is ever required, either just delete all data or set the environment variable QUANTDATA_CACHE_MODE to "update", see also below.)
Please also check the documentation for @version and @cache with the package cdxcore.
You will need to set your directory and data access variables.
All directory definitions follow the cdxcore.subdir notation, i.e. you can refer to the temporary directory by "!" and to your home directory by "~". Note that under Linux data stored in /tmp/ are deleted upon reboot.
If you do not need to/can/want to split data across storage locations, you can just set:
QUANTDATA_ROOT: if you have everything in one place anyway, use this to set all of the above in one step.
To optimize access speed, you can split each area by device:
-
QUANTDATA_RAW_DATA: root directory for storing raw cached data. Usually does not need to be fast, e.g. a network drive. Defaults toQUANTDATA_ROOT/raw. -
QUANTDATA_PROCESSING_DATA: intermediate processing: good for multi-process CPU access, but does not need to deliver to GPU. Defaults toQUANTDATA_ROOT/processing. -
QUANTDATA_TRAINING_DATA: training data with the most effective disk to GPU location. Defaults toQUANTDATA_ROOT/training. -
QUANTDATA_EXPERIMENTS: where to store data for experiments, e.g. losses, metrics, weights and biases locally. Defaults toQUANTDATA_ROOT/experiments. -
QUANTDATA_TEMP_DATA: local ephermal caching. Very useful for exceedingly slow processes. Not used at the moment. Defaults to!/quantdata.
In Linux, setting the environment variables can be achieved by adding the following to ~/.bashrc:
# Data directories
export QUANTDATA_RAW_DATA=/path/to/data/raw
export QUANTDATA_PROCESSING_DATA=/path/to/data/processing
export QUANTDATA_TRAINING_DATA=/fast_path/to/data/training
export QUANTDATA_EXPERIMENTS=/fast_path/to/data/trainingAlternatively, you can set a global directory. Example here to make it relative to $HOME:
export QUANTDATA_ROOT=~/quantdata_data/Each data source support by our project will require you to register the respective login data, also as environment variables. At this point the following two sources are supported:
-
- Ensure you have an account and test it.
- Set the
MASSIVE_TOKENenvironment variable to your API token.
-
WRDS Option Metrics including Option Metrics
- Create an account and enable MFA.
- Follow the WRDS instructions to ensure you have
the
pgpasspass file. This is important. - Set the
WRDS_USERNAMEvariable to your username.
-
- Ensure you have an account and test it.
- Set the
MASSIVE_TOKENenvironment variable to your API token.
QUANTDATA_CACHE_MODE: sets the globalcdxcore.subdir.CacheMode. For example, to turn off all caching useQUANTDATA_CACHE_MODE="off". To update all cached data used, use ``QUANTDATA_CACHE_MODE="update".QUANTDATA_TEST_MODE: global flag to tell the project we are in testing mode. In this mode, no data is downloaded from any data source.
The pytest test suite enables test mode automatically by setting QUANTDATA_TEST_MODE=1 in
tests/conftest.py before test modules are imported. This avoids access to external data sources during testing.
Run the test suite from the project root with:
python -m pytest
- Open the repository folder in VS Code.
- Select the Python interpreter for the workspace.
- Use the local
.vqdataenvironment once it has been created with Python 3.12. - Open the Command Palette and run
Python: Select Interpreterif needed. - Open the Testing view in the left sidebar.
- If VS Code prompts to configure Python tests, choose
pytest. - When it asks for the test folder, choose
tests.
The workspace already includes .vscode/settings.json, so VS Code should use pytest with the tests directory automatically.
After configuration, you can run tests in any of these ways:
Testing sidebar -> Run All Tests
Command Palette -> Python: Run All Tests
Editor gutter -> Run Test on an individual test function
Terminal -> python -m pytest
If test discovery is empty, the usual fix is:
- Confirm the selected interpreter is the
.vqdataenvironment. - Install the test dependencies with
python -m pip install -e .[dev]. - Run
Python: Refresh Testsfrom the Command Palette.
Install the Git hook runner and register the local pytest pre-commit hook with:
python -m pre_commit install
GitHub Actions also runs the test suite on pushes to main and on pull requests.
Code generated docuemntation is still in progress; it can be found here https://hansbuehler.github.io/quant_data.
Please make an effort to document any files you commit. The project uses reStructuredText which is tedious but the standard. Here is a comprehensive introduction.
You build the documentation with
cd docs
make htmlThe reStructuredText entry point is docs/index.rst.