Datasets
Paths follow vdc.data.paths.data_root():
$DATA_ROOT if set, else $OUTPUT_BASE/datasets,
else <repo>/data. Loaders such as
vdc.data.tabular.maybe_load_uci(name, data_root) read
<data root>/uci/<name>.npz (arrays
train and test).
Install the optional readers first:
pip install -e ".[data]" # pandas (GAS is a pandas pickle) and openpyxl (CCPP is an Excel workbook)
UCI density benchmarks (MAF splits)
The paper's POWER, GAS, HEPMASS and MINIBOONE results use the
preprocessed train/validation/test splits of Papamakarios et al. (2017),
Masked Autoregressive Flow for Density Estimation. They are
rebuilt from the authors' Zenodo archive (record 1161203,
data.tar.gz, 856,664,546 bytes, CC BY 4.0) with the
preprocessing of gpapamak/maf
(datasets/*.py):
python scripts/download_uci_datasets.py # download and prepare all four
python scripts/download_uci_datasets.py --archive /path/data.tar.gz # or use a local copy
export DATA_ROOT=<data root>/uci_maf # point the loaders at the MAF splits
The script checks the archive (sha256
e56bef4c...7d353d3) and each member it reads, extracts only
those members, and writes
<out>/uci/{power,gas,hepmass,miniboone}.npz (float64
arrays train, validate, test) and
<out>/uci/manifest.json. The default
<out> is <data root>/uci_maf; the
paper's runners use train and test.
| Dataset | Train | Validate | Test | d |
|---|---|---|---|---|
| POWER | 1,659,917 | 184,435 | 204,928 | 6 |
| GAS | 852,174 | 94,685 | 105,206 | 8 |
| HEPMASS | 315,123 | 35,013 | 174,987 | 21 |
| MINIBOONE | 29,556 | 3,284 | 3,648 | 43 |
Every output array is compared with the corresponding array of the
paper's runs (SHA-256 of its bytes); the script stops on a difference
unless --allow-hash-mismatch is given. The recorded hashes
were reproduced bit for bit with NumPy 2.2.6 and pandas 2.3.3; other
library versions could change the last bits of the standardized values.
Preparing all four takes about one minute and 1.2 GB of memory.
The GAS file in the archive is a pandas pickle. Unpickling runs code from the file, so the script reads it only after its hash has matched the recorded one.
Combined Cycle Power Plant (CCPP)
The Combined Cycle Power Plant data (UCI dataset 294; 9,568 rows,
d=5) are a different dataset from MAF POWER. Earlier versions of
scripts/download_uci_datasets.py saved them as
uci/power.npz; results labelled "Power" that were computed
from that file used CCPP. The paper's sampling-quality and imputation
experiments use CCPP and label it as such. Prepare it with:
python scripts/download_uci_datasets.py --ccpp # writes <data root>/uci/ccpp.npz
python scripts/download_uci_datasets.py --ccpp --ccpp-archive /path/CCPP.zip
The processing is unchanged from the earlier downloader: the
workbook's active sheet of Folds5x2_pp.xlsx, float32, and
an 80/20 split with the legacy NumPy seed 42. The script checks the
workbook (sha256 ccd49098...396f4651; the paper used
CCPP.zip with sha256 cc7b2a49...8c7a) and
compares the arrays with those of the earlier file. It never writes
power.npz, and it refuses to replace an existing output
whose arrays differ unless --force is given.
S&P100 returns
The S&P100 experiments read
<data root>/finance/sp100_returns.npy (daily returns
of 100 stocks, shape (T, 100)). The price data are not
redistributed; the
paper symbol list lists the symbols and date range used in the
paper.