eval Command Reference¶

The eval command runs a rolling-origin backtest evaluation on a disease prediction model and exports results in NetCDF format for analysis with scientific tools.

Synopsis¶

chap eval --model-name <MODEL> --dataset-csv <CSV_FILE> --output-file <OUTPUT.nc> [OPTIONS]

Description¶

This command evaluates a single model by:

Loading disease and climate data from a CSV file
Auto-discovering GeoJSON polygon boundaries (same name as CSV with .geojson extension)
Splitting historical data into multiple train/test sets using rolling-origin backtesting
Training the model (once by default, see n-retrain) and generating probabilistic forecasts for each split
Comparing forecasts against actual observations
Exporting results (predictions, observations, metrics) to NetCDF format

The output NetCDF file can be used with plot-backtest for visualization and export-metrics for metric extraction.

Required Parameters¶

Parameter	Description
`--model-name`	Model identifier. Can be a local directory path, GitHub URL, chapkit service URL, chapkit container image (`docker://<image>`, run as a container for the duration of the command) or built-in model (`builtin:<name>`)
`--dataset-csv`	Path to CSV file containing disease data
`--output-file`	Path for output NetCDF file (use `.nc` extension)

Backtest Parameters¶

Control how the evaluation splits and processes data:

Parameter	Description	Default
`--backtest-params.n-periods`	Forecast horizon (number of periods to predict ahead)	3
`--backtest-params.n-splits`	Number of test windows to evaluate	7
`--backtest-params.stride`	Step size (in periods) between consecutive splits	1
`--backtest-params.n-retrain`	Number of times the model is trained, evenly spaced across the splits	1

Understanding Backtest Parameters¶

n-periods: How far ahead the model forecasts. For monthly data, n-periods=3 means 3-month forecasts.
n-splits: How many test windows to evaluate. More splits = more robust evaluation but longer runtime.
stride: How much to advance between splits. stride=1 means every period gets a forecast; stride=2 skips every other period.
n-retrain: How many times the model is trained. The default is 1: the model is trained once on the data before the first test window, and later splits only receive more historic data.

Splits are counted back from the end of the dataset, so the last test window ends at the final period. Example: With 36 months of data, n-periods=3, n-splits=7, stride=1: - Train once on months 1-27 - Split 1: historic data months 1-27, test on months 28-30 - Split 2: historic data months 1-28, test on months 29-31 - ... and so on until split 7, which tests on months 34-36

See How Split Indices Are Calculated for the formula.

Run Configuration¶

Control model execution environment:

Parameter	Description	Default
`--run-config.is-chapkit-model`	Set when using a chapkit REST API model	false
`--run-config.ignore-environment`	Skip automatic environment setup	false
`--run-config.debug`	Enable verbose debug logging	false
`--run-config.log-file`	Path to write log output	None
`--run-config.run-directory-type`	Directory handling: `latest`, `timestamp`, or `use_existing`	timestamp
`--run-config.runs-dir`	Directory for the per-run model working copies	`CHAP_RUNS_DIR`, or `runs/`

Run Directory Types¶

timestamp: Create a new timestamped directory for each run (recommended for tracking)
latest: Reuse the most recent run directory
use_existing: Use existing run directory without modification

Additional Options¶

Parameter	Description	Default
`--model-configuration-yaml`	Path to YAML file with model-specific parameters	None
`--historical-context-years`	Years of historical data to include for plotting context	6
`--data-source-mapping`	Path to JSON file for column name mapping	None
`--data-configuration`	Path to JSON or YAML file with data settings, such as masked ranges	None
`--dry-run`	Write data files and print commands without executing the model	false
`--strict`	Stop before running when the dataset does not meet the model's requirements, as `chap validate` reports them, or the model configuration sets options the model does not have. Without it, these are warnings	false

Dry Run¶

Use --dry-run to inspect what commands would be executed and verify data files without running the model. This writes all input data (training, historic, and future CSVs) to the run directory and prints the fully-assembled commands for each backtest split.

chap eval \
    --model-name ./my_model \
    --dataset-csv ./data.csv \
    --output-file ./eval.nc \
    --dry-run

Example output:

[dry-run] cd runs/my_model/2025-01-15_10-30-00_abc123 && uv run python train.py training_data.csv model
[dry-run] cd runs/my_model/2025-01-15_10-30-00_abc123 && uv run python predict.py model historic_data_2024-01-01.csv future_data_2024-01-01.csv predictions_2024-01-01.csv
[dry-run] cd runs/my_model/2025-01-15_10-30-00_abc123 && uv run python predict.py model historic_data_2024-02-01.csv future_data_2024-02-01.csv predictions_2024-02-01.csv
...

Each predict split writes its own date-suffixed data files, so you can inspect the exact inputs for any split. The printed commands can be copy-pasted into a terminal to run individual steps manually.

Data Source Mapping¶

When your CSV column names don't match the names expected by the model, use --data-source-mapping to provide a JSON mapping file.

Format¶

The JSON file maps model covariate names (keys) to CSV column names (values):

{
  "model_covariate_name": "csv_column_name"
}

Example¶

If your model expects a column named rainfall but your CSV has precipitation_mm:

mapping.json:

{
  "rainfall": "precipitation_mm",
  "mean_temperature": "temp_avg_celsius"
}

Usage:

chap eval \
    --model-name ./my_model \
    --dataset-csv ./data.csv \
    --output-file ./eval.nc \
    --data-source-mapping ./mapping.json

Common Use Cases¶

Adapting DHIS2 exports with different column naming conventions
Using the same model with datasets from different sources
Renaming columns without modifying the original CSV

Data Configuration¶

Use --data-configuration to mask disease_cases in periods that should not be used for training, such as COVID years or a reporting outage. On the command line the values are not deleted from the CSV; only the loaded copy is masked. (Creating a dataset with masked ranges through the REST API is different: it deletes the values inside the ranges, see Masked periods.) Inside a masked range the model receives disease_cases as empty (NaN) and an extra column target_masked set to 1. Everywhere else target_masked is 0, so a model can tell a masked period from a genuinely missing value.

Format¶

The file is JSON or YAML, in the same shape as the maskedRanges field of the REST dataset payload. Both bounds are inclusive and use the dataset's period type: months as 2021-06 or 202106, weeks as 2021W01 or 2021-W1.

data.json:

{"maskedRanges": [{"start": "2021-06", "end": "2021-08", "name": "reporting outage"}]}

data.yaml:

maskedRanges:
  - start: "2021-06"
    end: "2021-08"
    name: reporting outage

name is optional and only labels the range.

Usage:

chap eval \
    --model-name ./my_model \
    --dataset-csv ./data.csv \
    --output-file ./eval.nc \
    --data-configuration ./data.json

The same option is available on chap evaluate-ensemble (with --dataset-csv) and chap validate.

Rules¶

A range must use the same period type as the dataset: week ranges on monthly data are rejected.
Ranges may not overlap, and start may not be after end.
A masked period may not fall in the evaluation window, the last n_periods + (n_splits - 1) * stride periods. The backtest is rejected before any model runs. chap validate --data-configuration --backtest-params ... reports this too, as an error; without --backtest-params it warns for the window of the default backtest.
The CSV may not contain a target_masked column of its own; that name is reserved.
Future data passed to the model has no disease_cases. It does have target_masked, always 0, because a forecast period is never masked.

Model compatibility¶

On a masked dataset the training, historic and future data all have one extra column, target_masked. It is 1 in masked periods and 0 elsewhere, and always 0 in future data. Because the column is in both history and future, models that stack the two tables or fit on every column keep working. A model that uses every column will pick the indicator up as a covariate. That is harmless, but a model can also use it explicitly, for example to drop masked rows before fitting.

Model Types¶

Standard Models (GitHub or Local)¶

Models that follow the Chap model specification can be loaded from:

GitHub URL: https://github.com/dhis2-chap/minimalist_example_r
Local directory: /path/to/model or ./model

chap eval \
    --model-name https://github.com/dhis2-chap/minimalist_example_r \
    --dataset-csv ./data/vietnam.csv \
    --output-file ./eval.nc

Chapkit Models (REST API)¶

Chapkit models are REST API-based prediction services. Use --run-config.is-chapkit-model flag:

From a running service:

chap eval \
    --model-name http://localhost:8000 \
    --dataset-csv ./data/vietnam.csv \
    --output-file ./eval.nc \
    --run-config.is-chapkit-model

From a local directory (auto-starts the service):

chap eval \
    --model-name /path/to/chapkit-model \
    --dataset-csv ./data/vietnam.csv \
    --output-file ./eval.nc \
    --run-config.is-chapkit-model

From a container image (runs the image for the duration of the command):

chap eval \
    --model-name docker://ghcr.io/chap-models/chapkit_ghr_model:sha-84fbe77 \
    --dataset-csv ./data/vietnam.csv \
    --output-file ./eval.nc

A docker:// image is always a chapkit model, so it needs no --run-config.is-chapkit-model. Chap pulls the image if it is missing, runs it on a free local port and removes the container when the command ends, also when it is interrupted. This needs Docker.

When providing a directory path with --run-config.is-chapkit-model, Chap automatically: 1. Starts a FastAPI dev server using uv run fastapi dev 2. Waits for the service to become healthy 3. Runs the evaluation 4. Stops the service when complete

The service's own output is forwarded to Chap's debug log (--run-config.debug), and the last lines are included in the error message if the service fails to start.

Running an MLproject as a chapkit service¶

Use --run-config.as-chapkit to serve an MLproject model (a local directory or GitHub URL) with chapkit mlproject run instead of running it with Chap's runners. The model is then exercised the same way it is in production: through chapkit's REST API, with its typed user options and adapters.

chap eval \
    --model-name https://github.com/dhis2-chap/minimalist_example_uv \
    --dataset-csv ./data/vietnam.csv \
    --output-file ./eval.nc \
    --run-config.as-chapkit

Chap copies the model into the run directory, prepares its environment, starts the service on a free local port, runs the evaluation and stops the service when done. The service never registers with an orchestrator. --dry-run is not supported in this mode, since chapkit runs the model commands itself.

The model's scripts read config.yml in chap-core's layout, with the user options under user_option_values. A model whose scripts expect every option at the top level declares config_format: flat under meta_data in its MLproject (chapkit 2.4.0 or newer).

MLproject environment	What Chap does
none, or `--run-config.ignore-environment`	Runs the commands in the current environment
`uv_env`	Runs `uv sync` in the run directory and starts chapkit through `uv run`
`renv_env`	Runs `renv::restore()` before starting the service
`docker_env`, `conda_env`, `python_env`	Not supported

For a model with a docker_env, start a chapkit image yourself and pass its URL instead:

docker run -p 8000:8000 -v $(pwd):/work ghcr.io/dhis2-chap/chapkit-r-inla-cli
chap eval --model-name http://localhost:8000 --dataset-csv ./data/vietnam.csv --output-file ./eval.nc

Input Data Format¶

CSV File Requirements¶

The CSV file must contain:

Column	Description
`time_period`	Time period identifier (ISO format: YYYY-MM for monthly, YYYY-Www for weekly)
`location`	Location identifier matching GeoJSON feature IDs
`disease_cases`	Number of disease cases (target variable)
Climate covariates	Model-specific columns (e.g., `rainfall`, `mean_temperature`)
`population`	(Optional) Population for the location

Example CSV:

time_period,location,disease_cases,rainfall,mean_temperature,population
2020-01,region_1,45,120.5,28.3,50000
2020-01,region_2,32,98.2,27.1,35000
2020-02,region_1,52,145.8,29.1,50000
...

GeoJSON Auto-Discovery¶

Chap automatically looks for a GeoJSON file with the same base name as the CSV: - CSV: vietnam_data.csv - GeoJSON: vietnam_data.geojson (auto-discovered in the same directory)

The GeoJSON features must have id properties matching the location values in the CSV.

Output Format¶

The output is a NetCDF file containing:

Predictions: Probabilistic forecasts at multiple quantiles (0.025, 0.25, 0.5, 0.75, 0.975)
Observations: Actual disease case counts
Metadata: Model name, version, configuration, evaluation parameters
Dimensions: time, location, quantile, split

Use plot-backtest to visualize and export-metrics to extract metrics from the output file.

Examples¶

Basic Evaluation¶

chap eval \
    --model-name https://github.com/chap-models/chap_auto_ewars \
    --dataset-csv ./example_data/laos_subset.csv \
    --output-file ./eval_ewars.nc

Custom Backtest Parameters¶

chap eval \
    --model-name ./my_model \
    --dataset-csv ./data.csv \
    --output-file ./eval.nc \
    --backtest-params.n-periods 6 \
    --backtest-params.n-splits 12 \
    --backtest-params.stride 2

With Model Configuration¶

chap eval \
    --model-name https://github.com/dhis2-chap/minimalist_example_r \
    --dataset-csv ./data.csv \
    --output-file ./eval.nc \
    --model-configuration-yaml ./model_config.yaml

Dry Run¶

chap eval \
    --model-name ./my_model \
    --dataset-csv ./data.csv \
    --output-file ./eval.nc \
    --dry-run

Debug Mode with Logging¶

chap eval \
    --model-name ./my_model \
    --dataset-csv ./data.csv \
    --output-file ./eval.nc \
    --run-config.debug \
    --run-config.log-file ./evaluation.log

Complete Workflow¶

# Step 1: Evaluate the model
chap eval \
    --model-name https://github.com/chap-models/chap_auto_ewars \
    --dataset-csv ./data/vietnam.csv \
    --output-file ./results/ewars_eval.nc \
    --backtest-params.n-splits 10

# Step 2: Generate visualization
chap plot-backtest \
    --input-file ./results/ewars_eval.nc \
    --output-file ./results/ewars_plot.html

# Step 3: Export metrics
chap export-metrics \
    --input-files ./results/ewars_eval.nc \
    --output-file ./results/metrics.csv

plot-backtest - Generate visualizations from evaluation results
export-metrics - Export and compare metrics across evaluations

eval Command Reference¶

Synopsis¶

Description¶

Required Parameters¶

Backtest Parameters¶

Understanding Backtest Parameters¶

Run Configuration¶

Run Directory Types¶

Additional Options¶

Dry Run¶

Data Source Mapping¶

Format¶

Example¶

Common Use Cases¶

Data Configuration¶

Format¶

Rules¶

Model compatibility¶

Model Types¶

Standard Models (GitHub or Local)¶

Chapkit Models (REST API)¶

Running an MLproject as a chapkit service¶

Input Data Format¶

CSV File Requirements¶

GeoJSON Auto-Discovery¶

Output Format¶

Examples¶

Basic Evaluation¶

Custom Backtest Parameters¶

With Model Configuration¶

Dry Run¶

Debug Mode with Logging¶

Complete Workflow¶

Related Commands¶

See Also¶