curryer.correction.pipeline

Main correction pipeline orchestration.

This module contains the public-facing loop() function that drives the Monte Carlo parameter sensitivity analysis, plus all of the helper functions it calls:

• Adapter functions that bridge between the geolocation/image-matching sub-modules and the correction loop.

• _load_file() – internal helper that reads CSV/NetCDF/HDF5 files into DataFrames, replacing the old mission-specific loader callables.

• _load_image_pair_data(), _load_calibration_data(), _geolocate_and_match() – per-iteration computation helpers.

• loop() – outer GCP-pair loop, inner parameter-set loop.

Architecture note

The correction pipeline follows the pipeline → verification dependency direction. The core image-matching and aggregation logic lives in verification.py; pipeline.py imports it from there. This means verification is a standalone module (GCP pairing + image matching + error stats) that the correction loop reuses for its own last three steps.

Attributes

logger

Functions

call_error_stats_module(image_matching_results, setup)	Call the error_stats module with image matching output.
_resolve_gcp_pairs(→ list[tuple[str, str]])	Return the [(sci_key, gcp_key)] pair, validating that gcp_key is set.
_load_file(→ pandas.DataFrame)	Load a telemetry or science data file into a pandas DataFrame.
_load_calibration_data(...)	Load LOS vectors and optical PSF when setup.calibration is configured.
_require_image_matching_inputs(→ None)	Fail fast when the built-in image matching has no calibration to run with.
_load_image_pair_data(→ tuple[pandas.DataFrame, ...)	Load telemetry and science data for an image pair from files.
_geolocate_and_match(→ tuple[xarray.Dataset, ...)	Perform geolocation and image matching for a parameter set.
_resolve_netcdf_config(...)	Return the output's NetCDFConfig with the threshold pinned to the requirement.
loop(setup, sweep, work_dir, tlm_sci_gcp_sets[, ...])	Correction loop for parameter sensitivity analysis.
_extract_parameter_values(params)	Extract parameter values from a parameter set into a dictionary.
_store_parameter_values(netcdf_data, param_idx, ...)	Store parameter values in the NetCDF data structure.
_extract_error_metrics(stats_dataset)	Extract error metrics from error statistics dataset.
_store_gcp_pair_results(netcdf_data, param_idx, ...)	Store GCP pair results in the NetCDF data structure.
_compute_parameter_set_metrics(netcdf_data, param_idx, ...)	Compute overall performance metrics for a parameter set.
run_correction(...)	Run the correction parameter sweep.
compute_error_stats(image_matching_results, setup)	Compute error statistics from image matching results.
run_image_matching(→ xarray.Dataset)	Run image matching against GCP reference.

Module Contents

curryer.correction.pipeline.logger

curryer.correction.pipeline.call_error_stats_module(image_matching_results, setup: curryer.correction.config.GeolocationSetup)

Call the error_stats module with image matching output.

Parameters:

• image_matching_results – Either a single image matching result (xarray.Dataset) or a list of image matching results from multiple GCP pairs

• setup – GeolocationSetup with variable names and thresholds (REQUIRED)

Returns:

Aggregate error statistics dataset

curryer.correction.pipeline._resolve_gcp_pairs(sci_key: str, gcp_key: str, setup: curryer.correction.config.GeolocationSetup) → list[tuple[str, str]]

Return the [(sci_key, gcp_key)] pair, validating that gcp_key is set.

Parameters:

• sci_key (str) – Science file path for this outer-loop iteration.

• gcp_key (str) – GCP .mat file path supplied as the third element of the tlm_sci_gcp_sets tuple. Must be non-empty.

• setup (GeolocationSetup) – Unused directly; reserved for future extension.

Return type:

list of (sci_key, gcp_path) tuples — always length 1.

Raises:

ValueError – If gcp_key is empty or whitespace-only.

Notes

For spatial-overlap-based pairing (many L1A images × many GCP chips) call pair_files() before loop() to build tlm_sci_gcp_sets from the (l1a_file, gcp_file) results.

curryer.correction.pipeline._load_file(file_path: str | pathlib.Path, file_format: str = 'csv') → pandas.DataFrame

Load a telemetry or science data file into a pandas DataFrame.

Parameters:

• file_path (str | Path) – Local path or S3 URI (s3://bucket/key).

• file_format (str) – One of "csv", "netcdf", or "hdf5".

Return type:

pd.DataFrame

Raises:

• FileNotFoundError – If file_path is local and does not exist.

• ImportError – If file_path is an S3 URI and boto3 is not installed.

• ValueError – If file_format is not recognised.

curryer.correction.pipeline._load_calibration_data(setup: curryer.correction.config.GeolocationSetup) → curryer.correction.config.CalibrationData

Load LOS vectors and optical PSF when setup.calibration is configured.

This function centralizes calibration data loading, which is now called once per GCP pair in the optimized implementation (previously called once per parameter set).

Parameters:

setup (GeolocationSetup) – Setup carrying optional CalibrationFiles with direct los_vectors_file / psf_file paths.

Returns:

NamedTuple containing (los_vectors, optical_psfs), or (None, None) when no calibration files are configured.

Return type:

CalibrationData

Raises:

• FileNotFoundError – If a calibration file is configured but does not exist.

• ValueError – If a calibration file exists but fails to load properly.

Note

Calibration files are optional and interim: real LOS/spacecraft geometry will be SPICE-derived. When setup.calibration is None (or both file fields are None), returns CalibrationData(None, None) so that missions without calibration files still work.

Examples

>>> calib_data = _load_calibration_data(setup)
>>> if calib_data.los_vectors is not None:
...     # Use calibration data in image matching
...     pass

curryer.correction.pipeline._require_image_matching_inputs(setup: curryer.correction.config.GeolocationSetup, calibration_data: curryer.correction.config.CalibrationData) → None

Fail fast when the built-in image matching has no calibration to run with.

When no setup.image_matching_func override is provided, the loop falls back to the built-in image_matching(), which requires LOS/PSF calibration. Raising here — before geolocation and kernel creation — gives a clear, early error instead of a less contextual failure deep inside the loop. A custom override may legitimately need no calibration, so the check only applies when no override is set.

Raises:

ValueError – If no override is set and no calibration data is configured.

curryer.correction.pipeline._load_image_pair_data(tlm_key: str, sci_key: str, setup: curryer.correction.config.GeolocationSetup) → tuple[pandas.DataFrame, pandas.DataFrame, Any]

Load telemetry and science data for an image pair from files.

Parameters:

• tlm_key (str) – Path to the telemetry data file.

• sci_key (str) – Path to the science frame timing file.

• setup (GeolocationSetup) – Setup containing geolocation settings, file format, and time-scaling options (via setup.data_config).

Returns:

• tlm_dataset (pandas.DataFrame) – DataFrame containing spacecraft state / telemetry records.

• sci_dataset (pandas.DataFrame) – DataFrame containing science frame timing information.

• ugps_times (array_like) – Time array extracted from the science dataset (uGPS values).

Raises:

• FileNotFoundError – If the telemetry or science file does not exist.

• ValueError – If the science file is missing the required time field.

curryer.correction.pipeline._geolocate_and_match(setup: curryer.correction.config.GeolocationSetup, kernel_ctx: curryer.correction.config.KernelContext, ugps_times_modified: Any, tlm_dataset: pandas.DataFrame, calibration: curryer.correction.config.CalibrationData, image_matching_func: Any, match_ctx: curryer.correction.config.ImageMatchingContext) → tuple[xarray.Dataset, xarray.Dataset]

Perform geolocation and image matching for a parameter set.

This function loads SPICE kernels, performs geolocation, and runs image matching against GCP reference data. It’s the core computation step that combines all previous setup (kernels, data loading) into results.

Parameters:

• setup (GeolocationSetup) – Setup with geo and image matching settings

• kernel_ctx (KernelContext) – NamedTuple containing: - mkrn: MetaKernel instance with SDS and mission kernels - dynamic_kernels: List of dynamic kernel file paths - param_kernels: List of parameter-specific kernel file paths

• ugps_times_modified (array-like) – Time array (possibly modified by OFFSET_TIME parameter)

• tlm_dataset (pd.DataFrame) – Spacecraft state telemetry data

• calibration (CalibrationData) – NamedTuple containing: - los_vectors: Pre-loaded LOS vectors (or None) - optical_psfs: Pre-loaded optical PSF (or None)

• image_matching_func (callable) – Function to perform image matching; defaults to the built-in image_matching function when not overridden in config.

• match_ctx (ImageMatchingContext) – NamedTuple containing: - gcp_pairs: List of GCP pairing tuples - params: List of (ParameterConfig, parameter_value) tuples - pair_idx: Index of current GCP pair - sci_key: Science dataset identifier for this pair

Returns:

• geo_dataset (xr.Dataset) – Geolocated points with latitude, longitude, altitude

• image_matching_output (xr.Dataset) – Matching results with error measurements and metadata

Examples

>>> kernel_ctx = KernelContext(mkrn, dynamic_kernels, param_kernels)
>>> calibration = CalibrationData(los_vectors, optical_psfs)
>>> match_ctx = ImageMatchingContext(gcp_pairs, params, 0, "sci_001")
>>> geo, matching = _geolocate_and_match(
...     setup, kernel_ctx, times, tlm_dataset,
...     calibration, integrated_image_match, match_ctx
... )

curryer.correction.pipeline._resolve_netcdf_config(setup: curryer.correction.config.GeolocationSetup, output: curryer.correction.config.OutputConfig) → curryer.correction.config.NetCDFConfig

Return the output’s NetCDFConfig with the threshold pinned to the requirement.

setup.requirements.performance_threshold_m is the single source of truth for the threshold: it drives the computed pass-rate metric and the output variable’s name and metadata. Pinning it here keeps the written NetCDF accurate to what is actually computed, even when a caller supplies an OutputConfig.netcdf — whose other fields (title, description, parameter metadata) are preserved.

curryer.correction.pipeline.loop(setup: curryer.correction.config.GeolocationSetup, sweep: curryer.correction.config.Sweep, work_dir: pathlib.Path, tlm_sci_gcp_sets: list[tuple[str, str, str]], output: curryer.correction.config.OutputConfig | None = None, resume_from_checkpoint: bool = False)

Correction loop for parameter sensitivity analysis.

Parameters:

• setup (GeolocationSetup) – Durable mission setup: SPICE kernels/instrument (geo), pass/fail requirements, data_config (file format + time scaling), optional calibration files, mission variable names, and an optional image_matching_func override.

• sweep (Sweep) – The parameter-variation experiment: parameters, search_strategy, n_iterations, seed, and grid settings.

• work_dir (Path) – Working directory for temporary files.

• tlm_sci_gcp_sets (list of (str, str, str)) – List of (telemetry_key, science_key, gcp_key) tuples. File paths are expected to be local. S3 URIs (s3://…) are also accepted as a convenience when boto3 is installed; see resolve_path().

• output (OutputConfig or None, optional) – Output settings (NetCDF metadata + filename). None uses defaults derived from setup.requirements.

• resume_from_checkpoint (bool, optional) – If True, resume from an existing checkpoint.

Returns:

• results (list) – List of iteration results (order: pair_idx * N + param_idx).

• netcdf_data (dict) – Dictionary of NetCDF variables indexed as [param_idx, pair_idx].

Notes

This implementation uses a pair-outer, parameter-inner loop order: - Outer loop: GCP pairs (load data once per image) - Inner loop: Parameter sets (reuse loaded data) This reduces file I/O and centralizes mission-specific behavior through the setup object.

Examples

Correction mode (parameter optimization):

from curryer.correction.config import DataConfig

results, netcdf_data = loop(setup, sweep, work_dir, tlm_sci_gcp_sets)

Where each element of tlm_sci_gcp_sets is a tuple of file paths:

tlm_sci_gcp_sets = [
    ("telemetry.csv", "science.csv", "landsat_chip_001.mat"),
]

curryer.correction.pipeline._extract_parameter_values(params)

Extract parameter values from a parameter set into a dictionary.

curryer.correction.pipeline._store_parameter_values(netcdf_data, param_idx, param_values)

Store parameter values in the NetCDF data structure.

This function maps parameter names to NetCDF variable names for storage. It handles the naming convention used by _build_netcdf_structure.

curryer.correction.pipeline._extract_error_metrics(stats_dataset)

Extract error metrics from error statistics dataset.

curryer.correction.pipeline._store_gcp_pair_results(netcdf_data, param_idx, pair_idx, error_metrics)

Store GCP pair results in the NetCDF data structure.

curryer.correction.pipeline._compute_parameter_set_metrics(netcdf_data, param_idx, pair_errors, threshold_m=250.0)

Compute overall performance metrics for a parameter set.

Parameters:

• netcdf_data – NetCDF data dictionary

• param_idx – Parameter set index

• pair_errors – Array of RMS errors for each GCP pair

• threshold_m – Performance threshold in meters

curryer.correction.pipeline.run_correction(setup: curryer.correction.config.GeolocationSetup, sweep: curryer.correction.config.Sweep, inputs: collections.abc.Sequence[curryer.correction.config.CorrectionInput | tuple[str, str, str]], work_dir: pathlib.Path, output: curryer.correction.config.OutputConfig | None = None, resume_from_checkpoint: bool = False) → curryer.correction.results.CorrectionResult

Run the correction parameter sweep.

This is the preferred user-facing entry point (compared to loop()). Returns a structured CorrectionResult with the best parameter set, pass/fail verdict, recommendation, and a human-readable summary table. The raw results list and netcdf_data dict from loop() are available as result.results and result.netcdf_data for advanced use.

Parameters:

• setup (GeolocationSetup) – Durable mission setup (kernels, requirements, calibration, names).

• sweep (Sweep) – The parameter-variation experiment to run against setup.

• inputs (list of CorrectionInput or list of (str, str, str)) – Each element is either a CorrectionInput (named fields) or a legacy (telemetry_key, science_key, gcp_key) tuple. Both forms may be mixed in the same list. File paths are expected to be local. S3 URIs (s3://…) are also accepted as a convenience when boto3 is installed; see resolve_path().

• work_dir (Path) – Working directory for temporary files.

• output (OutputConfig or None, optional) – Output settings (NetCDF metadata + filename). None uses defaults derived from setup.requirements.

• resume_from_checkpoint (bool, optional) – If True, resume from an existing checkpoint.

Returns:

Structured result with best parameters, pass/fail verdict, recommendation, summary table, and raw NetCDF/intermediate data available on the returned object (for example, result.netcdf_data).

Return type:

CorrectionResult

curryer.correction.pipeline.compute_error_stats(image_matching_results, setup: curryer.correction.config.GeolocationSetup)

Compute error statistics from image matching results.

This is the preferred name for call_error_stats_module(). See call_error_stats_module() for full documentation.

Parameters:

• image_matching_results (xr.Dataset or list of xr.Dataset) – Output from image matching, either a single dataset or a list.

• setup (GeolocationSetup) – Geolocation setup used to initialise the error stats processor.

Returns:

Aggregate error statistics dataset.

Return type:

xr.Dataset

curryer.correction.pipeline.run_image_matching(geolocated_data: xarray.Dataset, gcp_reference_file: pathlib.Path, telemetry: pandas.DataFrame, params_info: list, setup: curryer.correction.config.GeolocationSetup, los_vectors_cached: numpy.ndarray | None = None, optical_psfs_cached: list | None = None) → xarray.Dataset

Run image matching against GCP reference.

This is the preferred name for image_matching(). See image_matching() for full documentation.

Parameters:

• geolocated_data (xr.Dataset) – Geolocated scene data with latitude/longitude.

• gcp_reference_file (Path) – Path to the GCP reference image (.mat file).

• telemetry (pd.DataFrame) – Telemetry DataFrame with spacecraft state.

• params_info (list) – Parameter information for the current iteration.

• setup (GeolocationSetup) – Geolocation setup (calibration paths, variable names, instrument name).

• los_vectors_cached (np.ndarray or None, optional) – Pre-loaded LOS vectors; loaded from disk if None.

• optical_psfs_cached (list or None, optional) – Pre-loaded optical PSFs; loaded from disk if None.

Returns:

Image matching results dataset.

Return type:

xr.Dataset