curryer.correction.pipeline =========================== .. py:module:: curryer.correction.pipeline .. autoapi-nested-parse:: Main correction pipeline orchestration. This module contains the public-facing :func:`loop` function that drives the Monte Carlo parameter sensitivity analysis, plus all of the helper functions it calls: - Adapter functions that bridge between the geolocation/image-matching sub-modules and the correction loop. - :func:`_load_file` -- internal helper that reads CSV/NetCDF/HDF5 files into DataFrames, replacing the old mission-specific loader callables. - :func:`_load_image_pair_data`, :func:`_load_calibration_data`, :func:`_geolocate_and_match` -- per-iteration computation helpers. - :func:`loop` -- outer GCP-pair loop, inner parameter-set loop. Architecture note ----------------- The correction pipeline follows the **pipeline → verification** dependency direction. The core image-matching and aggregation logic lives in ``verification.py``; ``pipeline.py`` imports it from there. This means *verification* is a standalone module (GCP pairing + image matching + error stats) that the correction loop reuses for its own last three steps. Attributes ---------- .. autoapisummary:: curryer.correction.pipeline.logger Functions --------- .. autoapisummary:: curryer.correction.pipeline.call_error_stats_module curryer.correction.pipeline._resolve_gcp_pairs curryer.correction.pipeline._load_file curryer.correction.pipeline._load_calibration_data curryer.correction.pipeline._require_image_matching_inputs curryer.correction.pipeline._load_image_pair_data curryer.correction.pipeline._geolocate_and_match curryer.correction.pipeline._resolve_netcdf_config curryer.correction.pipeline.loop curryer.correction.pipeline._extract_parameter_values curryer.correction.pipeline._store_parameter_values curryer.correction.pipeline._extract_error_metrics curryer.correction.pipeline._store_gcp_pair_results curryer.correction.pipeline._compute_parameter_set_metrics curryer.correction.pipeline.run_correction curryer.correction.pipeline.compute_error_stats curryer.correction.pipeline.run_image_matching Module Contents --------------- .. py:data:: logger .. py:function:: call_error_stats_module(image_matching_results, setup: curryer.correction.config.GeolocationSetup) Call the error_stats module with image matching output. :param image_matching_results: Either a single image matching result (xarray.Dataset) or a list of image matching results from multiple GCP pairs :param setup: GeolocationSetup with variable names and thresholds (REQUIRED) :returns: Aggregate error statistics dataset .. py:function:: _resolve_gcp_pairs(sci_key: str, gcp_key: str, setup: curryer.correction.config.GeolocationSetup) -> list[tuple[str, str]] Return the ``[(sci_key, gcp_key)]`` pair, validating that ``gcp_key`` is set. :param sci_key: Science file path for this outer-loop iteration. :type sci_key: str :param gcp_key: GCP ``.mat`` file path supplied as the third element of the ``tlm_sci_gcp_sets`` tuple. Must be non-empty. :type gcp_key: str :param setup: Unused directly; reserved for future extension. :type setup: GeolocationSetup :rtype: list of (sci_key, gcp_path) tuples — always length 1. :raises ValueError: If ``gcp_key`` is empty or whitespace-only. .. rubric:: Notes For spatial-overlap-based pairing (many L1A images × many GCP chips) call :func:`~curryer.correction.pairing.pair_files` *before* :func:`loop` to build ``tlm_sci_gcp_sets`` from the ``(l1a_file, gcp_file)`` results. .. py:function:: _load_file(file_path: str | pathlib.Path, file_format: str = 'csv') -> pandas.DataFrame Load a telemetry or science data file into a pandas DataFrame. :param file_path: Local path or S3 URI (``s3://bucket/key``). :type file_path: str | Path :param file_format: One of ``"csv"``, ``"netcdf"``, or ``"hdf5"``. :type file_format: str :rtype: pd.DataFrame :raises FileNotFoundError: If *file_path* is local and does not exist. :raises ImportError: If *file_path* is an S3 URI and boto3 is not installed. :raises ValueError: If ``file_format`` is not recognised. .. py:function:: _load_calibration_data(setup: curryer.correction.config.GeolocationSetup) -> curryer.correction.config.CalibrationData Load LOS vectors and optical PSF when ``setup.calibration`` is configured. This function centralizes calibration data loading, which is now called once per GCP pair in the optimized implementation (previously called once per parameter set). :param setup: Setup carrying optional :class:`~curryer.correction.config.CalibrationFiles` with direct ``los_vectors_file`` / ``psf_file`` paths. :type setup: GeolocationSetup :returns: NamedTuple containing (los_vectors, optical_psfs), or (None, None) when no calibration files are configured. :rtype: CalibrationData :raises FileNotFoundError: If a calibration file is configured but does not exist. :raises ValueError: If a calibration file exists but fails to load properly. .. note:: Calibration files are optional and interim: real LOS/spacecraft geometry will be SPICE-derived. When ``setup.calibration`` is ``None`` (or both file fields are ``None``), returns ``CalibrationData(None, None)`` so that missions without calibration files still work. .. rubric:: Examples >>> calib_data = _load_calibration_data(setup) >>> if calib_data.los_vectors is not None: ... # Use calibration data in image matching ... pass .. py:function:: _require_image_matching_inputs(setup: curryer.correction.config.GeolocationSetup, calibration_data: curryer.correction.config.CalibrationData) -> None Fail fast when the built-in image matching has no calibration to run with. When no ``setup.image_matching_func`` override is provided, the loop falls back to the built-in :func:`~curryer.correction.verification.image_matching`, which requires LOS/PSF calibration. Raising here — before geolocation and kernel creation — gives a clear, early error instead of a less contextual failure deep inside the loop. A custom override may legitimately need no calibration, so the check only applies when no override is set. :raises ValueError: If no override is set and no calibration data is configured. .. py:function:: _load_image_pair_data(tlm_key: str, sci_key: str, setup: curryer.correction.config.GeolocationSetup) -> tuple[pandas.DataFrame, pandas.DataFrame, Any] Load telemetry and science data for an image pair from files. :param tlm_key: Path to the telemetry data file. :type tlm_key: str :param sci_key: Path to the science frame timing file. :type sci_key: str :param setup: Setup containing geolocation settings, file format, and time-scaling options (via ``setup.data_config``). :type setup: GeolocationSetup :returns: * **tlm_dataset** (*pandas.DataFrame*) -- DataFrame containing spacecraft state / telemetry records. * **sci_dataset** (*pandas.DataFrame*) -- DataFrame containing science frame timing information. * **ugps_times** (*array_like*) -- Time array extracted from the science dataset (uGPS values). :raises FileNotFoundError: If the telemetry or science file does not exist. :raises ValueError: If the science file is missing the required time field. .. py:function:: _geolocate_and_match(setup: curryer.correction.config.GeolocationSetup, kernel_ctx: curryer.correction.config.KernelContext, ugps_times_modified: Any, tlm_dataset: pandas.DataFrame, calibration: curryer.correction.config.CalibrationData, image_matching_func: Any, match_ctx: curryer.correction.config.ImageMatchingContext) -> tuple[xarray.Dataset, xarray.Dataset] Perform geolocation and image matching for a parameter set. This function loads SPICE kernels, performs geolocation, and runs image matching against GCP reference data. It's the core computation step that combines all previous setup (kernels, data loading) into results. :param setup: Setup with geo and image matching settings :type setup: GeolocationSetup :param kernel_ctx: NamedTuple containing: - mkrn: MetaKernel instance with SDS and mission kernels - dynamic_kernels: List of dynamic kernel file paths - param_kernels: List of parameter-specific kernel file paths :type kernel_ctx: KernelContext :param ugps_times_modified: Time array (possibly modified by OFFSET_TIME parameter) :type ugps_times_modified: array-like :param tlm_dataset: Spacecraft state telemetry data :type tlm_dataset: pd.DataFrame :param calibration: NamedTuple containing: - los_vectors: Pre-loaded LOS vectors (or None) - optical_psfs: Pre-loaded optical PSF (or None) :type calibration: CalibrationData :param image_matching_func: Function to perform image matching; defaults to the built-in ``image_matching`` function when not overridden in config. :type image_matching_func: callable :param match_ctx: NamedTuple containing: - gcp_pairs: List of GCP pairing tuples - params: List of (ParameterConfig, parameter_value) tuples - pair_idx: Index of current GCP pair - sci_key: Science dataset identifier for this pair :type match_ctx: ImageMatchingContext :returns: * **geo_dataset** (*xr.Dataset*) -- Geolocated points with latitude, longitude, altitude * **image_matching_output** (*xr.Dataset*) -- Matching results with error measurements and metadata .. rubric:: Examples >>> kernel_ctx = KernelContext(mkrn, dynamic_kernels, param_kernels) >>> calibration = CalibrationData(los_vectors, optical_psfs) >>> match_ctx = ImageMatchingContext(gcp_pairs, params, 0, "sci_001") >>> geo, matching = _geolocate_and_match( ... setup, kernel_ctx, times, tlm_dataset, ... calibration, integrated_image_match, match_ctx ... ) .. py:function:: _resolve_netcdf_config(setup: curryer.correction.config.GeolocationSetup, output: curryer.correction.config.OutputConfig) -> curryer.correction.config.NetCDFConfig Return the output's NetCDFConfig with the threshold pinned to the requirement. ``setup.requirements.performance_threshold_m`` is the single source of truth for the threshold: it drives the computed pass-rate metric *and* the output variable's name and metadata. Pinning it here keeps the written NetCDF accurate to what is actually computed, even when a caller supplies an ``OutputConfig.netcdf`` — whose other fields (title, description, parameter metadata) are preserved. .. py:function:: loop(setup: curryer.correction.config.GeolocationSetup, sweep: curryer.correction.config.Sweep, work_dir: pathlib.Path, tlm_sci_gcp_sets: list[tuple[str, str, str]], output: curryer.correction.config.OutputConfig | None = None, resume_from_checkpoint: bool = False) Correction loop for parameter sensitivity analysis. :param setup: Durable mission setup: SPICE kernels/instrument (``geo``), pass/fail ``requirements``, ``data_config`` (file format + time scaling), optional ``calibration`` files, mission variable names, and an optional ``image_matching_func`` override. :type setup: GeolocationSetup :param sweep: The parameter-variation experiment: ``parameters``, ``search_strategy``, ``n_iterations``, ``seed``, and grid settings. :type sweep: Sweep :param work_dir: Working directory for temporary files. :type work_dir: Path :param tlm_sci_gcp_sets: List of (`telemetry_key`, `science_key`, `gcp_key`) tuples. File paths are expected to be local. S3 URIs (``s3://…``) are also accepted as a convenience when ``boto3`` is installed; see :func:`~curryer.correction.io.resolve_path`. :type tlm_sci_gcp_sets: list of (str, str, str) :param output: Output settings (NetCDF metadata + filename). ``None`` uses defaults derived from ``setup.requirements``. :type output: OutputConfig or None, optional :param resume_from_checkpoint: If True, resume from an existing checkpoint. :type resume_from_checkpoint: bool, optional :returns: * **results** (*list*) -- List of iteration results (order: `pair_idx * N + param_idx`). * **netcdf_data** (*dict*) -- Dictionary of NetCDF variables indexed as `[param_idx, pair_idx]`. .. rubric:: Notes This implementation uses a pair-outer, parameter-inner loop order: - Outer loop: GCP pairs (load data once per image) - Inner loop: Parameter sets (reuse loaded data) This reduces file I/O and centralizes mission-specific behavior through the ``setup`` object. .. rubric:: Examples Correction mode (parameter optimization):: from curryer.correction.config import DataConfig results, netcdf_data = loop(setup, sweep, work_dir, tlm_sci_gcp_sets) Where each element of ``tlm_sci_gcp_sets`` is a tuple of file paths:: tlm_sci_gcp_sets = [ ("telemetry.csv", "science.csv", "landsat_chip_001.mat"), ] .. py:function:: _extract_parameter_values(params) Extract parameter values from a parameter set into a dictionary. .. py:function:: _store_parameter_values(netcdf_data, param_idx, param_values) Store parameter values in the NetCDF data structure. This function maps parameter names to NetCDF variable names for storage. It handles the naming convention used by _build_netcdf_structure. .. py:function:: _extract_error_metrics(stats_dataset) Extract error metrics from error statistics dataset. .. py:function:: _store_gcp_pair_results(netcdf_data, param_idx, pair_idx, error_metrics) Store GCP pair results in the NetCDF data structure. .. py:function:: _compute_parameter_set_metrics(netcdf_data, param_idx, pair_errors, threshold_m=250.0) Compute overall performance metrics for a parameter set. :param netcdf_data: NetCDF data dictionary :param param_idx: Parameter set index :param pair_errors: Array of RMS errors for each GCP pair :param threshold_m: Performance threshold in meters .. py:function:: run_correction(setup: curryer.correction.config.GeolocationSetup, sweep: curryer.correction.config.Sweep, inputs: collections.abc.Sequence[curryer.correction.config.CorrectionInput | tuple[str, str, str]], work_dir: pathlib.Path, output: curryer.correction.config.OutputConfig | None = None, resume_from_checkpoint: bool = False) -> curryer.correction.results.CorrectionResult Run the correction parameter sweep. This is the preferred user-facing entry point (compared to :func:`loop`). Returns a structured :class:`~curryer.correction.results.CorrectionResult` with the best parameter set, pass/fail verdict, recommendation, and a human-readable summary table. The raw ``results`` list and ``netcdf_data`` dict from :func:`loop` are available as ``result.results`` and ``result.netcdf_data`` for advanced use. :param setup: Durable mission setup (kernels, requirements, calibration, names). :type setup: GeolocationSetup :param sweep: The parameter-variation experiment to run against *setup*. :type sweep: Sweep :param inputs: Each element is either a :class:`~curryer.correction.config.CorrectionInput` (named fields) or a legacy ``(telemetry_key, science_key, gcp_key)`` tuple. Both forms may be mixed in the same list. File paths are expected to be local. S3 URIs (``s3://…``) are also accepted as a convenience when ``boto3`` is installed; see :func:`~curryer.correction.io.resolve_path`. :type inputs: list of CorrectionInput or list of (str, str, str) :param work_dir: Working directory for temporary files. :type work_dir: Path :param output: Output settings (NetCDF metadata + filename). ``None`` uses defaults derived from ``setup.requirements``. :type output: OutputConfig or None, optional :param resume_from_checkpoint: If True, resume from an existing checkpoint. :type resume_from_checkpoint: bool, optional :returns: Structured result with best parameters, pass/fail verdict, recommendation, summary table, and raw NetCDF/intermediate data available on the returned object (for example, ``result.netcdf_data``). :rtype: CorrectionResult .. py:function:: compute_error_stats(image_matching_results, setup: curryer.correction.config.GeolocationSetup) Compute error statistics from image matching results. This is the preferred name for :func:`call_error_stats_module`. See :func:`call_error_stats_module` for full documentation. :param image_matching_results: Output from image matching, either a single dataset or a list. :type image_matching_results: xr.Dataset or list of xr.Dataset :param setup: Geolocation setup used to initialise the error stats processor. :type setup: GeolocationSetup :returns: Aggregate error statistics dataset. :rtype: xr.Dataset .. py:function:: run_image_matching(geolocated_data: xarray.Dataset, gcp_reference_file: pathlib.Path, telemetry: pandas.DataFrame, params_info: list, setup: curryer.correction.config.GeolocationSetup, los_vectors_cached: numpy.ndarray | None = None, optical_psfs_cached: list | None = None) -> xarray.Dataset Run image matching against GCP reference. This is the preferred name for :func:`image_matching`. See :func:`image_matching` for full documentation. :param geolocated_data: Geolocated scene data with latitude/longitude. :type geolocated_data: xr.Dataset :param gcp_reference_file: Path to the GCP reference image (.mat file). :type gcp_reference_file: Path :param telemetry: Telemetry DataFrame with spacecraft state. :type telemetry: pd.DataFrame :param params_info: Parameter information for the current iteration. :type params_info: list :param setup: Geolocation setup (calibration paths, variable names, instrument name). :type setup: GeolocationSetup :param los_vectors_cached: Pre-loaded LOS vectors; loaded from disk if None. :type los_vectors_cached: np.ndarray or None, optional :param optical_psfs_cached: Pre-loaded optical PSFs; loaded from disk if None. :type optical_psfs_cached: list or None, optional :returns: Image matching results dataset. :rtype: xr.Dataset