curryer.correction.dataio¶

Validation helpers and S3 data-access utilities for the correction pipeline.

S3 access relies on the boto3 S3 client. Callers may either provide an explicit client instance (useful for testing) or rely on the default client, in which case boto3 must be installed and AWS credentials are read from the standard AWS_* environment variables.

Attributes¶

boto3

Classes¶

S3Configuration

Configuration describing how data is organised within an S3 bucket.

Functions¶

`validate_telemetry_output`(→ None)	Validate that telemetry loader output has expected structure.
`validate_science_output`(→ None)	Validate that science loader output has expected structure.
`find_netcdf_objects`(→ list[str])	Return S3 object keys for NetCDF files in the given date range.
`download_netcdf_objects`(→ list[pathlib.Path])	Download the specified S3 objects to `destination`.

Module Contents¶

curryer.correction.dataio.boto3 = None¶

curryer.correction.dataio.validate_telemetry_output(df: pandas.DataFrame, config) → None¶

Validate that telemetry loader output has expected structure.

Parameters:

df – DataFrame returned by telemetry loader
config – GeolocationSetup object

Raises:

TypeError – If not a DataFrame
ValueError – If DataFrame is empty

Note

Specific column requirements depend on mission and kernel configs. This performs basic structure checks only.

curryer.correction.dataio.validate_science_output(df: pandas.DataFrame, config) → None¶

Validate that science loader output has expected structure.

Parameters:

df – DataFrame returned by science loader
config – GeolocationSetup object

Raises:

TypeError – If not a DataFrame
ValueError – If DataFrame is empty or missing required time field

Example

>>> sci_df = pd.read_csv("science.csv")
>>> validate_science_output(sci_df, config)

class curryer.correction.dataio.S3Configuration(bucket: str, base_prefix: str)¶

Configuration describing how data is organised within an S3 bucket.

bucket¶

base_prefix¶

date_prefix(date: datetime.date) → str¶: Return the S3 prefix for date.

curryer.correction.dataio.find_netcdf_objects(config: S3Configuration, start_date: datetime.date, end_date: datetime.date, *, s3_client=None) → list[str]¶

Return S3 object keys for NetCDF files in the given date range.

Parameters:

config (S3Configuration) – Describes the bucket and prefix layout.
start_date (datetime.date) – Inclusive date range to scan for NetCDF files.
end_date (datetime.date) – Inclusive date range to scan for NetCDF files.
s3_client (boto3 S3 client, optional) – Client instance to use. If omitted, a default client is created.

curryer.correction.dataio.download_netcdf_objects(config: S3Configuration, object_keys: collections.abc.Iterable[str], destination: os.PathLike[str] | str, *, s3_client=None) → list[pathlib.Path]¶

Download the specified S3 objects to destination.

Parameters:

config (S3Configuration) – Describes the bucket hosting the objects.
object_keys (iterable of str) – S3 object keys to download.
destination (path-like) – Directory where the files should be stored. It is created if needed.
s3_client (boto3 S3 client, optional) – Client instance to use. If omitted, a default client is created.