VIIRS extraction with AerEO

This chapter extracts VIIRS imagery from NASA’s Earthdata holdings. Unlike Sentinel-2, which is served through a public STAC API, VIIRS data requires authentication and is stored in a swath projection that must be reprojected to a regular grid.

AerEO hides most of that complexity behind the same ExtractionJob API. The config file selects a NASA-aware search provider, a Satpy-based reader, and a swath reprojection step.

Environment setup¶

The first cell installs AerEO plus the VIIRS-specific reader plugin (aereo-read-satpy) and the swath reprojection extra. On Binder these are pre-installed, so pip will skip the download.

NASA Earthdata authentication¶

NASA data access is handled by earthaccess. Before running the search or read steps, you need a valid ~/.netrc file containing your Earthdata login. If you see authentication errors, follow the earthaccess authentication guide.

In short:

1. Register at urs.earthdata.nasa.gov.

2. Link your application (e.g., LP DAAC, OB_DAAC, or GES DISC).

3. Run earthaccess.login(persist=True) once, or write the credentials manually to ~/.netrc.

If you are running interactively, you can create the file with:

import os
from getpass import getpass

netrc_path = os.path.expanduser("~/.netrc")
with open(netrc_path, "w") as f:
    f.write(
        "machine urs.earthdata.nasa.gov login {username} password {password}\n".format(
            username=getpass("Earthdata username: "),
            password=getpass("Earthdata password: "),
        )
    )
os.chmod(netrc_path, 0o600)

Config used in this notebook¶

job_viirs.yaml configures a swath-based extraction:

aoi_path: config/aoi/chocon.geojson
name: viirs_sample
grid_dist: 50_000
grid_cells_margin: 10
target_aoi: ${aoi_path}
output_uri: /tmp/aereo_extraction
overwrite: false

search:
  _target_: aereo.builtins.search_earthaccess
  _partial_: true
  collections:
    VJ202IMG: ["I04"]
    VJ203IMG: []
  intersects: ${aoi_path}
  start_datetime: "2024-01-01T00:00:00Z"
  end_datetime: "2024-01-02T23:59:59Z"

read:
  _partial_: true
  _target_: aereo.read_satpy.read_satpy
  reader: viirs_l1b
  wishlist: [I04]
  downloader:
    _target_: aereo.asset_downloader.core._download_with_earthaccess
    _partial_: true

reproject:
  _target_: aereo.builtins.reproject_swath
  _partial_: true
reproject_mode: grid
resolution: 375

write:
  _target_: aereo.builtins.write.write_geotiff

Key differences from Sentinel-2:

• search_earthaccess queries NASA CMR/DAAC holdings.

• read_satpy uses Satpy to read VIIRS L1B files.

• reproject_swath resamples the VIIRS swath to the output grid.

• resolution: 375 matches the VIIRS I-band nominal resolution.

• downloader is explicitly set to earthaccess to avoid S3 region retries.

Loading the job¶

ExtractionJob.load_from_config() parses the YAML with Hydra, resolves every _target_ callable, and validates the resulting ExtractionJob. The config_name argument is the YAML filename without extension, and config_dir is the folder that contains it.

A job is a declarative bundle of pipeline steps. Once loaded, the same job can be searched, can have tasks built from it, and can be executed.

Search and task building¶

job.search() calls the configured search provider and returns a GeoDataFrame of matched assets. This is a separate step from execution: it only discovers what data is available, without reading or writing anything.

job.build_tasks(assets) turns those assets into a list of ExtractionTask objects. Each task groups the assets needed for one grid cell and time slice, so the work can be parallelised later.

The executor¶

For swath data the reprojection step is CPU-bound, so this notebook uses use_threads=False to run tasks with process-based parallelism (joblib’s loky backend). This avoids the Python GIL and the fork-after-read deadlock that can occur with netCDF/HDF5 state.

Running the extraction¶

job.execute(tasks, executor=...) hands the prepared tasks to the executor. The executor runs each task independently; inside every task AerEO performs the pipeline:

read -> preprocess -> reproject -> postprocess -> write

The returned object is a GeoDataFrame of artifacts — one row per output GeoTIFF with its metadata, CRS, and footprint.

Visualizing VIIRS¶

The output is a reprojected GeoTIFF at 375 m resolution. plot_artifact_patches overlays the extracted patches on the AOI so you can verify spatial coverage.