nsidc_icesat2_convert.py

Syncs all available ICESat-2 data converting from HDF5 to zarr or rechunked HDF5 files for a specified data product, release, granule and track.
zarr files make large datasets easily accessible to distributed computing on both local filesystems and cloud-based object stores
rechunked HDF5 files can be more optimized for cloud-based object stores
The first time we run the script, it will copy the necessary dataset in the selected local directory.
If we already have all the data, and we run the script again: only files added or modified on the remote server will downloaded.

Calling Sequence

usage: nsidc_icesat2_convert.py [-h] [--user USER] [--password PASSWORD]
                                [--netrc NETRC] [--directory DIRECTORY]
                                [--year YEAR [YEAR ...]]
                                [--subdirectory SUBDIRECTORY [SUBDIRECTORY ...]]
                                [--release RELEASE]
                                [--version VERSION [VERSION ...]]
                                [--granule REGION [REGION ...]]
                                [--cycle CYCLE [CYCLE ...]]
                                [--track RGT [RGT ...]] [--format {zarr,HDF5}]
                                [--chunks CHUNKS] [--auxiliary]
                                [--index INDEX] [--flatten] [--np PROCESSES]
                                [--timeout TIMEOUT] [--retry RETRY] [--log]
                                [--list] [--clobber] [--mode MODE]
                                [PRODUCTS ...]

Positional Arguments

PRODUCTS: ICESat-2 products to convert

Named Arguments

--user, -U

Username for NASA Earthdata Login

--password, -W

Password for NASA Earthdata Login

--netrc, -N

Path to .netrc file for authentication

--directory, -D

Working data directory

--year, -Y

Years to run

--subdirectory, -S

subdirectories of data to run

--release, -r

ICESat-2 Data Release

--version, -v

ICESat-2 Data Version

--granule, -g

Possible choices: 1–14

ICESat-2 Granule Region

--cycle, -c

ICESat-2 orbital cycles to sync

--track, -t

Possible choices: 1–1387

ICESat-2 Reference Ground Tracks (RGTs)

--format, -f

Possible choices: zarr, HDF5

Output file format

--chunks

Rechunk output files to size

--auxiliary, -a

Sync ICESat-2 auxiliary files for each HDF5 file

--index, -i

Input index of ICESat-2 files to sync

--flatten, -F

Do not create subdirectories

--np, -P

Number of processes to use in file conversion

--timeout, -T

Timeout in seconds for blocking operations

--retry, -R

Connection retry attempts

--log, -l

Output log file

--list, -L

Only print files that could be transferred

--clobber, -C

Overwrite existing data

--mode, -M

Permissions mode of output files