utilities

Download and management utilities for syncing time and auxiliary files

  • Can list a directory on a ftp host

  • Can download a file from a ftp or http host

  • Can download a file from NSIDC via https when Earthdata credentials are supplied

  • Checks MD5 or sha1 hashes between local and remote files

Source code

General Methods

icesat2_toolkit.utilities.get_data_path(relpath: list | str | pathlib.Path)[source]

Get the absolute path within a package from a relative path

Parameters
relpath: list, str or pathlib.Path

relative path

icesat2_toolkit.utilities.import_dependency(name: str, extra: str = '', raise_exception: bool = False)[source]

Import an optional dependency

Adapted from pandas.compat._optional::import_optional_dependency

Parameters
name: str

Module name

extra: str, default “”

Additional text to include in the ImportError message

raise_exception: bool, default False

Raise an ImportError if the module is not found

Returns
module: obj

Imported module

icesat2_toolkit.utilities.get_hash(local: str | io.IOBase | pathlib.Path, algorithm: str = 'MD5')[source]

Get the hash value from a local file or BytesIO object

Parameters
local: obj, str or pathlib.Path

BytesIO object or path to file

algorithm: str, default ‘MD5’

hashing algorithm for checksum validation

  • 'MD5': Message Digest

  • 'sha1': Secure Hash Algorithm

icesat2_toolkit.utilities.get_git_revision_hash(refname: str = 'HEAD', short: bool = False)[source]

Get the git hash value for a particular reference

Parameters
refname: str, default HEAD

Symbolic reference name

short: bool, default False

Return the shorted hash value

icesat2_toolkit.utilities.get_git_status()[source]

Get the status of a git repository as a boolean value

icesat2_toolkit.utilities.url_split(s: str)[source]

Recursively split a url path into a list

Parameters
s: str

url string

icesat2_toolkit.utilities.get_unix_time(time_string: str, format: str = '%Y-%m-%d %H:%M:%S')[source]

Get the Unix timestamp value for a formatted date string

Parameters
time_string: str

formatted time string to parse

format: str, default ‘%Y-%m-%d %H:%M:%S’

format for input time string

icesat2_toolkit.utilities.even(value: float)[source]

Rounds a number to an even number less than or equal to original

Parameters
value: float

number to be rounded

icesat2_toolkit.utilities.ceil(value: float)[source]

Rounds a number upward to its nearest integer

Parameters
value: float

number to be rounded upward

icesat2_toolkit.utilities.copy(source: str, destination: str, move: bool = False, **kwargs)[source]

Copy or move a file with all system information

Parameters
source: str

source file

destination: str

copied destination file

move: bool, default False

remove the source file

icesat2_toolkit.utilities.s3_client(HOST: str = 'https://data.nsidc.earthdatacloud.nasa.gov/s3credentials', timeout: int | None = None, region_name: str = 'us-west-2')[source]

Get AWS s3 client for NSIDC data in the cloud https://data.nsidc.earthdatacloud.nasa.gov/s3credentials

Parameters
HOST: str

NSIDC AWS S3 credential host

timeout: int or NoneType, default None

timeout in seconds for blocking operations

region_name: str, default ‘us-west-2’

AWS region name

Returns
client: obj

AWS s3 client for NSIDC Cumulus

icesat2_toolkit.utilities.s3_filesystem(HOST: str = 'https://data.nsidc.earthdatacloud.nasa.gov/s3credentials', timeout: int | None = None, region_name: str = 'us-west-2')[source]

Get AWS s3 file system object for NSIDC data in the cloud https://data.nsidc.earthdatacloud.nasa.gov/s3credentials

Parameters
HOST: str

NSIDC AWS S3 credential host

timeout: int or NoneType, default None

timeout in seconds for blocking operations

region_name: str, default ‘us-west-2’

AWS region name

Returns
session: obj

AWS s3 file system session for NSIDC Cumulus

icesat2_toolkit.utilities.s3_bucket(presigned_url: str)[source]

Get a s3 bucket name from a presigned url

Parameters
presigned_url: str

s3 presigned url

Returns
bucket: str

s3 bucket name

icesat2_toolkit.utilities.s3_key(presigned_url: str)[source]

Get a s3 bucket key from a presigned url

Parameters
presigned_url: str

s3 presigned url or https url

Returns
key: str

s3 bucket key for object

icesat2_toolkit.utilities.s3_presigned_url(bucket: str, key: str)[source]

Get a s3 presigned url from a bucket and object key

Parameters
bucket: str

s3 bucket name

key: str

s3 bucket key for object

Returns
presigned_url: str

s3 presigned url

icesat2_toolkit.utilities.generate_presigned_url(bucket: str, key: str, expiration: int = 3600)[source]

Generate a presigned https URL to share an S3 object

Parameters
bucket: str

s3 bucket name

key: str

s3 bucket key for object

expiration: int

Time in seconds for the presigned URL to remain valid

Returns
presigned_url: str

s3 presigned https url

icesat2_toolkit.utilities.check_ftp_connection(HOST: str, username: str | None = None, password: str | None = None)[source]

Check internet connection with ftp host

Parameters
HOST: str

remote ftp host

username: str or NoneType

ftp username

password: str or NoneType

ftp password

icesat2_toolkit.utilities.ftp_list(HOST: str | list, username: str | None = None, password: str | None = None, timeout: int | None = None, basename: bool = False, pattern: str | None = None, sort: bool = False)[source]

List a directory on a ftp host

Parameters
HOST: str or list

remote ftp host path split as list

username: str or NoneType

ftp username

password: str or NoneType

ftp password

timeout: int or NoneType, default None

timeout in seconds for blocking operations

basename: bool, default False

return the file or directory basename instead of the full path

pattern: str or NoneType, default None

regular expression pattern for reducing list

sort: bool, default False

sort output list

Returns
output: list

items in a directory

mtimes: list

last modification times for items in the directory

icesat2_toolkit.utilities.from_ftp(HOST: str | list, username: str | None = None, password: str | None = None, timeout: int | None = None, local: str | pathlib.Path | None = None, hash: str = '', chunk: int = 8192, verbose: bool = False, fid=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, mode: oct = 509)[source]

Download a file from a ftp host

Parameters
HOST: str or list

remote ftp host path

username: str or NoneType

ftp username

password: str or NoneType

ftp password

timeout: int or NoneType, default None

timeout in seconds for blocking operations

local: str, pathlib.Path or NoneType, default None

path to local file

hash: str, default ‘’

MD5 hash of local file

chunk: int, default 8192

chunk size for transfer encoding

verbose: bool, default False

print file transfer information

fid: obj, default sys.stdout

open file object to print if verbose

mode: oct, default 0o775

permissions mode of output local file

Returns
remote_buffer: obj

BytesIO representation of file

icesat2_toolkit.utilities._create_default_ssl_context() SSLContext[source]

Creates the default SSL context

icesat2_toolkit.utilities._create_ssl_context_no_verify() SSLContext[source]

Creates an SSL context for unverified connections

icesat2_toolkit.utilities._set_ssl_context_options(context: SSLContext) None[source]

Sets the default options for the SSL context

icesat2_toolkit.utilities.check_connection(HOST: str, context=<ssl.SSLContext object>)[source]

Check internet connection with http host

Parameters
HOST: str

remote http host

context: obj, default ssl.SSLContext(ssl.PROTOCOL_TLS)

SSL context for urllib opener object

icesat2_toolkit.utilities.http_list(HOST: str | list, timeout: int | None = None, context=<ssl.SSLContext object>, parser=<lxml.etree.HTMLParser object>, format: str = '%Y-%m-%d %H:%M', pattern: str = '', sort: bool = False)[source]

List a directory on an Apache http Server

Parameters
HOST: str or list

remote http host path

timeout: int or NoneType, default None

timeout in seconds for blocking operations

context: obj, default ssl.SSLContext(ssl.PROTOCOL_TLS)

SSL context for urllib opener object

parser: obj, default lxml.etree.HTMLParser()

HTML parser for lxml

format: str, default ‘%Y-%m-%d %H:%M’

format for input time string

pattern: str, default ‘’

regular expression pattern for reducing list

sort: bool, default False

sort output list

Returns
colnames: list

column names in a directory

collastmod: list

last modification times for items in the directory

colerror: list

notification for list error

icesat2_toolkit.utilities.from_http(HOST: str | list, timeout: int | None = None, context=<ssl.SSLContext object>, local: str | pathlib.Path | None = None, hash: str = '', chunk: int = 16384, verbose: bool = False, fid=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, mode: oct = 509)[source]

Download a file from a http host

Parameters
HOST: str or list

remote http host path split as list

timeout: int or NoneType, default None

timeout in seconds for blocking operations

context: obj, default ssl.SSLContext(ssl.PROTOCOL_TLS)

SSL context for urllib opener object

local: str, pathlib.Path or NoneType, default None

path to local file

hash: str, default ‘’

MD5 hash of local file

chunk: int, default 16384

chunk size for transfer encoding

verbose: bool, default False

print file transfer information

fid: obj, default sys.stdout

open file object to print if verbose

mode: oct, default 0o775

permissions mode of output local file

Returns
remote_buffer: obj

BytesIO representation of file

icesat2_toolkit.utilities.attempt_login(urs: str, context=<ssl.SSLContext object>, password_manager: bool = True, get_ca_certs: bool = False, redirect: bool = False, authorization_header: bool = False, **kwargs)[source]

attempt to build a urllib opener for NASA Earthdata

Parameters
urs: str

Earthdata login URS 3 host

context: obj, default ssl.SSLContext(ssl.PROTOCOL_TLS)

SSL context for urllib opener object

password_manager: bool, default True

Create password manager context using default realm

get_ca_certs: bool, default False

Get list of loaded “certification authority” certificates

redirect: bool, default False

Create redirect handler object

authorization_header: bool, default False

Add base64 encoded authorization header to opener

username: str, default from environmental variable

NASA Earthdata username

password: str, default from environmental variable

NASA Earthdata password

retries: int, default 5

number of retry attempts

netrc: str, default ~/.netrc

path to .netrc file for authentication

Returns
opener: obj

OpenerDirector instance

icesat2_toolkit.utilities.build_opener(username: str, password: str, context=<ssl.SSLContext object>, password_manager: bool = True, get_ca_certs: bool = False, redirect: bool = False, authorization_header: bool = False, urs: str = 'https://urs.earthdata.nasa.gov')[source]

Build urllib opener for NASA Earthdata with supplied credentials

Parameters
username: str or NoneType, default None

NASA Earthdata username

password: str or NoneType, default None

NASA Earthdata password

context: obj, default ssl.SSLContext(ssl.PROTOCOL_TLS)

SSL context for urllib opener object

password_manager: bool, default True

Create password manager context using default realm

get_ca_certs: bool, default True

Get list of loaded “certification authority” certificates

redirect: bool, default True

Create redirect handler object

authorization_header: bool, default False

Add base64 encoded authorization header to opener

urs: str, default ‘https://urs.earthdata.nasa.gov’

Earthdata login URS 3 host

Returns
opener: obj

OpenerDirector instance

icesat2_toolkit.utilities.get_token(HOST: str = 'https://urs.earthdata.nasa.gov/api/users/token', username: str | None = None, password: str | None = None, build: bool = True, urs: str = 'urs.earthdata.nasa.gov')[source]

Generate a NASA Earthdata User Token

Parameters
HOST: str or list

NASA Earthdata token API host

username: str or NoneType, default None

NASA Earthdata username

password: str or NoneType, default None

NASA Earthdata password

build: bool, default True

Build opener and check WebDAV credentials

timeout: int or NoneType, default None

timeout in seconds for blocking operations

urs: str, default ‘urs.earthdata.nasa.gov’

NASA Earthdata URS 3 host

Returns
token: dict

JSON response with NASA Earthdata User Token

icesat2_toolkit.utilities.list_tokens(HOST: str = 'https://urs.earthdata.nasa.gov/api/users/tokens', username: str | None = None, password: str | None = None, build: bool = True, urs: str = 'urs.earthdata.nasa.gov')[source]

List the current associated NASA Earthdata User Tokens

Parameters
HOST: str

NASA Earthdata list token API host

username: str or NoneType, default None

NASA Earthdata username

password: str or NoneType, default None

NASA Earthdata password

build: bool, default True

Build opener and check WebDAV credentials

timeout: int or NoneType, default None

timeout in seconds for blocking operations

urs: str, default ‘urs.earthdata.nasa.gov’

NASA Earthdata URS 3 host

Returns
tokens: list

JSON response with NASA Earthdata User Tokens

icesat2_toolkit.utilities.revoke_token(token: str, HOST: str = 'https://urs.earthdata.nasa.gov/api/users/revoke_token', username: str | None = None, password: str | None = None, build: bool = True, urs: str = 'urs.earthdata.nasa.gov')[source]

Generate a NASA Earthdata User Token

Parameters
token: str

NASA Earthdata token to be revoked

HOST: str

NASA Earthdata revoke token API host

username: str or NoneType, default None

NASA Earthdata username

password: str or NoneType, default None

NASA Earthdata password

build: bool, default True

Build opener and check WebDAV credentials

timeout: int or NoneType, default None

timeout in seconds for blocking operations

urs: str, default ‘urs.earthdata.nasa.gov’

NASA Earthdata URS 3 host

icesat2_toolkit.utilities.check_credentials()[source]

Check that entered NASA Earthdata credentials are valid

icesat2_toolkit.utilities.nsidc_list(HOST: str | list, username: str | None = None, password: str | None = None, build: bool = True, timeout: int | None = None, urs: str = 'urs.earthdata.nasa.gov', parser=<lxml.etree.HTMLParser object>, pattern: str = '', sort: bool = False)[source]

List a directory on NSIDC

Parameters
HOST: str or list

remote https host

username: str or NoneType, default None

NASA Earthdata username

password: str or NoneType, default None

NASA Earthdata password

build: bool, default True

Build opener and check WebDAV credentials

timeout: int or NoneType, default None

timeout in seconds for blocking operations

urs: str, default ‘urs.earthdata.nasa.gov’

NASA Earthdata URS 3 host

parser: obj, default lxml.etree.HTMLParser()

HTML parser for lxml

pattern: str, default ‘’

regular expression pattern for reducing list

sort: bool, default False

sort output list

Returns
colnames: list

Column names in a directory

collastmod: list

Last modification times for items in the directory

colerror: list

Notification for list error

icesat2_toolkit.utilities.from_nsidc(HOST: str | list, username: str | None = None, password: str | None = None, build: bool = True, timeout: int | None = None, urs: str = 'urs.earthdata.nasa.gov', local: str | pathlib.Path | None = None, hash: str = '', chunk: int = 16384, verbose: bool = False, fid=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, mode: oct = 509)[source]

Download a file from a NSIDC https server

Parameters
HOST: str or list

remote https host

username: str or NoneType, default None

NASA Earthdata username

password: str or NoneType, default None

NASA Earthdata password

build: bool, default True

Build opener and check WebDAV credentials

timeout: int or NoneType, default None

timeout in seconds for blocking operations

urs: str, default ‘urs.earthdata.nasa.gov’

NASA Earthdata URS 3 host

local: str or NoneType, default None

path to local file

hash: str, default ‘’

MD5 hash of local file

chunk: int, default 16384

chunk size for transfer encoding

verbose: bool, default False

print file transfer information

fid: obj, default sys.stdout

open file object to print if verbose

mode: oct, default 0o775

permissions mode of output local file

Returns
remote_buffer: obj

BytesIO representation of file

response_error: str or None

notification for response error

icesat2_toolkit.utilities.cmr_query_release(release: str | int | None)[source]

Build formatted query string for ICESat-2 release

Parameters
release: str, int or None

ICESat-2 data release to query

Returns
query_params: str

formatted string for CMR queries

icesat2_toolkit.utilities.cmr_cycles(cycle: str | int | list | None)[source]

Check if the submitted cycles are valid

Parameters
cycle: str, int, list or NoneType

ICESat-2 91-day orbital cycle

Returns
cycle_list: list

formatted available 91-day orbital cycles

icesat2_toolkit.utilities.cmr_tracks(track: str | int | list | None)[source]

Check if the submitted RGTs are valid

Parameters
track: str, int, list or NoneType

ICESat-2 reference ground track (RGT)

Returns
track_list: list

formatted available reference ground tracks (RGTs)

icesat2_toolkit.utilities.cmr_granules(granule: str | int | list | None)[source]

Check if the submitted granule regions are valid

Parameters
granule: str, int, list or NoneType

ICESat-2 granule region

Returns
granule_list: list

formatted available granule regions

icesat2_toolkit.utilities.cmr_regions(region: str | int | list | None)[source]

Check if the submitted ATL07 or ATL14/ATL15 regions are valid

Parameters
region: str, int, list or NoneType

ICESat-2 ATL07 or ATL14/ATL15 region

Returns
region_list: list

formatted available regions

icesat2_toolkit.utilities.cmr_resolutions(resolution: str | list | None)[source]

Check if the submitted ATL14/ATL15 resolutions are valid

Parameters
resolution: str, list or NoneType

ICESat-2 ATL14/ATL15 spatial resolution

Returns
resolution_list: list

formatted available ATL14/ATL15 resolutions

icesat2_toolkit.utilities.cmr_readable_granules(product: str, **kwargs)[source]

Create list of readable granule names for CMR queries

Parameters
product: str

ICESat-2 data product

cycles: str, list or NoneType, default None

91-day orbital cycle strings to query

tracks: str, list or NoneType, default None

Reference Ground Track (RGT) strings to query

granules: str, list or NoneType, default None

ICESat-2 granule region strings to query

regions: str, list or NoneType, default None

ICESat-2 ATL07 or ATL14/ATL15 region

resolutions: str, list or NoneType, default None

ICESat-2 ATL14/ATL15 spatial resolution

Returns
readable_granule_list: list

readable granule names for CMR queries

icesat2_toolkit.utilities.cmr_filter_json(search_results: dict, endpoint: str = 'data', request_type: str = 'application/x-hdf(eos|5)')[source]

Filter the CMR json response for desired data files

Parameters
search_results: dict

json response from CMR query

endpoint: str, default ‘data’

url endpoint type

  • 'data': NASA Earthdata https archive

  • 'opendap': NASA Earthdata OPeNDAP archive

  • 's3': NASA Earthdata Cumulus AWS S3 bucket

request_type: str, default ‘application/x-hdfeos’

data type for reducing CMR query

Returns
producer_granule_ids: list

ICESat-2 granules

granule_urls: list

ICESat-2 granule urls from NSIDC

icesat2_toolkit.utilities.cmr(product: str = None, release: str = None, cycles: str | int | list | None = None, tracks: str | int | list | None = None, granules: str | int | list | None = None, regions: str | list | None = None, resolutions: str | list | None = None, bbox: list | None = None, start_date: str | None = None, end_date: str | None = None, provider: str = 'NSIDC_ECS', endpoint: str = 'data', request_type: str = 'application/x-hdf(eos|5)', opener=None, verbose: bool = False, fid=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>)[source]

Query the NASA Common Metadata Repository (CMR) for ICESat-2 data

Parameters
product: str or NoneType, default None

ICESat-2 data product to query

release: str or NoneType, default None

ICESat-2 data release to query

cycles: str, list or NoneType, default None

91-day orbital cycle strings to query

tracks: str, list or NoneType, default None

Reference Ground Track (RGT) strings to query

granules: str, list or NoneType, default None

ICESat-2 granule region strings to query

regions: str, list or NoneType, default None

ICESat-2 ATL07 or ATL14/15 region strings to query

resolutions: str, list or NoneType, default None

ICESat-2 ATL14/15 resolution strings to query

bbox: list or NoneType, default None

Spatial bounding box for CMR query in form (lon_min, lat_min, lon_max, lat_max)

start_date: str or NoneType, default None

starting date for CMR product query

end_date: str or NoneType, default None

ending date for CMR product query

provider: str, default ‘NSIDC_ECS’

CMR data provider

endpoint: str, default ‘data’

url endpoint type

  • 'data': NASA Earthdata https archive

  • 'opendap': NASA Earthdata OPeNDAP archive

  • 's3': NASA Earthdata Cumulus AWS S3 bucket

request_type: str, default ‘application/x-hdfeos’

data type for reducing CMR query

opener: obj or NoneType, default None

OpenerDirector instance

verbose: bool, default False

print file transfer information

fid: obj, default sys.stdout

open file object to print if verbose

Returns
producer_granule_ids: list

ICESat-2 granules

granule_urls: list

ICESat-2 granule urls from NSIDC