utilities
Download and management utilities for syncing time and auxiliary files
Can list a directory on a ftp host
Can download a file from a ftp or http host
Can download a file from NSIDC via https when Earthdata credentials are supplied
Checks
MD5
orsha1
hashes between local and remote files
General Methods
- icesat2_toolkit.utilities.get_data_path(relpath: list | str | pathlib.Path)[source]
Get the absolute path within a package from a relative path
- Parameters
- relpath: list, str or pathlib.Path
relative path
- icesat2_toolkit.utilities.import_dependency(name: str, extra: str = '', raise_exception: bool = False)[source]
Import an optional dependency
Adapted from
pandas.compat._optional::import_optional_dependency
- Parameters
- name: str
Module name
- extra: str, default “”
Additional text to include in the
ImportError
message- raise_exception: bool, default False
Raise an
ImportError
if the module is not found
- Returns
- module: obj
Imported module
- icesat2_toolkit.utilities.get_hash(local: str | io.IOBase | pathlib.Path, algorithm: str = 'MD5')[source]
Get the hash value from a local file or
BytesIO
object- Parameters
- local: obj, str or pathlib.Path
BytesIO object or path to file
- algorithm: str, default ‘MD5’
hashing algorithm for checksum validation
'MD5'
: Message Digest'sha1'
: Secure Hash Algorithm
- icesat2_toolkit.utilities.get_git_revision_hash(refname: str = 'HEAD', short: bool = False)[source]
Get the
git
hash value for a particular reference- Parameters
- refname: str, default HEAD
Symbolic reference name
- short: bool, default False
Return the shorted hash value
- icesat2_toolkit.utilities.get_git_status()[source]
Get the status of a
git
repository as a boolean value
- icesat2_toolkit.utilities.url_split(s: str)[source]
Recursively split a url path into a list
- Parameters
- s: str
url string
- icesat2_toolkit.utilities.get_unix_time(time_string: str, format: str = '%Y-%m-%d %H:%M:%S')[source]
Get the Unix timestamp value for a formatted date string
- Parameters
- time_string: str
formatted time string to parse
- format: str, default ‘%Y-%m-%d %H:%M:%S’
format for input time string
- icesat2_toolkit.utilities.even(value: float)[source]
Rounds a number to an even number less than or equal to original
- Parameters
- value: float
number to be rounded
- icesat2_toolkit.utilities.ceil(value: float)[source]
Rounds a number upward to its nearest integer
- Parameters
- value: float
number to be rounded upward
- icesat2_toolkit.utilities.copy(source: str, destination: str, move: bool = False, **kwargs)[source]
Copy or move a file with all system information
- Parameters
- source: str
source file
- destination: str
copied destination file
- move: bool, default False
remove the source file
- icesat2_toolkit.utilities.s3_client(HOST: str = 'https://data.nsidc.earthdatacloud.nasa.gov/s3credentials', timeout: int | None = None, region_name: str = 'us-west-2')[source]
Get AWS s3 client for NSIDC data in the cloud https://data.nsidc.earthdatacloud.nasa.gov/s3credentials
- Parameters
- HOST: str
NSIDC AWS S3 credential host
- timeout: int or NoneType, default None
timeout in seconds for blocking operations
- region_name: str, default ‘us-west-2’
AWS region name
- Returns
- client: obj
AWS s3 client for NSIDC Cumulus
- icesat2_toolkit.utilities.s3_filesystem(HOST: str = 'https://data.nsidc.earthdatacloud.nasa.gov/s3credentials', timeout: int | None = None, region_name: str = 'us-west-2')[source]
Get AWS s3 file system object for NSIDC data in the cloud https://data.nsidc.earthdatacloud.nasa.gov/s3credentials
- Parameters
- HOST: str
NSIDC AWS S3 credential host
- timeout: int or NoneType, default None
timeout in seconds for blocking operations
- region_name: str, default ‘us-west-2’
AWS region name
- Returns
- session: obj
AWS s3 file system session for NSIDC Cumulus
- icesat2_toolkit.utilities.s3_bucket(presigned_url: str)[source]
Get a s3 bucket name from a presigned url
- Parameters
- presigned_url: str
s3 presigned url
- Returns
- bucket: str
s3 bucket name
- icesat2_toolkit.utilities.s3_key(presigned_url: str)[source]
Get a s3 bucket key from a presigned url
- Parameters
- presigned_url: str
s3 presigned url or https url
- Returns
- key: str
s3 bucket key for object
- icesat2_toolkit.utilities.s3_presigned_url(bucket: str, key: str)[source]
Get a s3 presigned url from a bucket and object key
- Parameters
- bucket: str
s3 bucket name
- key: str
s3 bucket key for object
- Returns
- presigned_url: str
s3 presigned url
- icesat2_toolkit.utilities.generate_presigned_url(bucket: str, key: str, expiration: int = 3600)[source]
Generate a presigned https URL to share an S3 object
- Parameters
- bucket: str
s3 bucket name
- key: str
s3 bucket key for object
- expiration: int
Time in seconds for the presigned URL to remain valid
- Returns
- presigned_url: str
s3 presigned https url
- icesat2_toolkit.utilities.check_ftp_connection(HOST: str, username: str | None = None, password: str | None = None)[source]
Check internet connection with ftp host
- Parameters
- HOST: str
remote ftp host
- username: str or NoneType
ftp username
- password: str or NoneType
ftp password
- icesat2_toolkit.utilities.ftp_list(HOST: str | list, username: str | None = None, password: str | None = None, timeout: int | None = None, basename: bool = False, pattern: str | None = None, sort: bool = False)[source]
List a directory on a ftp host
- Parameters
- HOST: str or list
remote ftp host path split as list
- username: str or NoneType
ftp username
- password: str or NoneType
ftp password
- timeout: int or NoneType, default None
timeout in seconds for blocking operations
- basename: bool, default False
return the file or directory basename instead of the full path
- pattern: str or NoneType, default None
regular expression pattern for reducing list
- sort: bool, default False
sort output list
- Returns
- output: list
items in a directory
- mtimes: list
last modification times for items in the directory
- icesat2_toolkit.utilities.from_ftp(HOST: str | list, username: str | None = None, password: str | None = None, timeout: int | None = None, local: str | pathlib.Path | None = None, hash: str = '', chunk: int = 8192, verbose: bool = False, fid=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, mode: oct = 509)[source]
Download a file from a ftp host
- Parameters
- HOST: str or list
remote ftp host path
- username: str or NoneType
ftp username
- password: str or NoneType
ftp password
- timeout: int or NoneType, default None
timeout in seconds for blocking operations
- local: str, pathlib.Path or NoneType, default None
path to local file
- hash: str, default ‘’
MD5 hash of local file
- chunk: int, default 8192
chunk size for transfer encoding
- verbose: bool, default False
print file transfer information
- fid: obj, default sys.stdout
open file object to print if verbose
- mode: oct, default 0o775
permissions mode of output local file
- Returns
- remote_buffer: obj
BytesIO representation of file
- icesat2_toolkit.utilities._create_default_ssl_context() SSLContext [source]
Creates the default SSL context
- icesat2_toolkit.utilities._create_ssl_context_no_verify() SSLContext [source]
Creates an SSL context for unverified connections
- icesat2_toolkit.utilities._set_ssl_context_options(context: SSLContext) None [source]
Sets the default options for the SSL context
- icesat2_toolkit.utilities.check_connection(HOST: str, context=<ssl.SSLContext object>)[source]
Check internet connection with http host
- Parameters
- HOST: str
remote http host
- context: obj, default ssl.SSLContext(ssl.PROTOCOL_TLS)
SSL context for
urllib
opener object
- icesat2_toolkit.utilities.http_list(HOST: str | list, timeout: int | None = None, context=<ssl.SSLContext object>, parser=<lxml.etree.HTMLParser object>, format: str = '%Y-%m-%d %H:%M', pattern: str = '', sort: bool = False)[source]
List a directory on an Apache http Server
- Parameters
- HOST: str or list
remote http host path
- timeout: int or NoneType, default None
timeout in seconds for blocking operations
- context: obj, default ssl.SSLContext(ssl.PROTOCOL_TLS)
SSL context for
urllib
opener object- parser: obj, default lxml.etree.HTMLParser()
HTML parser for
lxml
- format: str, default ‘%Y-%m-%d %H:%M’
format for input time string
- pattern: str, default ‘’
regular expression pattern for reducing list
- sort: bool, default False
sort output list
- Returns
- colnames: list
column names in a directory
- collastmod: list
last modification times for items in the directory
- colerror: list
notification for list error
- icesat2_toolkit.utilities.from_http(HOST: str | list, timeout: int | None = None, context=<ssl.SSLContext object>, local: str | pathlib.Path | None = None, hash: str = '', chunk: int = 16384, verbose: bool = False, fid=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, mode: oct = 509)[source]
Download a file from a http host
- Parameters
- HOST: str or list
remote http host path split as list
- timeout: int or NoneType, default None
timeout in seconds for blocking operations
- context: obj, default ssl.SSLContext(ssl.PROTOCOL_TLS)
SSL context for
urllib
opener object- local: str, pathlib.Path or NoneType, default None
path to local file
- hash: str, default ‘’
MD5 hash of local file
- chunk: int, default 16384
chunk size for transfer encoding
- verbose: bool, default False
print file transfer information
- fid: obj, default sys.stdout
open file object to print if verbose
- mode: oct, default 0o775
permissions mode of output local file
- Returns
- remote_buffer: obj
BytesIO representation of file
- icesat2_toolkit.utilities.attempt_login(urs: str, context=<ssl.SSLContext object>, password_manager: bool = True, get_ca_certs: bool = False, redirect: bool = False, authorization_header: bool = False, **kwargs)[source]
attempt to build a urllib opener for NASA Earthdata
- Parameters
- urs: str
Earthdata login URS 3 host
- context: obj, default ssl.SSLContext(ssl.PROTOCOL_TLS)
SSL context for
urllib
opener object- password_manager: bool, default True
Create password manager context using default realm
- get_ca_certs: bool, default False
Get list of loaded “certification authority” certificates
- redirect: bool, default False
Create redirect handler object
- authorization_header: bool, default False
Add base64 encoded authorization header to opener
- username: str, default from environmental variable
NASA Earthdata username
- password: str, default from environmental variable
NASA Earthdata password
- retries: int, default 5
number of retry attempts
- netrc: str, default ~/.netrc
path to .netrc file for authentication
- Returns
- opener: obj
OpenerDirector instance
- icesat2_toolkit.utilities.build_opener(username: str, password: str, context=<ssl.SSLContext object>, password_manager: bool = True, get_ca_certs: bool = False, redirect: bool = False, authorization_header: bool = False, urs: str = 'https://urs.earthdata.nasa.gov')[source]
Build
urllib
opener for NASA Earthdata with supplied credentials- Parameters
- username: str or NoneType, default None
NASA Earthdata username
- password: str or NoneType, default None
NASA Earthdata password
- context: obj, default ssl.SSLContext(ssl.PROTOCOL_TLS)
SSL context for
urllib
opener object- password_manager: bool, default True
Create password manager context using default realm
- get_ca_certs: bool, default True
Get list of loaded “certification authority” certificates
- redirect: bool, default True
Create redirect handler object
- authorization_header: bool, default False
Add base64 encoded authorization header to opener
- urs: str, default ‘https://urs.earthdata.nasa.gov’
Earthdata login URS 3 host
- Returns
- opener: obj
OpenerDirector instance
- icesat2_toolkit.utilities.get_token(HOST: str = 'https://urs.earthdata.nasa.gov/api/users/token', username: str | None = None, password: str | None = None, build: bool = True, urs: str = 'urs.earthdata.nasa.gov')[source]
Generate a NASA Earthdata User Token
- Parameters
- HOST: str or list
NASA Earthdata token API host
- username: str or NoneType, default None
NASA Earthdata username
- password: str or NoneType, default None
NASA Earthdata password
- build: bool, default True
Build opener and check WebDAV credentials
- timeout: int or NoneType, default None
timeout in seconds for blocking operations
- urs: str, default ‘urs.earthdata.nasa.gov’
NASA Earthdata URS 3 host
- Returns
- token: dict
JSON response with NASA Earthdata User Token
- icesat2_toolkit.utilities.list_tokens(HOST: str = 'https://urs.earthdata.nasa.gov/api/users/tokens', username: str | None = None, password: str | None = None, build: bool = True, urs: str = 'urs.earthdata.nasa.gov')[source]
List the current associated NASA Earthdata User Tokens
- Parameters
- HOST: str
NASA Earthdata list token API host
- username: str or NoneType, default None
NASA Earthdata username
- password: str or NoneType, default None
NASA Earthdata password
- build: bool, default True
Build opener and check WebDAV credentials
- timeout: int or NoneType, default None
timeout in seconds for blocking operations
- urs: str, default ‘urs.earthdata.nasa.gov’
NASA Earthdata URS 3 host
- Returns
- tokens: list
JSON response with NASA Earthdata User Tokens
- icesat2_toolkit.utilities.revoke_token(token: str, HOST: str = 'https://urs.earthdata.nasa.gov/api/users/revoke_token', username: str | None = None, password: str | None = None, build: bool = True, urs: str = 'urs.earthdata.nasa.gov')[source]
Generate a NASA Earthdata User Token
- Parameters
- token: str
NASA Earthdata token to be revoked
- HOST: str
NASA Earthdata revoke token API host
- username: str or NoneType, default None
NASA Earthdata username
- password: str or NoneType, default None
NASA Earthdata password
- build: bool, default True
Build opener and check WebDAV credentials
- timeout: int or NoneType, default None
timeout in seconds for blocking operations
- urs: str, default ‘urs.earthdata.nasa.gov’
NASA Earthdata URS 3 host
- icesat2_toolkit.utilities.check_credentials()[source]
Check that entered NASA Earthdata credentials are valid
- icesat2_toolkit.utilities.nsidc_list(HOST: str | list, username: str | None = None, password: str | None = None, build: bool = True, timeout: int | None = None, urs: str = 'urs.earthdata.nasa.gov', parser=<lxml.etree.HTMLParser object>, pattern: str = '', sort: bool = False)[source]
List a directory on NSIDC
- Parameters
- HOST: str or list
remote https host
- username: str or NoneType, default None
NASA Earthdata username
- password: str or NoneType, default None
NASA Earthdata password
- build: bool, default True
Build opener and check WebDAV credentials
- timeout: int or NoneType, default None
timeout in seconds for blocking operations
- urs: str, default ‘urs.earthdata.nasa.gov’
NASA Earthdata URS 3 host
- parser: obj, default lxml.etree.HTMLParser()
HTML parser for
lxml
- pattern: str, default ‘’
regular expression pattern for reducing list
- sort: bool, default False
sort output list
- Returns
- colnames: list
Column names in a directory
- collastmod: list
Last modification times for items in the directory
- colerror: list
Notification for list error
- icesat2_toolkit.utilities.from_nsidc(HOST: str | list, username: str | None = None, password: str | None = None, build: bool = True, timeout: int | None = None, urs: str = 'urs.earthdata.nasa.gov', local: str | pathlib.Path | None = None, hash: str = '', chunk: int = 16384, verbose: bool = False, fid=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, mode: oct = 509)[source]
Download a file from a NSIDC https server
- Parameters
- HOST: str or list
remote https host
- username: str or NoneType, default None
NASA Earthdata username
- password: str or NoneType, default None
NASA Earthdata password
- build: bool, default True
Build opener and check WebDAV credentials
- timeout: int or NoneType, default None
timeout in seconds for blocking operations
- urs: str, default ‘urs.earthdata.nasa.gov’
NASA Earthdata URS 3 host
- local: str or NoneType, default None
path to local file
- hash: str, default ‘’
MD5 hash of local file
- chunk: int, default 16384
chunk size for transfer encoding
- verbose: bool, default False
print file transfer information
- fid: obj, default sys.stdout
open file object to print if verbose
- mode: oct, default 0o775
permissions mode of output local file
- Returns
- remote_buffer: obj
BytesIO representation of file
- response_error: str or None
notification for response error
- icesat2_toolkit.utilities.cmr_query_release(release: str | int | None)[source]
Build formatted query string for ICESat-2 release
- Parameters
- release: str, int or None
ICESat-2 data release to query
- Returns
- query_params: str
formatted string for CMR queries
- icesat2_toolkit.utilities.cmr_cycles(cycle: str | int | list | None)[source]
Check if the submitted cycles are valid
- Parameters
- cycle: str, int, list or NoneType
ICESat-2 91-day orbital cycle
- Returns
- cycle_list: list
formatted available 91-day orbital cycles
- icesat2_toolkit.utilities.cmr_tracks(track: str | int | list | None)[source]
Check if the submitted RGTs are valid
- Parameters
- track: str, int, list or NoneType
ICESat-2 reference ground track (RGT)
- Returns
- track_list: list
formatted available reference ground tracks (RGTs)
- icesat2_toolkit.utilities.cmr_granules(granule: str | int | list | None)[source]
Check if the submitted granule regions are valid
- Parameters
- granule: str, int, list or NoneType
ICESat-2 granule region
- Returns
- granule_list: list
formatted available granule regions
- icesat2_toolkit.utilities.cmr_regions(region: str | int | list | None)[source]
Check if the submitted ATL07 or ATL14/ATL15 regions are valid
- Parameters
- region: str, int, list or NoneType
ICESat-2 ATL07 or ATL14/ATL15 region
- Returns
- region_list: list
formatted available regions
- icesat2_toolkit.utilities.cmr_resolutions(resolution: str | list | None)[source]
Check if the submitted ATL14/ATL15 resolutions are valid
- Parameters
- resolution: str, list or NoneType
ICESat-2 ATL14/ATL15 spatial resolution
- Returns
- resolution_list: list
formatted available ATL14/ATL15 resolutions
- icesat2_toolkit.utilities.cmr_readable_granules(product: str, **kwargs)[source]
Create list of readable granule names for CMR queries
- Parameters
- product: str
ICESat-2 data product
- cycles: str, list or NoneType, default None
91-day orbital cycle strings to query
- tracks: str, list or NoneType, default None
Reference Ground Track (RGT) strings to query
- granules: str, list or NoneType, default None
ICESat-2 granule region strings to query
- regions: str, list or NoneType, default None
ICESat-2 ATL07 or ATL14/ATL15 region
- resolutions: str, list or NoneType, default None
ICESat-2 ATL14/ATL15 spatial resolution
- Returns
- readable_granule_list: list
readable granule names for CMR queries
- icesat2_toolkit.utilities.cmr_filter_json(search_results: dict, endpoint: str = 'data', request_type: str = 'application/x-hdf(eos|5)')[source]
Filter the CMR json response for desired data files
- Parameters
- search_results: dict
json response from CMR query
- endpoint: str, default ‘data’
url endpoint type
'data'
: NASA Earthdata https archive'opendap'
: NASA Earthdata OPeNDAP archive's3'
: NASA Earthdata Cumulus AWS S3 bucket
- request_type: str, default ‘application/x-hdfeos’
data type for reducing CMR query
- Returns
- producer_granule_ids: list
ICESat-2 granules
- granule_urls: list
ICESat-2 granule urls from NSIDC
- icesat2_toolkit.utilities.cmr(product: str = None, release: str = None, cycles: str | int | list | None = None, tracks: str | int | list | None = None, granules: str | int | list | None = None, regions: str | list | None = None, resolutions: str | list | None = None, bbox: list | None = None, start_date: str | None = None, end_date: str | None = None, provider: str = 'NSIDC_ECS', endpoint: str = 'data', request_type: str = 'application/x-hdf(eos|5)', opener=None, verbose: bool = False, fid=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>)[source]
Query the NASA Common Metadata Repository (CMR) for ICESat-2 data
- Parameters
- product: str or NoneType, default None
ICESat-2 data product to query
- release: str or NoneType, default None
ICESat-2 data release to query
- cycles: str, list or NoneType, default None
91-day orbital cycle strings to query
- tracks: str, list or NoneType, default None
Reference Ground Track (RGT) strings to query
- granules: str, list or NoneType, default None
ICESat-2 granule region strings to query
- regions: str, list or NoneType, default None
ICESat-2 ATL07 or ATL14/15 region strings to query
- resolutions: str, list or NoneType, default None
ICESat-2 ATL14/15 resolution strings to query
- bbox: list or NoneType, default None
Spatial bounding box for CMR query in form (
lon_min
,lat_min
,lon_max
,lat_max
)- start_date: str or NoneType, default None
starting date for CMR product query
- end_date: str or NoneType, default None
ending date for CMR product query
- provider: str, default ‘NSIDC_ECS’
CMR data provider
- endpoint: str, default ‘data’
url endpoint type
'data'
: NASA Earthdata https archive'opendap'
: NASA Earthdata OPeNDAP archive's3'
: NASA Earthdata Cumulus AWS S3 bucket
- request_type: str, default ‘application/x-hdfeos’
data type for reducing CMR query
- opener: obj or NoneType, default None
OpenerDirector instance
- verbose: bool, default False
print file transfer information
- fid: obj, default sys.stdout
open file object to print if verbose
- Returns
- producer_granule_ids: list
ICESat-2 granules
- granule_urls: list
ICESat-2 granule urls from NSIDC