hydrotools.nwm_client.FileDownloader module

File Downloader

Tool for downloading files asynchronously.

Classes

FileDownloader

class hydrotools.nwm_client.FileDownloader.FileDownloader(output_directory: str | ~pathlib.Path = PosixPath('.'), create_directory: bool = False, ssl_context: ~ssl.SSLContext = <ssl.SSLContext object>, limit: int = 10)

Bases: object

Provides a convenient interface to download a list of files asynchronously using HTTP.

property create_directory: bool
get(src_dst_list: List[Tuple[str, str]], overwrite: bool = False) → None

Setup event loop and asynchronously download multiple files. If self.create_directory is True, an output directory will be created if needed.

Parameters:

  • src_dst_list (List[Tuple[str,str]], required) – List of tuples containing two strings. The first string is the source URL from which to retrieve a file, the second string is the local filename where the file will be saved.

  • overwrite (bool, optional, default False) – If True will overwrite destination file, if it exists. If False, download of this file is skipped.

Return type:

None

Examples

>>> from nwm_client.FileDownloader import FileDownloader
>>> downloader = FileDownloader(output_directory="my_output")

>>> # This will download the pandas homepage and save it to "my_output/index.html"
>>> downloader.get(
>>>     [("https://pandas.pydata.org/docs/user_guide/index.html","index.html")]
>>>     )
async get_file(url: str, filename: str, session: ClientSession) → None

Download a single file.

Parameters:

  • url (str, required) – URL path to file.

  • filename (str, required) – Local filename used to write downloaded file. This will save the file to self.output_directory/filename

  • session (aiohttp.ClientSession, required) – Session object used for retrieval.

Return type:

None

async get_files(src_dst_list: List[Tuple[str, str]]) → None

Asynchronously download multiple files.

Parameters:

src_dst_list (List[Tuple[str,str]], required) – List of tuples containing two strings. The first string is the source URL from which to retrieve a file, the second string is the local filename where the file will be saved.

Return type:

None

property limit: int
property output_directory: Path
property ssl_context: SSLContext