Downloader Configuration Objects

Configuration objects represent common configuration across a set of download requests. They are arbitrary objects in that any keyword value passed to the constructor will be a field in the configuration object. However, only a very specific set of fields are honored by the Downloader objects.

Construct Signature:

def __init__(self, **kwargs):

The currently honored fields (read: keyword arguments) are:

  • basic_auth_username
  • basic_auth_password
  • headers
  • max_concurrent
  • max_speed
  • proxy_url
  • proxy_port
  • proxy_username
  • proxy_password
  • ssl_validation
  • ssl_ca_cert
  • ssl_ca_cert_path
  • ssl_client_cert
  • ssl_client_cert_path
  • ssl_client_key
  • ssl_client_key_path

This list will continue to grow and evolve as more downloaders are added, especially downloaders that support protocols other than HTTP and HTTPS.

Download Control

max_concurrent is an integer that tells the downloader the maximum number of files to download concurrently (read: in parallel). If this number is not provided, each downloader has its own default value that will be used instead.

max_speed is an integer that tells the downloader at what speed to throttle the downloads. The units are: bytes/second.

HTTP Basic Auth Support

The fields basic_auth_username and basic_auth_password are used for the HTTP basic authorization header. The username and password fields must be provided in plain text. The downloaders will Base64 encode them.

SSL Support

ssl_validation is a boolean that tells the downloader to verify the identity of the remote server by checking its SSL certificate. If this parameter is not provided, validation is assumed to be set to True.

ssl_ca_cert and ssl_ca_cert_path parameters are used to provide an alternative CA cert to the downloader. The ssl_ca_cert parameter should point the CA pem data and the ssl_ca_cert_path is a file system path to the CA cert file. Both are strings. However, these parameters are mutually exclusive, and the behavior of the downloader is undefined if both are provided.

ssl_client_cert, ssl_client_cert_path, ssl_client_key, and ssl_client_key_path are used to provide two-way authentication via the SSL protocol. Just like the ssl_ca_cert params, these point to either the data or to a file path; and correlated parameters are mutually exclusive.

Proxy Support

proxy_url is string in the form of scheme://host, where scheme is either http or https.

proxy_port is an integer port number.

proxy_username and proxy_password are used for authentication and must be provided in plain text.

Headers

headers is a dictionary that can contain any additional headers that should be used for every request.