Downloader Configuration Objects¶
Configuration objects represent common configuration across a set of download requests. They are arbitrary objects in that any keyword value passed to the constructor will be a field in the configuration object. However, only a very specific set of fields are honored by the Downloader objects.
def __init__(self, **kwargs):
The currently honored fields (read: keyword arguments) are:
This list will continue to grow and evolve as more downloaders are added, especially downloaders that support protocols other than HTTP and HTTPS.
max_concurrent is an integer that tells the downloader the maximum number of
files to download concurrently (read: in parallel). If this number is not
provided, each downloader has its own default value that will be used instead.
max_speed is an integer that tells the downloader at what speed to throttle
the downloads. The units are: bytes/second.
HTTP Basic Auth Support¶
basic_auth_password are used for
the HTTP basic authorization header. The username and password fields must be
provided in plain text. The downloaders will Base64 encode them.
ssl_validation is a boolean that tells the downloader to verify the identity
of the remote server by checking its SSL certificate. If this parameter is not
provided, validation is assumed to be set to True.
ssl_ca_cert_path parameters are used to provide an
alternative CA cert to the downloader. The
ssl_ca_cert parameter should
point the CA pem data and the
ssl_ca_cert_path is a file system path to the
CA cert file. Both are strings. However, these parameters are mutually exclusive,
and the behavior of the downloader is undefined if both are provided.
ssl_client_key_path are used to provide two-way authentication via the SSL
protocol. Just like the ssl_ca_cert params, these point to either the data or
to a file path; and correlated parameters are mutually exclusive.
proxy_url is string in the form of scheme://host, where scheme is either
http or https.
proxy_port is an integer port number.
proxy_password are used for authentication and must
be provided in plain text.
headers is a dictionary that can contain any additional headers that should
be used for every request.