urlgrabber

urlgrabber is a pure python package that drastically simplifies
the fetching of files. It is designed to be used in programs
that need common (but not necessarily simple) url-fetching
features. It is extremely simple to drop into an existing
program and provides a clean interface to protocol-independant
file-access. Best of all, urlgrabber takes care of all those
pesky file-fetching details, and lets you focus on whatever it
is that your program is written to do!

urlgrabber came into existence as the part of yum that
downloads rpms and header files, but it quickly became clear
that this is a general problem that many applications must deal
with.

Features

Using urlgrabber, data can be fetched in three basic ways:

urlgrab(url) copy the file to the local filesystem
urlopen(url) open the remote file and return a file object
urlread(url) return the contents of the file as a string

When using these functions (or methods), urlgrabber supports the
following features:

  • identical behavior for http://, ftp://, and file:// urls
  • http keepalive - faster downloads of many files by using
    only a single connection
  • byte ranges - fetch only a portion of the file
  • reget - for a urlgrab, resume a partial download
  • progress meters - the ability to report download progress
    automatically, even when using urlopen!
  • throttling - restrict bandwidth usage
  • batched downloads using threads - download multiple files
    simultaneously (feature still in progress)
  • retries - automatically retry a download if it fails. The
    number of retries and failure types are configurable
  • authenticated server access for http and ftp
  • proxy support - support for authenticated http and ftp
    proxies
  • mirror groups - treat a list of mirrors as a single
    source, automatically switching mirrors if there is a
    failure
  • broad support - unix and windows, python 2.3 - 2.5 (it
    currently works for and is tested against 2.2 also, but that
    will be dropped if it becomes difficult)

Not sure if urlgrabber is the tool for you? Check out our comparison of the major options.

Documentation, Examples, and Help

There are many sources of urlgrabber-related assistance and
information

  • The urlgrabber package
    documentation, built from the __doc__ strings
    using pydoc
  • The examples page
  • Browsable urlgrabber
    git repo
  • The yum-devel
    mailing list. For now, urlgrabber is piggy-backing on this
    list. If it becomes necessary, we will get our own list.
    When posting to this list, please indicate that it is a
    urlgrabber-related post by beginning the subject with
    [UG].
声明：本站所有文章，如无特殊说明或标注，均为本站原创发布。任何个人或组织，在未征得本站同意时，禁止复制、盗用、采集、发布本站内容到任何网站、书籍等各类媒体平台。如若本站内容侵犯了原著者的合法权益，可联系我们进行处理。