Multiple File Download Utility Using wget – mget

mget allows unattended download of multiple files. The script is also ideal for use with cron, allowing, for example, overnight downloads.

The script is available as an AutoTools tarball or as a package for; Debian, Raspbian, Ubuntu, Fedora and openSUSE.

How mget works

This download utility consists of two components; a file containing the URLs of the files to download, and the script which processes the file and downloads the target files. The file containing the URLs can be named and located as you wish.

The easiest explanation is by example:-

• Create a file in your Download directory

  • /home/your_user_name/Downloads/download-requests.txt

• Place the following URLs in it:-

  • https://www.debian.org/doc/debian-policy/policy.txt

  • https://www.google.com/intl/en/policies/privacy/google_privacy_policy_en.pdf

• Type:-

  • mget -S /home/your_user_name/Downloads/download-requests.txt -T /home/your_user_name/Downloads

• and check the results which will have been placed in the target directory (-T) of /home/your_user_name/Downloads

If you want to use the same file and target directory each time, add the -p or –persist command line option to the above command and those locations will be stored for future use and a simple ‘mget’ command will perform the same command as above.

As mget processes each URL in the file that entry will be deleted if the download is successful.

Installing the mget package

The multiple file download package is called mget.
To prevent bitrot of this blog, please follow the installation instructions for your distribution at the project wiki:-
https://github.com/m-grant-prg/mget/wiki

An example cron setup

If you wanted to run the mget download overnight, every day except Friday at 00:10, you edit your crontab file by typing_:-_

crontab -e

and then entering the following line:-

10 00 * * 0,1,2,3,4,6 mget --quiet --sourcefile /home/your_user_name/Downloads/MultiGetSource.txt --targetdir /home/your_user_name/Downloads

Authoritative sources

The authoritative source for this project is held on GitHub at:-

The code repository:- https://github.com/m-grant-prg/mget

The wiki:- https://github.com/m-grant-prg/mget/wiki

To view or raise issues please visit:- https://github.com/m-grant-prg/mget/issues

My Projects

• Software and Debian packaging sources can be found on GitHub: https://github.com/m-grant-prg

• Debian, Fedora, openSUSE and Raspbian packages can be found on the openSUSE Build Service: https://build.opensuse.org/project/show/home:m-grant-prg

• Ubuntu packages can be found in my PPA on LaunchPad: https://code.launchpad.net/~m-grant-prg/+archive/ubuntu/utils

• The entry point to my Ubuntu packages on LaunchPad: https://launchpad.net/~m-grant-prg