downloader.sh

Introduction

Context

Let's consider the following case : you want to download pitures on a web server (one hundrer for example), neither using a complex software suite dedicated to massive downloading nor clicking one hundred times on a line to save the image (or typing one hundred shell commands to consider the console fans, even if in my opinion these one are laughing at me doing a whole tutorial for a that simple script). The proposed solution uses wget in a standard for loop structure.

Compatibility disclaimer

As I'm working under OpenBSD, I shall precise that this scripts might be compatible with most UNIX likes operating systems. Indeed it does not uses GNU/Linux specific commands such as seq which is used to generate a sequence of numbers, often used with for loop structures (NdM : the seqUsage: seq [OPTION]... LAST or: seq [OPTION]... FIRST LAST or: seq [OPTION]... FIRST INCREMENT LAST Displays the numbers from FIRST to LAST, with the INCREMENT step. command is included in the GNU shellutils package). Hence, the code can sometimes be optimized with such commands and you are invited to do so whenever you can.

For the same reason, I do not guarantee that some structures such as the use of parentheses instead of brackets fot test structures might not work properly under GNU/Linux distros (I don't have such a machine available to test this).

The shell used to run this script is the simple sh.

Operating the script

To test this script, you need only a standard text editor such as vi/vim, emacs or even nano to write down the commands in a .sh ascii file. Once the file created, you need to make it executable thanks to the chmodUsage: chmod [OPTION]... MODE[,MODE]... FILE... or: chmod [OPTION]... MODE-OCTAL FILE or: chmod [OPTION]... --reference=FILE-R FILE Change the mode of each file according to the selected MODE. command. Hence, to make the downloader.sh executable, you would enter the following command : chmod +x downloader.shUsage: chmod [OPTION]... MODE[,MODE]... FILE... or: chmod [OPTION]... MODE-OCTAL FILE or: chmod [OPTION]... --reference=FILE-R FILE Change the mode of each file according to the selected MODE..

First attempt

For a first version of this script, we will write the main part of the code "hard" to complete the task we want. We will operate the wgetGNU Wget 1.10.2, un récupérateur réseau non interactif. Usage: wget [OPTION]... [URL]... inside a loop structure (a do sctructure) to automate the downloading of a file pool which are named using a sequence of numbers.

Let's assume that we want to download pictures named IMG0102.JPG to IMG0193.JPG on the address http://www.address.com/photos/. To do so, we use a do loop which operates the following command : wget http://www.adresse.com/photos/IMG0xx.JPGGNU Wget 1.10.2, un récupérateur réseau non interactif. Usage: wget [OPTION]... [URL]... for each file, incrementing xx for each turn. Here is the code :

#!/usr/local/bin/bash

$i=102
while test $i -le $193
do
  wget http://www.address.com/photos/IMG0$min.JPG
  i=$(( $i + 1 ))
done

This script will do the job but is limited in adaptativeness and reusability : you will have to modify the source code each time you want to do massive downloading.

Script tuning

We start from the first attempt that uses a loop and we ameliorate the adaptability of the code. The idea here is to be able to pass the minimum and the amximum numbers of the files in arguments, as well as the beginning and the end of the string patterns. In our case, the arguments we would pass are the following :

greatest lower bound : 102
least upper bound : 193
beginning of the URL : http://www.address.com/photos/IMG0
end of the URL : .JPG

As we are using arguments in our script, we implement the interactivity of the script by verifying the numbers of arguments that are passed and by displaying a help message if this number is incorrect (the well-known USAGE:). Here is the final version of the code :

#!/usr/local/bin/bash

if (test -z $1) && (test -z $2) && (test -z $3) && (test -z $4)
then
  echo "Usage: $0 url id_min id_max extension"
  exit
fi

url=$(($1))
min=$(($2 + 0))
max=$(($3 + 0))
ext=$(($4))

while test $min -le $max
do
  wget $1$min$4

  min=$(( $min + 1 ))
done

Documentation pubished under the GNU/Free Documentation Licence version 1.2
http://www.gnu.org/copyleft/fdl.html
Christophe Eyquem