From: http://www.cyberciti.biz/tips/download-accelerator-for-linux-command-line-tools.html
Linux ultra fast command line download accelerator
Usually I use wget, which is ultimate command line downloader. However, wget failed to accelerate my download speed. I was downloading 1.4GB file around 800KB/s download speed (this box is hooked to uplink port speed 10 Mbps ). However, remote server was restricting me.
Therefore, I decided to get rid of wget. I have downloaded axel – a light download accelerator for Linux command line.
How Axel works?
Axel does the same thing any other accelerator does: it opens more than one HTTP/FTP connection per download and each connection transfers its own, separate, part of the file. It may sound weird, but it works very well in practice. For example, some FTP sites limit the speed of each connection, therefore opening more than one connection at a time multiplies the allowable bandwidth. Be forewarned that some FTP operators don’t like it when you do this. It is better to open the additional connections on several servers and download from all of them simultaneously. Axel supports this feature too. You may either specify a list of mirrors to use or tell the program to search for mirrors.
This is a perfect tool over remote ssh session for downloading large file.
Install axel
If you are using Debian, type the following command to install axel:
# apt-get install axel
Or you can download axel from official website:
$ wget http://wilmer.gaast.net/downloads/axel-1.0b.tar.gz
Untar the axel:
$ tar -zxvf axel-1.0b.tar.gz
Configre and compile axel:
$ ./configure
Install axel:
# make install
On the other hand, just upload a newly build axel binary to remote Linux server using scp. Usually I do not install gcc c/c++ compilers collection on any of my production web/ftp/MySql servers for security reasons.
How do I use axel?
Just type the command as follows:
$ axel http://download.com/file.tar.gz
Limit speed
You can also specify a speed (bytes per sec) for axel so that it will not eat up all your bandwidth. For example following will try to keep the average speed around 5242880 (5120 Kilobyte per/sec):
$ axel -s 5242880 http://download.com/my.iso
Limit a number of connection
You can also specify number of connection you want to open. For example open 3 connections for downloading:
$ axel -n 3 -s 5242880 http://download.com/my.iso
But how fast is axel?
Here is sample test that demonstrates how fast is axel
$ wget http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.17.11.tar.bz2
Output:
–12:10:31– http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.17.11.tar.bz2
=> `linux-2.6.17.11.tar.bz2′
Resolving kernel.org… 204.152.191.5, 204.152.191.37
Connecting to kernel.org|204.152.191.5|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 41,288,067 (39M) [application/x-bzip2]
100%[================================================================================>] 41,288,067 2.33M/s ETA 00:00
12:10:48 (2.31 MB/s) – `linux-2.6.17.11.tar.bz2′ saved [41288067/41288067]
$ axel http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.17.11.tar.bz2
Output:
Initializing download: http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.17.11.tar.bz2
File size: 41288067 bytes
Opening output file linux-2.6.17.11.tar.bz2.1
Starting download
[ 0%] ………. ………. ………. ………. ………. [ 247.1KB/s]
[ 0%] ………. ………. ………. ………. ………. [ 408.3KB/s]
[ 0%] ………. ………. ………. ………. ………. [ 566.3KB/s]
[ 0%] ………. ………. ………. ………. ………. [ 707.2KB/s]
[ 0%] ………. ………. ………. ………. ………. [ 836.5KB/s]
[ 0%] ………. ………. ………. ………. ………. [ 975.9KB/s]
[ 0%] ………. ………. ………. ………. ………. [1079.9KB/s]
[ 0%] ………. ………. ………. ………. ………. [1210.0KB/s]
[ 0%] ………. ………. ………. ………. ………. [1303.1KB/s]
[ 1%] ………. ………. ………. ………. ………. [1422.1KB/s]
[ 1%] ………. ………. ………. ………. ………. [1508.0KB/s]
[ 1%] ………. ………. ………. ………. ………. [1629.2KB/s]
……….
…
….
[ 99%] ………. ………. ………. ………. ………. [8710.2KB/s]
[ 99%] ………. ………. ………. ………. ………. [8680.7KB/s]
[100%] ………. ……….
Downloaded 39.4 megabytes in 4 seconds. (8681.65 KB/s)
As you see axel downloaded same file in 4 seconds. Another great thing its binary size, I can put axel on boot disk and replace a wget.
