Scraping websites with wget and httrack· 2 minute read
Scrapes can be useful to take static backups of websites or to catalogue a site before a rebuild. If you do online courses then it can also be useful to have as much of the course material as possible locally. Another use is to download HTML only ebooks for offline reading.
There are two ways that I generally do this - one on the command line with
wget and another through the GUI with
httrack. By far the easiest if you want an entire site is the
wget method so I’ll introduce that first.
I like to use the following command so that a browseable local copy is created. Two of the options that are useful to ensure this are
--restrict-file-names=windows. The former converts any links into local relative URLs so that the site can be browsed locally and I am using
--restrict-file-names for the purpose of ensuring safe file names. This is particularly relevant when the URLs you’re trying to scrape contain parameters.
wget -H -r --level=5 --restrict-file-names=windows --convert-links -e robots=off http://example.org
The rest of the options can easily be looked up on the
wget manual page so I’ll leave that as an exercise for the reader to save some time.
For more complicated scrapes and those that require authentication in particular
httrack is very handy. You do need a GUI though and it uses Chrome underneath to request the pages from what I can tell - at least for the WebHTTrack software for linux. There is a Windows version (I have never used it) which seems to have a typical Windows application interface from the screenshots on their site so I can’t be sure what this doing underneath.
The options for httrack are reasonably well documented on their site, but when you start it up it will walk you through a relatively straight forward wizard process anyway. Once it has started you’re able to pause and restart downloads, which is a nice feature and it automatically fixes the URLs so the site is browseable locally.
As I read on the train and complete course work I find both methods very handy - can also be used to download the videos too!
Image credit: www.flickr.com/photos/21649179@N00/9009772399