Downloading a single file via the Linux terminal is straightforward. But what if you need to download an entire directory of PDFs from a server, or create a complete offline mirror of a static website? Attempting to download each file individually would take hours. Instead, Linux system administrators use the wget command with its powerful recursive flags to automate the mass downloading of linked assets and entire web directory structures.
Why Use Recursive wget?
When run in recursive mode, wget acts like a web crawler. It downloads the target URL, scans the HTML for internal links, and then automatically follows those links to download the next set of files, rebuilding the website’s exact directory structure on your local hard drive. This is incredibly useful for backing up legacy websites, archiving open-source documentation, or pulling down massive datasets.
Step 1: Execute a Basic Recursive Download
To tell wget to follow links and download a directory, you must use the -r (recursive) flag.
- Open your Linux terminal.
- Type the following command to begin a recursive download:
wget -r https://example.com/documents/By default, wget will download the target page and follow links up to 5 levels deep. It will create a folder on your local machine named “example.com” and replicate the exact folder path inside it.
Step 2: Prevent Crawling Unwanted Domains
A dangerous pitfall of recursive downloading is that wget might follow a link to an external website (like Wikipedia) and attempt to download that entire site as well. You must restrict the crawler to the original domain.
- Add the –domains flag to strictly limit the download scope:
wget -r --domains=example.com https://example.com/documents/This guarantees wget will ignore any outbound links that point away from the target website.
Step 3: Convert Links for Offline Viewing
If you are mirroring a static HTML website to read offline, the internal HTML links will still point to the live internet URLs. You can force wget to rewrite these links to point to your newly downloaded local files.
- Use the -k (convert links) flag in combination with the recursive flag:
wget -r -k https://example.com/guide/Once the download finishes, wget will scan all the downloaded HTML files and rewrite the href tags to ensure the offline mirror is fully navigable without an internet connection.
Step 4: Download Specific File Types
If you only want to recursively scrape specific assets, such as PDFs or JPEGs, without downloading the HTML files themselves, use the “accept” flag.
- Add the -A flag followed by the file extension:
wget -r -A.pdf https://example.com/reports/By combining these advanced wget flags, Linux administrators can effortlessly scrape, mirror, and archive massive web directories with a single terminal command.