How to Automatically Check a Website for Broken Links Using Wget in Linux

Broken links (404 errors) on a website severely degrade the user experience and can penalize your search engine rankings. If you manage a large website or a complex documentation portal, manually clicking every single link on every page to ensure they still work is practically impossible. Fortunately, Linux users have access to a powerful command-line utility called wget. While primarily used for downloading files, wget can be configured to automatically crawl your entire website, follow every internal link, and generate a comprehensive report of any broken URLs it discovers.

How to Crawl a Website with Wget

The wget utility is installed by default on almost all Linux distributions, including Ubuntu, Debian, and CentOS. To turn it into a site crawler, we must use a specific combination of flags.

1. Open your Linux terminal application or SSH into your server.
2. To initiate a scan of your website, run the following command, replacing example.com with your actual website URL: wget --spider -r -nd -nv -l 5 -o broken_links_report.txt https://example.com

Understanding the Wget Flags

Because that command is quite dense, it is important to understand exactly what each flag is instructing the utility to do:

• –spider: This is the most crucial flag. It tells wget to act like a web spider. It will only check if the files exist; it will not actually download the HTML pages or images to your hard drive, saving you massive amounts of bandwidth and storage space.
• -r (recursive): This tells the spider to follow the links it finds on the homepage and navigate deeper into your site.
• -nd (no directories): Prevents wget from creating a complex hierarchy of empty folders on your local machine while it crawls.
• -nv (non-verbose): Suppresses the massive wall of text that wget usually outputs to the terminal, showing only errors and essential information.
• -l 5 (level): Limits the crawl depth to 5 clicks away from the homepage. This prevents the spider from getting stuck in an infinite loop on massive websites.
• -o (output): Directs all the results of the scan into a neatly formatted text file (in this case, broken_links_report.txt) instead of printing it to your screen.

How to Read the Broken Links Report

Depending on the size of your website, the scan may take anywhere from a few seconds to several minutes to complete. The terminal will appear frozen while it runs in the background. Once the command prompt returns, the scan is finished.

You can view the results by running: cat broken_links_report.txt | grep "404"

This will instantly filter the report and display a list of all the exact URLs on your website that returned a 404 Not Found error, allowing you to quickly log into your CMS and update or remove the broken hyperlinks.