Deduplicating URL Lists
Paste a list of URLs, one per line, and this tool removes duplicates and reports how many it stripped. Comparison ignores protocol, hostname capitalisation and trailing slashes, so variants of the same page are recognised as duplicates rather than treated as distinct.
That normalisation is the point. A naive deduplicator comparing raw strings leaves http://Example.com/page/ and https://example.com/page in the list as two entries, which is exactly the case you were trying to clean up.
What counts as a duplicate here
- Protocol differences — http and https versions of the same address.
- Hostname case — Example.com and example.com.
- Trailing slashes — /page and /page/.
- Exact repeats, including ones separated by hundreds of lines.
What is deliberately not treated as a duplicate
Path case is preserved. /Page and /page stay as separate entries, because many servers genuinely serve different content at those addresses and collapsing them could hide a real problem.
Query strings are compared in full, so /page?a=1 and /page?b=2 remain distinct — as they should, since parameters usually change what the page shows. Note that this also means /page?a=1&b=2 and /page?b=2&a=1 survive as two entries even though they serve identical content.
www and non-www hostnames are treated as different. They frequently are configured differently, and merging them silently could remove a URL you needed.
Where duplicate URL lists come from
Crawl exports are the most common source. A crawler following internal links will find the same page through several routes, and unless the crawl was configured to normalise, the export contains every variant it encountered.
Merged lists are the second: combining a sitemap export with a Search Console export with a list someone sent you produces overlap by definition.
Migration planning is the third, and the highest-stakes. A redirect mapping spreadsheet with duplicate source URLs produces conflicting rules, and which one wins depends on your server configuration rather than your intent.
Why this matters for sitemaps
A sitemap listing the same page at several URLs sends a contradictory signal about which version is canonical. It also wastes crawl requests on URLs that will be consolidated anyway.
The correct sitemap contains one entry per page — the canonical, indexable version returning HTTP 200. Deduplicating before generating is the simplest way to get there.
The usual sequence is: normalise the list with the Bulk URL Formatter, deduplicate here, confirm nothing redirects using the Redirect Chain Checker, then build the file with the Sitemap Generator.
Duplicates in your list versus duplicates on your site
This tool cleans a list. It does not fix the underlying reason your site serves the same content at several addresses — that is a separate problem with a separate fix.
If both http and https versions are genuinely reachable, you need a redirect forcing one. If both www and non-www resolve, same. If tracking parameters create indexable variants, you need a self-referencing canonical tag on every page.
Deduplicating the list without fixing the site means the duplicates come back the next time you export.
A practical cleanup workflow
- Export your URLs from a crawler, sitemap or Search Console.
- Strip tracking parameters with the URL Cleaner if the export contains campaign URLs.
- Normalise casing and trailing slashes with the Bulk URL Formatter.
- Deduplicate here and note how many were removed — a high number usually indicates a site-level configuration issue worth fixing.
- Spot-check a few for redirects before treating the list as final.
Frequently asked questions
Does it treat http and https as duplicates?
Yes. Comparison ignores the protocol, so http://example.com/page and https://example.com/page are recognised as the same page and only one is kept.
Why are /Page and /page kept separately?
Path case is preserved deliberately. Many servers serve different content at differently-cased paths, so collapsing them could remove a genuine URL or hide a real duplicate-content problem you should fix at the server.
Does it remove URLs with different query strings?
No. Query strings are compared in full, since parameters usually change what the page shows. Strip tracking parameters with the URL Cleaner first if you want those variants collapsed.
Are www and non-www treated as the same?
No, they are kept separate. The two hostnames are frequently configured differently, so merging them automatically could remove a URL you needed.
Will this fix duplicate content on my site?
No — it cleans a list, not a site. If the same page is genuinely reachable at several addresses, you need redirects or canonical tags. Otherwise the duplicates reappear in your next export.