Deduplicating URL Lists
Paste a list of URLs, one per line, and this tool removes duplicates and reports how many it stripped. Comparison ignores protocol, hostname capitalization and trailing slashes, so variants of the same page are recognized as duplicates rather than treated as distinct.
That normalization is the point. A naive deduplicator comparing raw strings leaves http://Example.com/page/ and https://example.com/page in the list as two entries, which is exactly the case you were trying to clean up.
How to Use This Tool
Paste your URL list
From a crawl, a sitemap, analytics, a backlink export — or several combined.
Let it compare
URLs that differ only in hostname capitalisation, http versus https, or a trailing slash are treated as the same.
Copy the unique list
The first occurrence of each URL is kept in its original order.
What Counts as a Duplicate Here
- Protocol differences — http and https versions of the same address.
- Hostname case — Example.com and example.com.
- Trailing slashes — /page and /page/.
- Exact repeats, including ones separated by hundreds of lines.
What is Deliberately Not Treated as a Duplicate
Path case is preserved. /Page and /page stay as separate entries, because many servers genuinely serve different content at those addresses and collapsing them could hide a real problem.
Query strings are compared in full, so /page?a=1 and /page?b=2 remain distinct — as they should, since parameters usually change what the page shows. Note that this also means /page?a=1&b=2 and /page?b=2&a=1 survive as two entries even though they serve identical content.
www and non-www hostnames are treated as different. They frequently are configured differently, and merging them silently could remove a URL you needed.
Where Duplicate URL Lists Come From
Crawl exports are the most common source. A crawler following internal links will find the same page through several routes, and unless the crawl was configured to normalize, the export contains every variant it encountered.
Merged lists are the second: combining a sitemap export with a Search Console export with a list someone sent you produces overlap by definition.
Migration planning is the third, and the highest-stakes. A redirect mapping spreadsheet with duplicate source URLs produces conflicting rules, and which one wins depends on your server configuration rather than your intent.
Why This Matters for Sitemaps
A sitemap listing the same page at several URLs sends a contradictory signal about which version is canonical. It also wastes crawl requests on URLs that will be consolidated anyway.
The correct sitemap contains one entry per page — the canonical, indexable version returning HTTP 200. Deduplicating before generating is the simplest way to get there.
The usual sequence is: normalize the list with the Bulk URL Formatter, deduplicate here, confirm nothing redirects using the Redirect Chain Checker, then build the file with the Sitemap Generator.
Duplicates in Your List Versus Duplicates on Your Site
This tool cleans a list. It does not fix the underlying reason your site serves the same content at several addresses — that is a separate problem with a separate fix.
If both http and https versions are genuinely reachable, you need a redirect forcing one. If both www and non-www resolve, same. If tracking parameters create indexable variants, you need a self-referencing canonical tag on every page.
Deduplicating the list without fixing the site means the duplicates come back the next time you export.
A Practical Cleanup Workflow
- Export your URLs from a crawler, sitemap or Search Console.
- Strip tracking parameters with the URL Cleaner if the export contains campaign URLs.
- Normalize casing and trailing slashes with the Bulk URL Formatter.
- Deduplicate here and note how many were removed — a high number usually indicates a site-level configuration issue worth fixing.
- Spot-check a few for redirects before treating the list as final.
Frequently Asked Questions
Does it treat http and https as duplicates?
Yes. Comparison ignores the protocol, so http://example.com/page and https://example.com/page are recognized as the same page and only one is kept.
Why are /Page and /page kept separately?
Path case is preserved deliberately. Many servers serve different content at differently-cased paths, so collapsing them could remove a genuine URL or hide a real duplicate-content problem you should fix at the server.
Does it remove URLs with different query strings?
No. Query strings are compared in full, since parameters usually change what the page shows. Strip tracking parameters with the URL Cleaner first if you want those variants collapsed.
Are www and non-www treated as the same?
No, they are kept separate. The two hostnames are frequently configured differently, so merging them automatically could remove a URL you needed.
Will this fix duplicate content on my site?
No — it cleans a list, not a site. If the same page is genuinely reachable at several addresses, you need redirects or canonical tags. Otherwise the duplicates reappear in your next export.
How do I remove duplicate URLs in Excel?
Put the URLs in one column and use Data → Remove Duplicates. Excel only removes exact text matches, though, so http and https, or a trailing slash, still count as different — this tool normalises those first.
Should I format URLs before deduplicating?
Yes. Running the list through the Bulk URL Formatter first puts every URL in the same shape, so variants of the same page are caught as duplicates.