Order preserved
- Input
- b a b c
- Output
- b a c
The first occurrence of each line is kept, so a list ordered by relevance or date keeps its ordering. Keeping the last occurrence instead would silently reshuffle it.
Duplicate Row Remover
This tool removes repeated lines from a list — email addresses, URLs, identifiers, log lines. It keeps the first occurrence of each rather than the last, which matters whenever the list is ordered by something: a ranked set of results, a prioritised backlog, a chronological log. It also reports what was duplicated, which is often more interesting than the cleaned list.
Runs entirely in your browser — nothing you paste is uploaded.
How to
How to use the Duplicate Row Remover
- 1
Paste your list
One value per line. Paste from a spreadsheet column, a log file or a query result.
- 2
Choose how to match
Exact matching compares lines literally. Trimmed ignores surrounding spaces, and case-insensitive treats Alice and alice as the same — which is usually right for email addresses.
- 3
Read what was duplicated
The most frequently repeated values are listed with their counts, which often reveals the reason the duplicates exist.
- 4
Copy the cleaned list
Copy or download the result. The original order is preserved unless you ask for it to be sorted.
Examples
Duplicate Row Remover examples
Case-insensitive email addresses
- Input
- Ada@x.com ada@x.com
- Output
- Ada@x.com
The domain part of an email address is case-insensitive and in practice the local part is too, so these are the same address. The first spelling is kept.
Trailing whitespace
- Input
- value value
- Output
- value
A trailing space from a copy-and-paste makes two lines differ. Trimmed matching treats them as one, which is almost always what was meant.
Why use it
What the Duplicate Row Remover gives you
Ordering survives
The first occurrence is kept, so a list sorted by rank, date or priority is still in that order afterwards.
Shows what repeated
The duplicate report with counts frequently explains the underlying problem — a join fanning out, an export run twice.
Matching you can choose
Whitespace and case differences are usually noise, but not always, so how strictly lines are compared is up to you.
Runs in your browser
Lists of email addresses and customer identifiers are exactly the data you should not paste into a random website. This one does not transmit it.
Good to know
Duplicate Row Remover limitations
- It works line by line, so a CSV row with the same values in a different column order is not detected as a duplicate.
- There is no fuzzy matching — near-duplicates such as a misspelt name are treated as distinct values.
- Case-insensitive matching uses simple lowercasing, which is correct for most scripts but not for every locale rule.
- Very large lists are held in memory, so a file of several million lines may be slow.
Summary
Duplicate Row Remover in short
- The first occurrence is kept, so ordering is preserved.
- Matching can ignore surrounding whitespace, letter case, or neither.
- The duplicate report often explains why the duplicates exist.
- Comparison is line by line, not field by field.
- Everything runs in your browser.
FAQ
Duplicate Row Remover questions
Which copy of a duplicate is kept?
The first. That preserves the order of a list sorted by relevance, date or priority. Keeping the last instead would reshuffle the list in a way that is easy to miss and hard to undo.
Should I use case-insensitive matching?
For email addresses and domain names, yes — they are not case-sensitive in practice. For anything case-sensitive, such as passwords, base64 or API keys, no: two values differing only in case are genuinely different.
Why do two identical-looking lines not match?
Almost always trailing whitespace from a copy and paste, or a mix of Windows and Unix line endings. Trimmed matching handles both, which is why it is worth trying when exact matching finds fewer duplicates than you expected.
What use is the duplicate report?
The counts usually explain the cause. A value appearing exactly twice throughout suggests an export was run twice or a join produced a fan-out; one value appearing hundreds of times suggests a default or placeholder leaking into real data.
Can it deduplicate CSV rows?
Only as whole lines. Two rows with identical values written in a different order, or differing only in quoting, will not match. For field-aware comparison, convert to JSON first and compare there.
Should I sort the output?
Only if the input order carried no meaning. Sorting makes a list easier to scan and compare, but it destroys any ordering the list already had — which is the thing keeping the first occurrence was protecting.
What happens to blank lines?
They are dropped by default, since a blank line is rarely meaningful data. They can be kept if the blank lines are structural — separating records, for instance.
How long a list can it handle?
Hundreds of thousands of lines are fine. The whole list and its index are held in memory at once, so several million lines will make the browser struggle.
Is my list uploaded anywhere?
No. Deduplication runs in this page. This is one of the tools where it matters most, because the lists people deduplicate are usually email addresses, customer ids or account numbers.
Discover
Related developer tools
CSV to JSONConvert CSV to JSON in your browser. Handles quoted fields and refuses to turn identifiers like 007 into numbers.JSON to CSVConvert JSON to CSV in your browser. Flattens nested objects into dotted columns and quotes values correctly.JSON CompareCompare two JSON documents structurally. Ignores key order and formatting so only real differences are reported.SQL FormatterFormat SQL for readability with support for PostgreSQL, MySQL, SQLite and more. Runs entirely in your browser.