Remove duplicate rows from CSV

Your files and your privacy

The rows being compared never leave your device. The file is read by the browser, parsed by a Web Worker in this tab, de-duplicated in memory and written back out by the browser’s download mechanism. The page’s Content-Security-Policy limits connect-src to this origin.

What this tool does

Remove rows that repeat a row already seen earlier in the file. The comparison covers every column, uses the parsed cell values, is case-sensitive and treats whitespace as significant. The first occurrence survives and the order of the surviving rows is unchanged.

De-duplication walks the data rows in order and keeps only the first appearance of each. A row is a duplicate when every one of its cells matches an earlier row exactly, compared after parsing, so a field written inside quotes equals the same field written without them. Case and whitespace both count: Ada, ada and Ada with a trailing space are three distinct rows. Duplicates need not be adjacent, and the rows that survive stay where they were.

How to use it

1. Select Choose a CSV or JSON file and load the export.
2. Note the row count first, because the tool reports how many rows remain rather than how many it removed.
3. Select Trim whitespace if cells may be padded, since rows differing only by a space are not duplicates.
4. Select Remove duplicate rows, read the remaining count in the message, then select Download CSV or Undo.

Example use cases

• A mailing list assembled from several exports, where the same address was contributed twice.
• A report downloaded twice and concatenated, so a whole block of rows appears again further down.
• Records re-inserted by a sync job that ran twice before anybody noticed.

Supported input and output

Accepts

• CSV, TSV or plain text separated by comma, semicolon, tab or pipe, up to 50 MB
• JSON containing an array of objects

Produces

• CSV, UTF-8, CRLF line endings, surviving rows in their original order
• JSON, a two-space indented array of objects

What this tool will not do

• The comparison covers every column, so two rows describing the same person both survive if any column differs, such as a row id or an export timestamp. Deselect that column with Apply column selection first.
• There is no by-column or case-insensitive control on this page, so Ada and ada are two rows.
• Whitespace is part of the value, so trim or collapse spaces first or padded copies survive.
• The header row is never compared and never removed. A second header line inside a concatenated file is an ordinary data row: it is unique, so it stays.
• A short row the parser padded with blanks can end up identical to a longer row whose trailing cells are empty, and one of the two is dropped.

Common mistakes

• De-duplicating before trimming. A trailing space makes two identical-looking rows distinct, both are kept, and nothing says duplicates remain.
• Expecting the most recent copy to win. The first occurrence survives, so if later rows carry corrected values you keep the stale version.
• Using this button to clear blank rows. Blank rows are identical to each other, so exactly one survives; Remove empty rows is what clears them.
• Spot-checking the preview afterwards: only the first 100 rows are shown, so duplicates below that line are removed or kept unseen.

Technical notes

Each row is reduced to an identity key by joining its cells with a NUL character, which cannot legally appear in CSV text. Joining on a comma instead would let one cell containing a,b collide with two cells containing a and b, which is how a de-duplicator silently deletes the wrong row. The keys go into a hash set, so the scan is one linear pass and duplicates are found wherever they sit rather than only when adjacent, and the transformation returns a new array so Undo restores the previous table exactly.

Frequently asked questions

Does it keep the first or the last copy?

The first. The original row stays in place and later copies are dropped, so the order of the data you keep is unchanged. If the later copies are the corrected ones, sort or edit the file so the version you want comes first.

Can I de-duplicate by one column, such as email?

Not directly; the button compares whole rows. You can tick only that column, select Apply column selection and then de-duplicate, but that keeps only the ticked column, so undo afterwards if you need the other fields.

Why is an obvious duplicate still there?

Something in the row differs: a trailing space, a non-breaking space pasted from a web page, a different letter case, or a column you had forgotten such as a row id or an export timestamp.