Duplicate Line Remover lines

What this tool does

Filtering duplicates out of a list is a routine chore: IP addresses pulled from logs, email addresses merged from several files, domains extracted from configuration. Being able to paste and clean immediately beats opening a spreadsheet or assembling a shell pipeline.

Four modes are available. Keeping the first occurrence preserves the original order while removing repeats, and keeping the last lets later values win. Showing only unique lines isolates entries that appear exactly once, while showing only duplicates finds everything that appears more than once, which is the mode that matters when investigating why duplicates exist.

Options control case sensitivity and trimming, and each line can be prefixed with how many times it occurred.

When to use it

Use it to clean a list of IPs or URLs extracted from logs, to merge address lists without repeats, to check whether a dataset contains duplicate keys, or to find configuration entries defined more than once.

Input and output examples

Notes and limitations

Duplicate detection is exact string comparison. Invisible leading or trailing whitespace, or a stray carriage return, makes visually identical lines count as different, which is why trimming is enabled by default. With case-insensitive matching the comparison is lowercased but the output keeps original casing, so which spelling survives depends on the mode you chose. Take care with data such as email addresses where case sensitivity rules vary by context.

Frequently asked questions