Remove duplicate lines

Remove repeated lines from text, keeping the first occurrence of each and the original order.

Input

The lines to clean up.
Treat upper and lower case as different lines.
Ignore spaces at the start and end of lines when comparing.
Keep every blank line instead of treating blank lines as duplicates.

Result

Fill in the input and press Run.

Your input is processed in memory and then discarded. It is not stored or logged. Even so, don't paste live passwords or keys into any online tool.

Use it from code or an agent

curl -s https://setget.net/v1/tools/dedupe-lines \
  -H 'Content-Type: application/json' \
  -d '{"input":{"text":"a\nb\na\nc\nb"}}'
import json, urllib.request

body = {'input': {'text': 'a\nb\na\nc\nb'}}
request = urllib.request.Request("https://setget.net/v1/tools/dedupe-lines", json.dumps(body).encode(),
                                 {"Content-Type": "application/json"})
print(json.load(urllib.request.urlopen(request))["result"])
const response = await fetch("https://setget.net/v1/tools/dedupe-lines", {
  method: "POST",
  headers: {"Content-Type": "application/json"},
  body: JSON.stringify({"input":{"text":"a\nb\na\nc\nb"}}),
});
console.log((await response.json()).result);
Tool: run_tool
Arguments:
{
  "slug": "dedupe-lines",
  "input": {
    "text": "a\nb\na\nc\nb"
  }
}
{
  "input": {
    "text": "a\nb\na\nc\nb"
  }
}

Endpoint: POST /v1/tools/dedupe-lines. Over MCP: run_tool with slug: "dedupe-lines". Machine-readable: input schema, Markdown. Connect an agent.

Inputs

NameTypeRequiredWhat it does
text string yes The lines to clean up.
case_sensitive boolean no Treat upper and lower case as different lines. Default: true.
trim boolean no Ignore spaces at the start and end of lines when comparing. Default: false.
keep_blank boolean no Keep every blank line instead of treating blank lines as duplicates. Default: false.

Examples

Input
{
  "text": "a\nb\na\nc\nb"
}
Result
{
  "text": "a\nb\nc",
  "lines": 3,
  "removed": 2
}

Ignoring case and stray spaces

Input
{
  "text": "Sam\nsam \nAlex",
  "case_sensitive": false,
  "trim": true
}
Result
{
  "text": "Sam\nAlex",
  "lines": 2,
  "removed": 1
}

Questions

Do the lines need to be sorted first?
No. Duplicates are found anywhere in the text, and the remaining lines keep their order.
Which copy is kept?
The first one, exactly as it was written.