Remove duplicate lines
Remove repeated lines from text, keeping the first occurrence of each and the original order.
Input
Result
Fill in the input and press Run.
Your input is processed in memory and then discarded. It is not stored or logged. Even so, don't paste live passwords or keys into any online tool.
Use it from code or an agent
curl -s https://setget.net/v1/tools/dedupe-lines \
-H 'Content-Type: application/json' \
-d '{"input":{"text":"a\nb\na\nc\nb"}}'
import json, urllib.request
body = {'input': {'text': 'a\nb\na\nc\nb'}}
request = urllib.request.Request("https://setget.net/v1/tools/dedupe-lines", json.dumps(body).encode(),
{"Content-Type": "application/json"})
print(json.load(urllib.request.urlopen(request))["result"])
const response = await fetch("https://setget.net/v1/tools/dedupe-lines", {
method: "POST",
headers: {"Content-Type": "application/json"},
body: JSON.stringify({"input":{"text":"a\nb\na\nc\nb"}}),
});
console.log((await response.json()).result);
Tool: run_tool
Arguments:
{
"slug": "dedupe-lines",
"input": {
"text": "a\nb\na\nc\nb"
}
}
{
"input": {
"text": "a\nb\na\nc\nb"
}
}
Endpoint: POST /v1/tools/dedupe-lines. Over MCP: run_tool with slug: "dedupe-lines".
Machine-readable: input schema, Markdown. Connect an agent.
Inputs
| Name | Type | Required | What it does |
|---|---|---|---|
text |
string | yes | The lines to clean up. |
case_sensitive |
boolean | no | Treat upper and lower case as different lines. Default: true. |
trim |
boolean | no | Ignore spaces at the start and end of lines when comparing. Default: false. |
keep_blank |
boolean | no | Keep every blank line instead of treating blank lines as duplicates. Default: false. |
Examples
Input
{
"text": "a\nb\na\nc\nb"
}Result
{
"text": "a\nb\nc",
"lines": 3,
"removed": 2
}Ignoring case and stray spaces
Input
{
"text": "Sam\nsam \nAlex",
"case_sensitive": false,
"trim": true
}Result
{
"text": "Sam\nAlex",
"lines": 2,
"removed": 1
}Questions
- Do the lines need to be sorted first?
- No. Duplicates are found anywhere in the text, and the remaining lines keep their order.
- Which copy is kept?
- The first one, exactly as it was written.
Related tools
- Sort linesSort lines alphabetically, naturally (file2 before file10), by number or by length, or reverse their order.
sort-lines - Find and replaceReplace every occurrence of a word or phrase in text, exactly as typed, with a count of the changes.
find-replace - CSV sort, filter and dedupeSort CSV by a column, keep rows containing some text, pick columns and remove duplicate rows.
csv-sort-filter-dedupe - Apply a patchApply a unified diff to a text and get the patched result, with clear errors when a hunk does not fit.
patch-apply - Case converterConvert text between lower, UPPER, Title, Sentence, camelCase, PascalCase, snake_case, kebab-case and more.
case-convert - Escape and unescape textEscape or unescape text for HTML, URLs, JSON strings, shell commands, regular expressions and CSV fields.
escape-unescape