pipr.tools
⚡ Pipes
Clean Email Strip formatting from pasted email for clean plain text Clean AI Output Clean plain text from ChatGPT or AI output Decode & Format JWT Decode a JWT and pretty-print header and payload Word Frequency Count word frequency in text
⟐ Data
csv-to-json csv-to-md-table csv-to-table json-flatten json-format json-keys json-minify json-query json-to-csv json-to-toml json-to-yaml sql-format toml-to-json yaml-to-json
</> Web
extract-codeblocks html-to-text md-to-html mermaid-render mustache-render strip-markdown syntax-highlight
{ } Encode
base64-decode base64-encode escape html-decode html-encode jwt-decode qr-encode url-decode url-encode
⇔ Decode
chmod-calc cron-explain nato
# Stats
char-freq token-chunk token-count token-split token-truncate wc word-freq
.* Regex
regex-extract regex-replace
⚡ Gen
ascii-art hash lorem timestamp uuid
◷ Time
epoch-to-human human-to-epoch
Aa Text
case-convert count diff grep head lowercase normalize-quotes number-lines prefix reverse-lines shuffle-lines slugify sort-lines split squeeze strip-ai-slop strip-invisible suffix tail transliterate trim-lines unique-lines uppercase wrap

unique-lines Remove duplicate lines

Aa Text
Input0 chars
Output0 chars

Examples

Deduplicate log error lines

Usage
"error: timeout
error: 404
error: timeout
error: 500
error: 4..." | unique-lines

Remove duplicate emails from a newsletter subscriber list

Usage
"alice@co.io
bob@co.io
alice@co.io
carol@co.io
bob@co.io" | unique-lines

Deduplicate a copy-pasted attendee list from a meeting invite

Usage
"Alice
Bob
Alice
Charlie
Bob
Dave
Charlie" | unique-lines
View source 
(input)=>[
                    ...new Set(input.split("\n"))
                ].join("\n")

Suggested Pipelines

$ sort-lines | unique-lines | number-lines $ trim-lines | unique-lines | wc $ lowercase | unique-lines | sort-lines

Also useful for

Find Unique Errors Split, Sort & Deduplicate Lowercase, Sort & Deduplicate Extract Unique Links Extract Unique Email Addresses Sort & Deduplicate Number Lines

Related Tools

case-convertcountdiffgrepheadlowercasenormalize-quotesnumber-linesprefixreverse-linesshuffle-linesslugifysort-linessplitsqueezestrip-ai-slopstrip-invisiblesuffixtailtransliteratetrim-linesuppercasewrap
← Back to pipr.tools shell