How to print unique lines with perl one-liner?

How to Print Unique Lines with a Perl One-Liner

Printing unique lines from input is a classic text processing task. Perl, with its flexible syntax and built-in data structures, makes this easy. The standard approach uses a hash to track seen lines and a simple condition to print only the first occurrence.

While the familiar one-liner is typically run directly from the shell (e.g., perl -ne 'print if !$seen{$_}++'), running this code within a Perl sandbox or from an inline Perl script requires a slightly different form. This is because command-line switches like -n are not valid Perl script code but command-line options.

Core Concepts

• $_: Default variable holding the current input line.
• %seen: A hash to record lines already printed.
• Checking if (!$seen{$_}++) lets you print the line only the first time it’s encountered.
• In a script, you loop over input lines using while (<>){...} instead of relying on -n.

Example: Unique Lines in a Self-Contained Script

This example reads from standard input, prints only unique lines, and runs successfully as a standalone Perl script. You can test this by piping input, e.g., echo -e "a\na\nb\nc\nb" | perl unique.pl or by running as perl - and typing lines manually.

use strict;
use warnings;

my %seen;

while (my $line = ) {
    print $line if !$seen{$line}++;
}

Explanation

• while (my $line = <STDIN>) reads lines one by one from standard input.
• %seen stores each line as a key; the post-increment $seen{$line}++ increases the count.
• Using if !$seen{$line}++ ensures that print is executed only the first time the line appears.
• The hash approach preserves the original input order of unique lines.

Common Pitfalls

• Remember to include the newline when using $line. The trailing newline keeps lines distinct.
• Using -n or -p works only in command-line one-liners, not inside script source code.
• Without strict and warnings, subtle bugs can creep in, so always enable them.
• Mind that this approach keeps all unique lines in memory; for very large inputs, this could be a concern.

Summary

Printing unique lines in Perl boils down to hashing lines seen so far. From the command line, you use the -n switch, but inside a Perl script or sandbox, explicitly write the input loop. The above example is portable, easy to understand, and works efficiently in all modern Perl versions.