↑↑ Home ↑ UNIX  

Indispensable UNIX utilities

Files, pipes and processes -- Plain text processing -- Binary files -- Arithmetic -- Networking and WWW

A list of utilities which might have saved me some work if I had known of them sooner. Also but not primarily contains the standards "everybody" knows. I added some brief explanations, and the place to download them from if your distro does not contain them. Sometimes there are several which do the same thing — usually developed independently. Those marked "shameless plug" are my own.

Files, pipes and processes

File basics

cat
Output file(s)
split
Split file into smaller files
packcd
Group files into sets of given maximum size
file
Determine file type / format
find
Search the file system

Multiple file operations

mmv
Multiple file copy, move, append or link
rename
Multiple file rename by Perl expression
multi
Run command multiple times on different file (or non-file) arguments

Directories

pushd, popd, dirs
bash commands for changing and saving directory locations on a stack
mkcd
Create a directory and descend (shameless plug)
rmcd
Ascend and remove empty directory (shameless plug)
pwdx
Working directory of a process

Pipes and redirection

tee
Copy input to output and to file
pee
Copy input to output and to another command's input
reservoir, sponge
Buffer input, then write to file; allows in-place automated editing of files
vipe
Edit data piped between commands
mispipe
Make pipe return exit status of first command
ifne
Run command on non-empty input

Processes

ps
List running processes
kill
Send signal to process, by process ID
pgrep
Obtain process IDs from command name
pkill
Send signal to process, by command name
skill
Send signal to process, by various criteria
watch
Run a process periodically and redisplay output
nice
Run a command with a given priority
renice, snice
Change the priority of a process
nohup
Run a command, ignoring signals
sudo
Run a command with the rights of a different user
taskset
Set the CPU affinity of a process

Commands

script
Record shell session
yes
Output "y" or other string repeatedly; useful for piping into interactive commands
zrun
Run command on decompressed copy of file
lckdo
Prevent concurrent execution
xargs
Run command on every argument from stdin, possibly in parallel
multi
Run command multiple times on argument and transformed argument
para
Run multiple instances of a command in parallel on different files/arguments (shameless plug)

Plain text processing

Line-based

tac
Reverse line order
head
Output initial lines only
tail
Output trailing lines only, or lines newly written
since
Output lines newly written (and keep track of status between calls)
csplit
Split text file on lines matching pattern
dog
Print file or URL; allows some editing such as line numbering, case conversion etc.
sort
Sort lines
comm
Print lines contained or not contained in two sorted files
uniq
Print or remove sequentially repeated lines
combine
Print lines which are in both/either/one of two files, which need not be sorted
nl
Number lines
ts
Prepend lines with time and date
sed
Stream editor; programmable automated line editing
perl -n -e 'expression'
Filter text line by line with Perl code
iconv
Convert between character encodings (part of GNU libc project)

Column-based

column
Fill words into columns the way ls does
colrm
Remove columns
cut
Remove columns, by word, character of byte
paste
Concatenate corresponding lines from two files
join
Concatenate lines of sorted files when starting with same word
awk
Line-by-line (by default) automated text processing

Paragraph formatting

fold
Line breaking
fmt
Line breaking and refilling
par
More powerful alternative to fmt; supports customised line prefixes, line suffixes and character types.

Searching

grep
Line filtering based on regular expressions
numgrep
Line filtering based on ranges of numbers or divisibility
bool
Word search with boolean operators and proximity

Comparison

diff
List line-by-line differences between files
patch
Apply differences found by diff to other file
wdiff
Find word-by-word differences between files
diff3
List line-by-line differences between three files
gendiff
Recursively diff files of a directory tree against originals

Program code

expand, unexpand
Expand and unexpand tab characters
unifdef
Expand C preprocessor #ifdef's

Binary files

Output

strings
Extract ASCII strings from binary data
hexdump, od
Output hexadecimal listing of binary files
dog
Among others, allows to hexdump a URL
xxd
Convert binary files to hexdump or vice versa; allows to do editing with sed or other text filters in between (comes with vim)

Editing

hexedit
Hexadecimal editor running in text terminal; supports bit mask search with this patch
bbe
Block-based binary editor
bvi
vi-like binary editor
xxd
Convert binary files to hexdump or vice versa; allows to do editing with sed or other text filters in between (comes with vim)
bined
Shameless plug: a Perl-based automated hexadecimal editor

Comparison

cmp
Find byte-by-byte difference between binary files
xdelta
Find, print or patch differences between binary files
diffn, duff
Compare a set of files to find duplicates

Binary executables

ldd
Print dynamic library dependencies
nm
List symbols
strip
Discard symbols
size
List section sizes
readelf, objdump
Print various information about object files

Arithmetic

seq, range
Print an arithmetic progression (an equidistant sequence of numbers)
factor
Print the prime factorisation of a number
bc
Interactive or batch calculator
dc
Reverse polish calculator
$RANDOM
Magic bash variable returning random numbers
/dev/random
Source of binary random numbers
units
Command-line unit conversion

Networking and WWW

Networking

ifdata
Machine-readable output of network interface properties
ping
Check if host is reachable by sending echo requests
nc, netcat
"Network cat", connect your terminal to network socket
cryptcat
netcat with twofish encryption
ncat
netcat clone with additional features, such as proxy support, session log, authentication etc.
socat
"Socket cat", more powerful than netcat, relay between two endpoints, which may be sockets, your terminal, files, other programs etc.
nmap
Port scanner and vulnerability checker

WWW

httping
Check connection to web server
curl
Download and output a set of URLs, supports glob-like syntax, and also allows HEAD (header) requests
wget
Download web pages recursively
wput
FTP uploader
lynx
Web browser which runs in text terminal; can also output formatted page
w3m
Web browser which runs in text terminal; can also output formatted page and supports tables
links
Browser both for text terminal and X (where it supports images); allows colour calibration
dog
Download URL and optionally number lines, hexdump, extract links, print line range etc.
xmllint --shell
Structure browser for XML and HTML files