Other Useful Unix Commands

Intro

This is the fifth in a series of videos on how to use Unix and Linux for software developers.

This section goes over several commonly used commands.

The cat, head, tail, and less commands are used for printing file content to the terminal.
Grep is used for searching for words in one or more files.
Pipes are used for running two or more commands sequentially, using the output of the first command as the input for the next command.
There are a number of system management related commands including disk usage, terminal usage, and processes commands.

It is recommended to open your system's shell program and follow along with the topics covered.

Read it, watch it, do it, review it:

There is both a written and accompanying video version of this tutorial. There are timestamps under the headings that align with the video version of the topic.

Read it: For each topic heading, First read the topic.
Watch it: Then watch the associated section of the video.
Do it: Then follow the instructions to replicate the steps in your Unix shell program.

The Unix CheatSheet has a major category that aligns with this tutorial.

Review it: When you are done with the whole tutorial, open the CheatSheet and review the Commands category. Make sure you understand everything, and can refer back to the CheatSheet when you are working on your own projects in the future.

Commands Overview

[0:00 Video timestamp]

Commands are mini programs. Most commands have their own executable program file in a system bin folder. Bin folders hold binary executable files.
The which command will return the given program's executable file path.

which mkdir returns /bin/mkdir
which cat returns /bin/cat
which less returns /usr/bin/less

Print file content to the Terminal

You can print all or part of a file to the Terminal with the cat, head, tail, and less commands. Each does different things and we'll go through them one by one.

Cat

[0:47 Video timestamp]

The cat command can be used to concatenate or print files.

First let's use the cat command to create and populate a file:

Cd into your Desktop directory: cd ~/Desktop

Lets create a file called breeds.txt and populate it with some cat breeds
Enter the cat command followed by > and the file name
cat > breeds.txt
You will be prompted to enter some lines. You can use these:

calico
persian
siamese

Press Control+D to finish. This will create the breeds.txt file with the lines entered.

Now use the cat command followed by the filename to print the file content to the Terminal:
cat breeds.txt
You should see the three lines we entered.

Let's create a new file called dogs.txt and populate it with dog breeds. We'll use the cat command again but with a different syntax. This syntax is called a Here Document. We populate the file line by line again, but this time we specify a word to indicate that the file input ends here. We will use "EOF" for end of file, to indicate that we are done creating the file.

cat << 'EOF' > dogs.txt
When prompted to enter data, you can enter the below dog breeds, ending the input with EOF.

dobermann
poodle
beagle
EOF

Let's view the file content. Use the -n option to print the file with line numbers.
cat -n dogs.txt
You should see the three lines we entered and the lines are numbered 1, 2, 3.

You can use the cat command to append more lines to an existing file.
cat >> dogs.txt
Takes you to a screen to enter content.

Great Dane
Boxer

Press Control+D to finish. This will add the new lines to the end of the dogs.txt file.

View it again and you should see the additional lines:
cat dogs.txt

You can use the cat command to concatenate two files, one after the other:
cat breeds.txt dogs.txt

This will print the breeds file then the dogs to the Terminal.

To save the concatenated output to a new file, add > and a new file name:
cat breeds.txt dogs.txt > cats-and-dogs.txt

Then print it to the Terminal:
cat cats-and-dogs.txt

Head / Tail

[4:24 Video timestamp]

Cat will print the entire file. To only print the top or bottom of the file use the head or tail command with the file name.
It will print 10 lines by default. To print a specified number of lines use the -n option, with n being the number of lines to print.

head -4 cats-and-dogs.txt
This will print the first four lines.

tail -4 cats-and-dogs.txt
This will print the last four lines.

The tail command is useful for viewing log files.
Log files can get huge so restricting it to the last 100 lines or so makes it much easier to sift through.

Less

[5:55 Video timestamp]

So far we have been printing small files. What if you want to print a large file to the terminal and navigate through it.

Unix has a paging program for that called Less.

Execute it with the less command:
less <filename>

To number the lines, add the -N option:
less -N <filename>

To quit the Less program, enter q

Internally Less is used with the man command to print and navigate through the manual.
The Git source control program (covered in a separate tutorial series) also uses Less when printing files to the terminal.

The Less program has its own key commands for navigating through the file and for searching for keywords or phrases.

You can test it out on one of your large files. Or since the man program uses Less to print the Manual document, you can open the manual to see the documentation for the Less command, and navigate through it using Less navigation key commands:
man less

Navigation key commands

To navigate through the page use the following commands:
One line:

downArrow or return will go down one line.
upArrow goes up one line.

Half screen:

d - Pressing d will take you down half a screen.
u - Pressing u will take you up half a screen.

One screen:

f or space - Pressing f or the space bar will take you forward (i.e., down) one full screen.
b - Pressing b will take you back (up) one full screen.

First/Last line:

g - By pressing g you will go to the first line (the top) of the document.
G - By pressing capital G you will go to the last line (the bottom) of the document.

Search the document

To search the document for a term or phrase enter a forward slash followed by the search term:

/< search_term>
n - Press n to go to the next occurrence of the search term.
N - Press capital N to go to the previous occurrence of the search term.

q - To quit the Less program press q.

Grep - Search for a term in a file or directory

[8:18 Video timestamp]

Grep stands for global regular expression print.
Grep is a utility command that allows you to search for text patterns in one or more files.

Create a file called find-waldo.txt and populate it with Lorem Ipsum filler text and the word waldo in three places as waldo, waldoworld and Waldo.

Run the below cat command:
cat << 'EOF' > find-waldo.txt

Then paste in the following text:

Lorem ipsum dolor waldo sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. 
Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo waldoworld consequat. 
Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. 
Waldo Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

EOF

Then you can use grep to find the occurrences of "waldo" in the file.

grep "waldo" find-waldo.txt Returns the lines that contain the search term "waldo". It is case sensitive so there are two lines with matches.

Let's add some options:
grep -iwn --color "waldo" find-waldo.txt Returns the lines that contain the search term "waldo".
We added the following options:

--color: Color code the matches.
-w: The -w option only searches for "waldo" as a complete word, so waldoworld doesn't qualify.
-i: The -i option makes it case insensitive so Waldo and waldo will both be highlighted.
-n: Puts the line number before each match printed.

grep -ic "waldo" find-waldo.txt Returns the number 3, meaning there are 3 lines that match the search term.
We added the following options:

-i: Makes it case insensitive.
-c: Returns the number of occurrences of the search term.

To search a directory for files that contain the search term, use the l and r options and put the directory as the argument:
grep -lr "waldo" ~/Documents Returns the file names in the Documents directory that contain the search term.
/Users/my-username/Documents/find-waldo.txt
To get that we had to add the following options:

-l: List filenames that contain the pattern
-r: Search recursively in all files in a directory.

Standard I/O and Pipes

[11:53 Video timestamp]

Unix uses standard Input/Output. It works as follows:

Standard Input: Commands are generally typed with the keyboard into the terminal.
Standard output: The input is processed by the Unix program. Then the result is the standard output. The standard output is generally printed to the terminal screen.
Standard error: If there is an error, the standard error message is generally printed to your terminal screen.

Example:
cat dogs.txt this command is the standard input.
Unix process this command by getting the content of the dogs.txt file. The result is the standard output.
The file content is printed to the Terminal.

Sometimes you may need to perform more than one command on a set of data.
For instance if you want to sort the content of the dogs.txt file before printing it. It contains a list of dog breeds.
The Unix sort program sorts a list alphabetically.
Also you can apply the Uniq program to remove any duplicate items.

You can execute each of these programs separated by the Pipe symbol, a vertical line.
cat dogs.txt | sort | uniq

It works like an assembly line where each command takes the output from the previous command as its input, processes it and outputs it for the next command to use as input.

The cat command:

The standard is the dogs.txt file.
Process: It gets the data from the file.
The standard output is the dog breed list from the file.

The sort program:

The standard input is the dog breed list.
Process: It sorts the list.
The standard output is the sorted list of dog breeds.

The uniq program:

The standard input is the sorted list of dog breeds.
Process: It looks for duplicates and removes any found.
The standard output is the sorted no-duplicate dog breed list.

The final output is printed to the terminal.

The list is printed in alphabetical order:

Beagle
Boxer
Dobermann
Great Dane
Poodle

System-related commands

The rest of the commands we'll talk about are system management related commands.
We'll start with Disk free and disk usage.

Disk Usage:

[13:51 Video timestamp]

df -h Is the disk free command with the human readable option.
It returns the space used and the remaining available space on your disk, which Unix calls volumes.
The -h option makes it human readable by appending Gi for gigabyte to the size number.
The avail result is how much space remains available

du -h -d0 <path> Returns the space this directory and its contents uses.
the -d depth option followed by a number is how many levels you want to get individual results for. -d0 means just get the results for the given directory.
To see how much space the Documents directory takes up run.
du -h -d0 ~/Documents

To see how much space each Documents sub-directory takes up, change the -d0 option to -d1 to go one level down.
du -h -d1 ~/Documents

Get users and terminals open:

[15:22 Video timestamp]

The next set of commands relates to users and terminals open.

users The users command will show all logged in usernames.
whoami Returns the username of the logged in user.
who The who command will show all the open terminals. Console is the terminal application itself. Then terminal names starting with tty are individual Unix shell windows open.
w The w command is like who, but it also tells you what process the shell is currently running. In this case it's running the w command.

Processes

[16:02 Video timestamp]

A process is an executing program.
Each command launches a program.
The kernel allocates memory and resources for the program to run.
Each process has an id called pid, a command name, and an owner.

ps The ps command shows all running processes owned by the user
ps au To show the processes owned by all users, run the ps command with the au argument without a dash. This includes processes that are user initiated but owned by the root user.

ps aux To show all processes both user and system owed run the ps command with the aux argument. This will include a lot of system programs that run in the background.
top The top command will return the processes with the top usage of whatever option you provide.

top -o cpu will list the processes with the most usage of the CPU. It refreshes every few seconds. It lets you know what programs are hogging your CPU.
top -o mem Sorts the processes by memory usage.

kill -9 <pid> Kill the process with the given process id.

Some processes end on their own once a program finishes executing. Other processes stay open as long as a program is open. Others keep running in the background.
If you happen to know a particular process is hogging a lot of CPU or memory and it is not needed, you can kill that process with the kill command followed by -9 and the pid process id number.

Other Useful Unix Commands (Video Tutorial)