courses

Introduction

OK, so there is a lot here, if you do everything asked of you. It is not intended that you go through this all in a single sitting. I’d suggest not doing more than a section or two at a time. The main point of this is to ensure we all are on the same baseline. This page describes tools that will be necessary or will make your life better when using Linux. It isn’t intended that this all be new, or all be review. It is intended to be a repository of useful information when dealing with Linux. Some tools have their own pages and are not here. Some information is on the Software Configuration page.

So, work your way through this page at your own pace over the next week or two. It is intended to be interactive – see a tool, look at the manual, try it out. Don’t just passively read your way through this, as that won’t be very helpful. Some tools I explain in detail and some are left as an exercise for the reader. If you run into something confusing, ask the question, and we’ll try to explain.

Some of you have used Linux before and are comfortable on the command line. Some of you may not be. This page is intended as both a refresher and an introduction to some tools that you may find very useful for this class. If you have taken other classes from me, you’ve likely seen this page before.

The idea here is that this isn’t just a list of commands or functionality. It is intended to be interactive, and you should try many of the listed commands on an actual Linux install. There are some Mac-specific flags to certain tools, due to its BSD heritage. If you find something that doesn’t work for you, and you’re on a Mac, please just look at the man page for the tool on your Mac!

Shell Basics

The shell is the basic unit of interaction on a Linux machine – at least remotely or without a GUI. Even if you do have access to a Linux GUI, you will be self-restricting if you don’t know how to use the command line interface.

You will see in my examples that I make use of the zsh shell. zsh is the modern default (or soon to be default) shell of Ubuntu, macOS, and several other Linux distributions. There are many advantages of zsh over older shells such as bash or tcsh, but ultimately you need to decide what shell you like, and roll with that.

For me, zsh brings the right combination of functionality, security, and community activity. If you like what you see of my shell look and feel, you can make use of oh-my-zsh (see below on startup script) with the jonathan theme. I also make use of antigen to manage zsh plugins. tmux also has a series of plugins I’d recommend (see Software Configuration for details).

There are three books worth downloading:

These books cover in great detail all the tools you are likely to need! What you see below is a very brief overview of common tools.

Startup Scripts

What are often called RC files or startup scripts are used to configure most tools in a Linux environment. These are plain text files which enable user-specified options, and usually are of the form .toolrc or .tool.rc or .config/tool.rc – hence the name RC files.

One of the most important is .$(basename $SHELL)rc. This controls your shell – options, look and feel, any extensions, whether ls uses colors, etc. The file is written in the shell syntax of your chosen shell. If you’re already run the script from the Software page, and then decide to change your shell, you’ll want to re-run the script.

If you do decide to take the zsh route, I strongly encourage you to make use of oh my zsh. It provides some significant improvements over the default settings in zsh.

The ~ character is a shortcut for a home directory. Your home directory is just plain ~. Kevin’s is ~dmcgrath

Nearly every other tool has some sort of configuration script, usually somewhere in your home directory. Take a look around, see what you find. Some specific directories to investigate:

Do keep in mind that dot files are by default hidden from ls. Look at the man page for ls to see that command line option will allow you to see all the files in a directory, including the hidden ones.

Questions

File/Directory Manipulation

One of the most basic sets of functionality in a shell is moving around and manipulating files in some way. A list of useful commands for this can be seen below. For further details, take a look at the man page for each of them.

Unix commands take a combination of short form and long form options. Short form options are usually of the form -x, where x can be any single letter. Long form options are of the form –long-option. Many commands have long and short options for the same thing, but you’ll need to investigate the commands man page for more details.

Viewing File Contents

After moving around and make directories and files, one of the most common things you’ll do on a Linux system is look at files. There are a variety of ways to do that, depending on your needs.

Questions

readelf, objdump, and nm are all part of the GNU binutils collection of tools for working with binaries. Which you would use in a given situation entirely depends on what you are trying to do. There is some overlap in output, but each brings unique strengths to the table.

man

man is the command which displays online manuals. This does not mean on the web – online in this case means on the current computer in use. I would suggest taking a look at the manual for man: man man

Reading man pages is a skill like any other. At first, it will seem difficult. As you practice more, you’ll start to recognize the way they are structured, gain an understanding of what’s in each section of a given man page, and how to make use of them to be more productive.

An example man page (man 3 fopen) with most of the DESCRIPTION section removed for brevity:

FOPEN(3)                                                          Linux Programmer's Manual                                                         FOPEN(3)

NAME
       fopen, fdopen, freopen - stream open functions

SYNOPSIS
       #include <stdio.h>

       FILE *fopen(const char *pathname, const char *mode);

       FILE *fdopen(int fd, const char *mode);

       FILE *freopen(const char *pathname, const char *mode, FILE *stream);

   Feature Test Macro Requirements for glibc (see feature_test_macros(7)):

       fdopen(): _POSIX_C_SOURCE

DESCRIPTION
       The fopen() function opens the file whose name is the string pointed to by pathname and associates a stream with it.
        .
        .
        .
       
RETURN VALUE
       Upon  successful  completion fopen(), fdopen() and freopen() return a FILE pointer.  Otherwise, NULL is returned and errno is set to indicate the er‐
       ror.

ERRORS
       EINVAL The mode provided to fopen(), fdopen(), or freopen() was invalid.

       The fopen(), fdopen() and freopen() functions may also fail and set errno for any of the errors specified for the routine malloc(3).

       The fopen() function may also fail and set errno for any of the errors specified for the routine open(2).

       The fdopen() function may also fail and set errno for any of the errors specified for the routine fcntl(2).

       The freopen() function may also fail and set errno for any of the errors specified for the routines open(2), fclose(3), and fflush(3).

ATTRIBUTES
       For an explanation of the terms used in this section, see attributes(7).
       ┌────────────────────────────────────────────────┬───────────────┬─────────┐
       │ Interface                                      │ Attribute     │ Value   │
       ├────────────────────────────────────────────────┼───────────────┼─────────┤
       │ fopen(), fdopen(), freopen()                   │ Thread safety │ MT-Safe │
       └────────────────────────────────────────────────┴───────────────┴─────────┘

CONFORMING TO
       fopen(), freopen(): POSIX.1-2001, POSIX.1-2008, C89, C99.

       fdopen(): POSIX.1-2001, POSIX.1-2008.

SEE ALSO
       open(2), fclose(3), fileno(3), fmemopen(3), fopencookie(3), open_memstream(3)

COLOPHON
       This  page  is  part  of release 5.05 of the Linux man-pages project.  A description of the project, information about reporting bugs, and the latest
       version of this page, can be found at https://www.kernel.org/doc/man-pages/.

GNU                                                                      2019-05-09                                                                 FOPEN(3)

Some important things to point out here:

tar

tar is one of those tools that everyone loves to hate. It has an enormous collection of flags and switches, and in reality is being shoe-horned into its current role as “master compression utility.” Historically it was created to make tape backups of a filesystem. Since compression can only work on a single file, it is common to first combine all necessary files into a single meta-file: a tar ball. This tar ball is then compressed using the compression algorithm of your choice.

There are a few common ways to use tar:

All of the above assume bzip compression. There are multiple other variants:

While there are a large number of options, the most common are bzip2, gzip, and xz.

Creating an archive is done with -c, extraction with -x. Archive name is specified with -f. -v requests verbose output.

There are a lot of places you will need to know how to work with tar, so learning the basics of it is quite important.

Lots more information can be found in the man pages.

grep

grep is a tool which allows for regular expression matching of contents of a line or file. It is commonly known as a filter – it removes unwanted cruft and only allows the display or further processing of data that matches (or doesn’t match, in the case of an inverted search).

grep has multiple modes, which can be invoked with an alternative name or a specific flag. For instance, to use extended regular expression syntax, you can invoke as egrep or call as grep -E. Both operate identically. The different modes are

grep is a tool you will likely make a ton of use of. That will require both a knowledge of regular expressions and an understanding of how grep works and is controlled by its flags. See the man page for grep for the latter, and you’re on your own for anything beyond very basic regular expressions. I’m a firm believer that if you try to solve a problem with regular expressions, you now have many problems, rather than just the one.

find

Ah, find. The true Swiss Army Knife of Linux. find is a great example of a simple tool that got way out of hand, yet still maintains a significant utility. While the feature creep is real, the features are just so dang useful!

find requires a path and a test (and sometimes an action). The path is just the root of the subtree in the filesystem you want to search, while the test can be…almost anything you can imagine.

Then, once you’ve written a test, what should you do with the files that match the test? The options are legion:

While that’s a relatively short list, I would direct your attention to the fact that find will execute a given command on every file that matches the pattern. Any command. Of your choosing. This could be as simple as a rename, or an extraction of $n$ lines of text. Or it could be as complex as a multi-stage pipeline that vastly transforms the input files into something altogether different. The choice is yours.

See the man page for all the gory details of the above.

asciinema

asciinema is a tool for creating casts of a terminal session. Specifically, it can be used to record and play back a series of shell commands, in an interactive fashion, allowing the user to copy the commands. You’ll see this littered throughout this course, in places where a simple interactive session will provide significant instructional value.

Think of it as a way to record the steps and output of commands that you will run. For instance, a testing session or proof of functionality that is requested. Very useful tool.

By default, it uploads to the asciinema servers. But if you provide a file name, instead of uploading, it saves it to the named file.

┌─(~)────────────────────────────────────────────────────────────(dmcgrath@DESKTOP-2JUKDNG:pts/2)─┐
└─(17:50:35)──> asciinema rec -t "Introduction to man pages" -i 2 --overwrite man.cast           ─┘

By default, it will not overwrite an existing file. As you can see above, I provide to --overwrite flag to have it ignore existing files. See the man page for details on the other options seen in the example invocation above.

Other Useful Tools

This list of utilities was generated by mining my own history file. I’ve left out anything that was very domain specific (reverse engineering, security, network), but they are mostly in reverse count order (most used at the top). I’m leaving them as a collection of tools you’ll want to look up in the man pages. Most can also be investigated with tool --help, but that is much more terse than the man pages.

Tools to pay specific attention to include make, curl, which, ps, and awk. So basically the top 5 most used in my history.

I will leave you with one of my most used commands on servers I manage:

$ ps -efH --no-header | awk '{print $1}' | grep -Ev $(python3 -c 'import sys; print("|".join(sys.argv[1:]))' $(cut -f1 -d':' /etc/passwd)) | sort | uniq -c | sort -n

No, I don’t type that every time. I have it as an alias in my startup script, since it’s how I print out a list of users who are causing excessive load on the server. Play with it, see if you can improve on it. You can also see the use of immediate python combined with the use of awk, cut, and grep.

Questions to Consider

What do you think the grep portion does? Why is that useful? What is an alias? Can you replace the awk pipeline component with a cut? What about the reverse? Take a look at a tool called paste. How might you use it to replace the use of python in this pipeline?