Hello! Hope you’re doing great. In my last post, I have written about commands in linux CLI. In this post we will understand how to play with any command’s output, store it in files or even connect multiple commands together into command pipelines. Let’s go!

Topics covered in this post:

1. What are Standard Input, Output and Error in Linux?
2. File Descriptors
3. Redirecting stdout
4. Redirecting stderr
5. cat command
6. Command pipelines

1. What is Standard Input, Output and Error in Linux?

Let's say we use a command like ls in the linux cli. ls will list all the files and directories in a given path and if the path is non-existent or incorrect path it will throw an error. This is shown below:

Now, every linux command like ls is designed to produce the following:

1. Command output
2. Status messages and error messages

The output and the error messages are displayed on the screen and we should know that everything is a file in linux. Which means that a command will (not necessarily) send its output to a special file called Standard Output(stdout) and error to Standard Error(stderr).

By default, both stdout and stdout are linked to the screen and not saved into a disk file. Also, many commands or programs take input from Standard Input (stdin) which is by default attached to the keyboard.

So stdout, stderr and stdin are files that are attached to screen and keyboard respectively. When a command is used in the cli, these files receive the output, errors and input.

Source: linuxhint.com/redirect-stderr-stdout-bash

2. File Descriptors

A file descriptor is a unique number that identifies an open file in an operating system. It has a record of all the files opened and their locations stored in a global table along with the permissions.

Source: What is a File Descriptor? (computerhope.com)

On a Unix-like operating system, the first three file descriptors, by default, are STDIN (standard input), STDOUT (standard output), and STDERR (standard error).

The file descriptors are used to input/output(I/O) redirection in the linux cli. Let's discuss this in the next one.

3. Redirecting stdout

I/O redirection basically means that we can define where the output ends up finally with the redirection operator >.

To redirect the stdout to a file, we will use <command> > <output_file_name>. Let's take a look at this with an example:

Here we are using the command ls -l > ls_op.txt to long list the files and write the output to ls_op.txt file. cat command will display the output on the screen

What happens if we provide a non-existing path to the ls command?

Here, /bn/usr/ path doesn’t exist and we anticipated an error message written to the ls_op.txt file. Instead, the error is shown on the screen. How about the output file?

ls_op.txt file is empty and the error is displayed on the screen. What happened here?

First, we received an error message for the non-existing path. The error will be sent to stderr instead of stdout. Next, the redirection operator (>) will overwrite the data on the output file. Since we didn’t receive any output, the file was overwritten with nothing and the error was redirected to screen.

Intrestingly, we can use > to create a new file or even truncate an existing file.

> <filename> will create a new file and if this file exists, its contents will be truncated.

What if we want to just append the output to an existing file? >> will append the data to the existing file and create a new file if it is not present.

4. Redirecting stderr

Redirecting the stderr must need its file descriptor. As discussed above, the file descriptor for the stderr is “2” and this will be used along with redirection operator for stderr.

The stderr can be redirected to a file as mentioned below:

ls -l /bn/usr 2> ls-err.txt. Here 2 is the file descriptor for stderr followed by the redirection operator. Let’s see the output

• Redirecting both stdout and stderr to single file

  In most scenarios we may want to capture both the output and error to a single file. Suppose, we have a job that is scheduled to run every day and write its output to a file. Writing both the stdout and stderr will help us identify the issues whenever the job fails

  ls -l /bin/usr > ls-output.txt 2>&1. This command helps us do that and it has two parts:

  • ls -l /bin/usr > ls-output.txt- This is normal stdout redirection and the stdout is redirected to ls-output.txt file.

  • 2>&1- the stderr (2) is again redirected to stdout(1). This should happen in the same order else it will not work.

    

4. cat command

cat, which is short for concatenate, can perform multiple operations like:

• Display file contents
• Concate multiple files into a new file using redirection operator
• Create a new file using cat command and redirection operator

• Display file contents

cat <filename> will display the contents of the given file. If multiple files were to be displayed, then add the option -n and pass the file names one after the other.

• Concate multiple files into a new file using redirection operator

What cat commad essentially did in the above step is that it read the file contents and passed it to stdout file which is attached to the screen. We can use the I/O redirection technique to redirect the output of cat command to a file. Let’s see this in action.

cat testfile > catfile command put the “testfile” contents to “catfile” using > operator. Also, the “catfile” is created by the command on the go.

This can be easily applied in a scenario where we want to write multiple file contents to a single file. cat <input1> <input2> > <ouputfile>

• Create a new file using cat command and redirection operator

What happens when we don’t pass a file name for the cat command? cat command expects a file name and when not provided, it will read from the stdin. This is provided in the manual (man cat)

Here, the cat command will continuously accept the input from keyboard and displays it back on the screen as the stdout is still attached to the screen. By pressing ctrl+d or cmd+d keys will exit the prompt.

What if we redirect the above example’s output to an output file? The command for this scenario would be cat > <filename>.

The input will be accepted continuously from the keyboard and output will be written to the file. Again, by pressing ctrl+d or cmd+d keys will exit the prompt.

6. Command Pipelines

Till now, we have used single commands in the cli and played with the output from those commands (>). Piplines in the shell help utilize the stdout of one command to be piped into stdin of other command with the help of pipe (|) operator.

ls -l /usr/bin | less. ls -l command will list all the contents in the path /usr/bin. Later the output of this command will be passed to less command. less command will display the contents page (one screen) by page.

Filters

• ls -l /usr/bin | sort- This command will provide the sorted lists
• ls -l /usr/bin | sort | uniq | less- This command will provide the sorted, unique list in pages. The uniq command is often used in conjunction with sort command. uniq command accepts either stdin or a single filename argument.

• ls -l | sort | uniq | grep zip- This command will search for a pattern in the sorted, unique list using grep command. grep command accepts a pattern within a file or a sdtin. A “pattern” here means a word or a regex pattern.

  

• ls /usr/bin | tee ls.txt | grep zip- The tee command reads the stdin from the previous command and copies it to the stdout and also to one or more files. This allows us to capture the output from the intermediate steps in the process

Conclusion

The I/O redirection is a powerful technique and pairing with the pipelines will help us build a quick script or first versions of data pipeline. Commands like grep will help trigger other commands based on the seach results or even help understand the output.

Resources

1. Linux Command Line Books by William Shotts
2. What is a File Descriptor? (computerhope.com)
3. Cat Command in Linux {15 Commands with Examples} | phoenixNAP KB
4. less command in Linux with Examples - GeeksforGeeks
5. How to Redirect stderr to stdout in Bash (linuxhint.com)