# Redirecting output Bash provides a wide range of features for manipulating and redirecting output. Here we'll cover how to redirect output in Bash, including standard output (`stdout`), standard error (`stderr`), and how to pipe data between tools. ## Input/Output streams In command-line tools, `stdin`, `stdout`, and `stderr` are standard streams that are typically used to handle input, output, and error messages, respectively. `stdin` is a standard input stream that is used to read data from the user or from another program: it is the input that a program receives. For example, when a program asks the user to enter data, the data is read from `stdin`. `stdout` is a standard output stream that is used to display data to the user or to another program: it is the output that a program sends to the user or to another program. For example, when a program displays the result of a calculation, the result is sent to `stdout`. `stderr` is a standard error stream that is used to display error messages or diagnostic messages to the user or to another program: it is the stream that a program uses to report errors, warnings, or other diagnostic information. For example, when a program encounters an error or a warning, the message is sent to `stderr`. In command-line tools, these streams are usually represented by file descriptors: `stdin` is represented by file descriptor 0, `stdout` is represented by file descriptor 1, and `stderr` is represented by file descriptor 2. By default, the output of a command is sent to `stdout`, while error messages are sent to `stderr`. ## Standard Output (stdout) redirection Standard output is the default output channel for command-line programs, where the output is displayed on the terminal. However, sometimes it is desirable to redirect the output to a file instead. This can be done using the greater-than symbol (`>`) followed by the name of the file to which the output should be redirected. For example, to redirect the output of a command to a file, use: ```sh ls > filelist.txt ``` This will redirect the output of `ls` to `filelist.txt` instead of displaying it on the terminal. This will *replace* the content of the target file! It is also possible to append output to an existing file by using the double greater-than symbol (`>>`). For example: ```bash ls >> filelist.txt ``` This will append the output of `ls` to the end of `filelist.txt`. The output of a program can be suppressed and discarded entirely by redirecting it to the special file `/dev/null`. The null file accepts any amount of data, and immediately discards it. ```bash ls > /dev/null ``` ## Standard Error (stderr) redirection Standard error is the default channel for error messages and diagnostics, which are displayed on the terminal. However, it is also possible to redirect standard error to a file. To redirect standard error, use the number 2 followed by the greater-than symbol (`>`). The number 2 represents the standard error stream. For example, to redirect the standard error of a command to a file, use: ```bash grep xyz somefile.txt 2> errors.txt ``` This will redirect any error messages generated by `grep` to `errors.txt`. This will replace the contents of `errors.txt`, and just like redirecting `stdout`, you can use the `>>` operator to append, instead: ```bash grep xyz somefile.txt 2>> errors.txt ``` Also just like how the stdout data can be suppressed by sending it to the\ `/dev/null` file, data on `stderr` can also be suppressed: ```bash grep xyz somefile.txt 2> /dev/null ``` ## Combining outputs The output of one stream can be combined with the output of another by using the `>&` option. This allows us to combine `stdout` and `stderr` for instance: ```bash grep xyz somefile.txt 2>&1 ``` {% hint style="info" %} The `stdout` stream has the id of `1` so we're sending `stderr` to `stdout` which will appear on the console. {% endhint %} Of course, the combined output can also be redirected: ```bash grep xyz somefile.txt 2>&1 > foo.txt ``` This will log both the result of grep and any errors together. ## Piping data between tools One of the most powerful features of Bash is the ability to pipe data between tools. Piping allows the output of one command to be used as the input of another command, without the need for intermediate files. To pipe output to another command, use the vertical bar (`|`) symbol. For example: ```bash ls | grep test ``` It is possible to chain multiple commands together using pipes. For example: ```sh ls | grep test | wc -l ``` This takes the output of `ls`, searches for `test`, and then uses the `wc` tool (word count) with the line option (`-l`) to count how many lines. It will print the number of files with `test` in the name! These features provide a powerful and flexible way to manipulate and process data on the command line, and can help streamline many common tasks. ## Incorporating other tools Now that we can connect output from one command to another, we can use many built-in tools for data manipulation. In the following examples, we use `echo` and `cat filename` as the source of data for demonstration purposes, but remember the beauty of piping between commands - the data can come from any other commands that output text, like `ls`, `ps`, `grep`, etc! ### The "tr" command The `tr` command in Linux is used to translate or delete characters from standard input and write to standard output. It can also be used to replace a set of characters with another set of characters. The basic syntax of the `tr` command is as follows: ```css tr [options] set1 set2 ``` where `set1` is the set of characters to be translated, and `set2` is the set of characters to translate `set1` into. If `set2` is shorter than `set1`, the extra characters in `set1` are deleted from the output. Some common options for the `tr` command include: * `-c`: complement the set of characters in `set1` * `-d`: delete characters in `set1` * `-s`: squeeze repeated occurrences of characters in `set1` into a single occurrence Here are some examples of how you can use the `tr` command in a payload: 1. Translate all lowercase letters to uppercase: ```bash echo "hello world" | tr '[:lower:]' '[:upper:]' ``` Output: `HELLO WORLD` 2. Delete all vowels from a string: ```bash echo "hello world" | tr -d 'aeiou' ``` Output: `hll wrld` 3. Replace all spaces with tabs: ```bash echo "hello world" | tr ' ' '\t' ``` Output: `hello world` 4. Complement the set of characters in a string (i.e., replace all non-alphanumeric characters with a space): ```bash echo "hello world!" | tr -c '[:alnum:]' ' ' ``` Output: `hello world` (Note the space at the end.) ### The "sed" command The `sed` command (short for "stream editor") is a powerful text-processing tool. It is used to manipulate, transform, and replace text in a file or a stream of text. It works by reading a file or stream line-by-line, making specified modifications, and then outputting the modified text. The basic syntax of the `sed` command is as follows: ```sh sed [options] 'command' filename ``` or chained with a pipe, ```bash cat filename | sed [options] 'command' ``` Here are some examples of how you can use the `sed` command in a payload: 1. Replace a string in a file: ```bash cat filename | sed -i 's/old_text/new_text/g' > newfile ``` This command will replace all occurrences of `old_text` with `new_text` in the `filename` file. The `-i` option specifies that the changes should be made in-place, meaning that the original file will be modified. The output will be sent to `newfile`. 2. Delete a line from a file: ```sh cat filename | sed -i '2d' > newfile ``` This command will delete the second line from the `filename` file. The `d` command is used to delete lines. The output will be sent to `newfile`. 3. Insert a line before or after a matching pattern: ```bash cat filename | sed -i '/pattern/i new_text' | sed -i '/pattern/a new_text' > newfile ``` These commands will insert `new_text` before or after the first occurrence of `pattern` in the `filename` file, respectively. The `i` command is used to insert text before a matching pattern, while the `a` command is used to insert text after a matching pattern. The output will be sent to `newfile`. 4. Replace a string in a file only in lines matching a pattern: ```sh cat filename | sed -i '/pattern/s/old_text/new_text/g' > newfile ``` This command will replace all occurrences of `old_text` with `new_text` only in the lines that match `pattern` in the `filename` file. The `s` command is used to substitute text, and the `/pattern/` specifies that the substitution should only be made in lines that match the pattern. The output will be sent to `newfile`. These are just a few examples of how you can use the `sed` command in a bash script. There are many more options and commands available that you can use to manipulate text in a file or stream. ### The "awk" tool `awk` is another powerful text-processing tool. It is used to process and manipulate text files that are formatted in a specific way. It can extract and print specific fields, perform calculations, and search and replace text. The basic syntax of the `awk` command is as follows: ```sh awk 'pattern { action }' filename ``` or using pipes: ```bash cat filename | awk 'pattern { action }' ``` Here are some examples of how you can use the `awk` command in a shell script: 1. Print specific fields from a file: ```sh cat filename | awk '{ print $1, $3 }' > newfile ``` This command will print the first and third fields from the `filename` file, separated by a space. The fields are separated by whitespace by default. The output will be sent to `newfile`. 2. Print lines matching a pattern: ```sh cat filename | awk '/pattern/ { print }' > newfile ``` This command will print all lines from the `filename` file that match the `pattern`. The `/pattern/` specifies the pattern to search for, and the `{ print }` specifies the action to perform on matching lines. The output will be sent to `newfile`. 3. Perform calculations: 
cat filename | awk '{ total += $1 } END { print total }' > newfile
This command will add up all the numbers in the first field of the `filename` file and print the total at the end. The `END` keyword specifies that the action should be performed after all lines have been processed. The output will be sent to `newfile`. 4. Search and replace text: ```sh cat filename | awk '{ gsub("old_text", "new_text", $0); print }' > newfile ``` This command will search for all occurrences of `old_text` in each line of the `filename` file and replace them with `new_text`. The `gsub` function is used to globally substitute text, and `$0` refers to the entire line. The output will be sent to `newfile`. These are just a few examples of how you can use the `awk` command in a shell script. There are many more functions and options available that you can use to manipulate text in a file or stream. ## Summary These tools provide a powerful and flexible way to manipulate and process data on the command line, and can help streamline many common tasks. Not every command line tool uses stdout and stderr in the expected ways, but the vast majority of them do. Being able to manipulate and redirect the data streams from tools enables chaining for complex behavior and creating entirely new tools.