Shell Fundamentals

If you have ever spent more than a few minutes working on the Linux command line, you know that simple commands (such as ls, cat, or grep) are incredibly powerful. Yet, taken alone, these tools can feel limiting. Imagine needing to extract a list of all the newest logfiles, sort them by size, filter out permission errors, and save the final result to a new file for analysis. Tackling an operation like this manually would be laborious and prone to error.

Fortunately, Linux's architecture was designed to solve this exact problem. Instead of relying on a single, monolithic tool, the Unix philosophy teaches us to combine small, specialized commands to create a complex workflow. Pipes and output redirection are the core components of this philosophy. Mastering these two concepts will not only transform how you interact with the operating system but will allow you to evolve from a simple shell user into a true data flow engineer. In the following sections, I will demonstrate how to route data between programs and files, unlocking the genuine power of your terminal.

The Three Standard Streams

To truly master the flow of data, you must first understand the fundamental concept that every running program in Linux (called a process) uses three established channels known as standard streams. These streams are the designated pathways for input and output, and, crucially, in the Linux philosophy, they are all treated as special types of files. Each stream is assigned a unique number, called a file descriptor, which is essential when you want to specifically redirect one stream and not the others.

[...]