Pipes, processes & your first scripts

Module: IT Foundations · Level: Beginner+ (assumes Part 1) · Time: ~25 min read + practice You'll be able to: chain commands into pipelines with pipes and redirection, search and transform text with grep/sed/awk, see and control running processes, manage your environment with variables and .bashrc, write and run your first shell script, install software from the command line, and connect to remote servers with SSH — the exact skill you'll use on every cloud machine.

1. Pipes and redirection — commands as LEGO

Part 1 gave you individual commands. Part 2 is about the idea that makes Unix Unix: small tools, each doing one job, snapped together like LEGO bricks. The stud that connects them is the pipe.

First, the plumbing. Every command has three standard streams:

• stdin — where it reads input (usually your keyboard)
• stdout — where it writes normal output (usually your screen)
• stderr — where it writes error messages (also your screen, but a separate stream)

The pipe: |

command1 | command2 connects command1's stdout to command2's stdin — data flows left to right, no temporary files needed.

ls | wc -l                 # how many files are here? (list → line count)
ps aux | grep firefox      # all processes → only the firefox lines
cat access.log | sort | uniq   # sort, then collapse duplicate lines

Each brick is simple. The pipeline is powerful. This is how engineers answer questions like "which IP hit this server most?" in one line instead of one spreadsheet afternoon.

Redirection: sending streams elsewhere

/dev/null is the black hole: anything written to it vanishes. You'll use it constantly to silence noisy commands. And burn in the > vs >> difference now — one overwrites, one appends, and mixing them up on a real file hurts.

2. The text trio: grep, sed, awk

grep — the command that searches anything

Logs, code, config files, the output of any other command: grep treats them all the same.

grep "error" logfile.txt        # lines containing "error"
grep -i "error" logfile.txt     # case-insensitive
grep -r "TODO" ~/projects/      # recursive through a whole directory
grep -v "comment" code.py       # invert: lines that DON'T match
grep -n "function" script.js    # show line numbers
grep -A 2 -B 2 "exception" err.log   # 2 lines of context after/before

Starter regex, two anchors: ^Start matches lines beginning with Start; End$ matches lines ending with End. That plus the flags above covers 90% of real usage. Combined with Part 1's tail -f: tail -f app.log | grep -i error — a live filtered error feed. That's an on-call engineer's screen.

sed — find and replace at stream speed

sed 's/old/new/' file.txt       # replace first "old" per line
sed 's/old/new/g' file.txt      # replace ALL occurrences (g = global)
sed '/pattern/d' file.txt       # delete matching lines

sed prints the transformed text to stdout; the file itself is untouched — redirect to a new file to keep the result. (In-place editing exists but behaves differently across systems; skip it while learning.)

awk — the column extractor

Text in columns (ps output, CSVs, logs) is awk's home turf:

awk '{print $1}' file.txt          # first column of every line
awk -F, '{print $2}' data.csv      # second column, comma-separated
awk '$3 > 100' data.txt            # only lines where column 3 > 100
awk '{sum += $1} END {print sum}' numbers.txt   # total a column

Honest guidance: learn grep deeply, and keep sed/awk as the four recipes above. Full mastery of either is a separate (optional) journey.

3. Processes — see it, control it, kill it

Everything running is a process with a PID (process ID).

See: ps aux (snapshot of everything, from all users) · top (live view — q quit, M sort by memory, P by CPU, k kill by PID) · htop (friendlier top; usually needs installing).

The classic combo: ps aux | grep firefox — pipe brick number one earning its keep.

Kill = send a signal, not just "destroy":

• kill PID (or kill -15) — SIGTERM, the polite request: "please shut down, clean up after yourself." Default choice.
• kill -9 PID — SIGKILL, the fire axe: no cleanup, no goodbye. For processes that ignored -15. Last resort, not first instinct.
• killall firefox — same idea, by name instead of PID.

Background jobs: append & to start a command in the background (long_job &). jobs lists them, fg brings one forward. Already running in the foreground? Ctrl+Z suspends it, then bg resumes it in the background. And nohup command & keeps it running even after you log out (output lands in nohup.out) — essential on remote servers.

4. Environment variables — the shell's settings

Variables the shell and programs read for configuration. See one: echo $HOME. See all: env.

The one that explains a famous error: PATH — the list of directories the shell searches when you type a command name. "command not found" usually means the program isn't in any PATH directory, not that it doesn't exist. (which from Part 1 shows what PATH found.)

• This session only: export MY_VAR=value
• Permanently: add the export line to ~/.bashrc (Bash/most Linux) or ~/.zshrc (Zsh/modern macOS), then source ~/.bashrc to reload without reopening the terminal.

That file — .bashrc — is your terminal's home base. You're about to put aliases there too.

5. Your first shell script

Anything you can type, you can save and rerun. A script is just commands in a file:

#!/bin/bash
# backup.sh — copy Documents to a dated backup folder
echo "Starting backup..."
mkdir -p ~/backup
cp -r ~/Documents ~/backup/
echo "Backup completed at $(date)"

Line 1 is the shebang — #!/bin/bash tells the system which interpreter runs this file. Then: chmod u+x backup.sh (Part 1 callback — scripts need execute permission) and run it with ./backup.sh.

The building blocks:

NAME="Priya"                  # variable — NO spaces around =
echo "Hello, $NAME"           # use with $
TODAY=$(date)                 # command substitution: output → variable

if [ -f ~/.bashrc ]; then     # conditionals — the spaces inside [ ] are required
    echo "found it"
fi

for i in {1..5}; do           # loops
    echo "run $i"
done

greet() {                     # functions; $1 = first argument
    echo "Hello, $1!"
}
greet "World"

Two beginner traps, named now: NAME = "Priya" (spaces around =) fails, and if [-f file] (no spaces inside brackets) fails. Everything else is the commands you already know, in a file. This is the moment the terminal stops being a tool and becomes leverage.

6. Installing software: package managers

No .exe downloads. A package manager fetches, installs, and updates software from trusted repositories:

Mind the classic confusion: update refreshes the catalog; upgrade actually installs newer versions. (Homebrew itself is a one-command install — copy the current command from brew.sh rather than from any tutorial, including this one.)

7. SSH — the bridge to the cloud

Here's the payoff of the whole series. SSH (Secure Shell) puts your terminal on someone else's computer — encrypted, from anywhere:

ssh username@server-address

That's it. Your prompt changes, and every skill from Parts 1 and 2 — navigation, permissions, pipes, tail -f, scripts — now operates a machine that could be on another continent. Useful options: -p 2222 (non-standard port), -i ~/.ssh/mykey (choose a specific key).

Copy files across the bridge with scp:

scp report.txt user@server:/home/user/     # local → remote
scp user@server:/var/log/app.log ./        # remote → local
scp -r project/ user@server:/opt/          # directories: -r

SSH keys — retire your password. ssh-keygen -t ed25519 creates a key pair: a public key (a lock — hand copies to any server) and a private key (the only key — it never leaves your machine, and it lives at permissions 600, exactly the number Part 1 taught you). Push the lock to a server with ssh-copy-id user@server, and from then on you log in without a password, more securely than with one. (Older guides show -t rsa -b 4096; still fine where required, ed25519 is the modern default.)

Why this is the bridge: this is how you'll operate every cloud server. When you launch your first AWS EC2 instance in the next track, AWS hands you a key pair, and your first command as a cloud engineer will be ssh -i mykey.pem ubuntu@your-server (the username varies by server image — ubuntu, ec2-user). Same command, same 600 permissions, same terminal. Cloud engineering is terminal skills pointed at rented computers.

8. The productivity layer

• Aliases — shortcuts you define in ~/.bashrc: alias ll='ls -la' · alias update='sudo apt update && sudo apt upgrade'. Reload with source ~/.bashrc.
• Line editing: Ctrl+A / Ctrl+E (start/end of line) · Ctrl+U / Ctrl+K (cut to start/end) · Ctrl+Y (paste it back) · Ctrl+D (exit shell). History: !! reruns the last command (sudo !! is the classic), !string reruns the last command starting with string.
• tmux — terminal sessions that survive disconnection. Start with tmux, detach with Ctrl+B d, walk away, reattach later with tmux attach — your session, still running, exactly as you left it. On a remote server this is the difference between "my Wi-Fi dropped" being a shrug or a disaster. (screen is the older equivalent; split panes with Ctrl+B % and Ctrl+B " when you're ready.)

Recap

In one breath: commands are LEGO bricks snapped together with |, streams go where you point them (> overwrites, >> appends, 2> catches errors, /dev/null swallows everything); grep searches anything, sed replaces, awk pulls columns; processes are seen with ps/top and signaled with kill (-15 politely, -9 as the fire axe) or backgrounded with & and nohup; the environment lives in variables like PATH and permanently in .bashrc; a script is commands in a file behind a shebang, made runnable with chmod u+x; software arrives via apt or brew (update the list, then upgrade); and ssh puts this whole toolkit on any server on Earth — which is precisely what a cloud career is.

Next: the AWS track — where "any server on Earth" becomes a server you launched, and this lesson's SSH section becomes your daily commute.

Quiz

1. What does ps aux | grep nginx do? a) Restarts nginx b) Filters the full process list down to lines mentioning nginx ✅ c) Kills nginx d) Installs nginx ps aux prints all processes to stdout; the pipe feeds that into grep, which passes through only matching lines. Bricks, snapped together.

2. You run a nightly job with report.sh > report.txt — but you want each night ADDED to the file, not replacing it. What changes? a) Use < b) Use >> ✅ c) Use 2> d) Use /dev/null > truncates the file every run; >> appends. 2> only redirects errors, and /dev/null would discard the report entirely.

3. A process is frozen and ignored kill 4172. What's the appropriate next step? a) kill -9 4172 ✅ b) kill -1 4172 c) Reboot immediately d) nohup 4172 Default kill sends SIGTERM (-15), the polite request. When a process ignores it, SIGKILL (-9) forces termination without cleanup — the fire axe, used second, not first.

4. You typed a program's name and got "command not found," but you know it's installed. The most likely explanation? a) The file was deleted b) Its directory isn't in your PATH ✅ c) You need >> d) The shell is broken The shell only searches the directories listed in PATH. If the program lives elsewhere, the shell can't see it — add its directory to PATH (export, then persist in .bashrc) or call it by full path.

5. In an SSH key pair, which part goes on the server? a) The private key b) Both halves c) The public key ✅ d) Neither — SSH keys stay local The public key is the lock: copy it to any server (ssh-copy-id). The private key is the key: it never leaves your machine and stays at permissions 600 — the same "owner-only" number from Part 1.