Bash scripts for CTF: Automation routine part 2

Depov

Moderator
Staff member
MODERATOR
ULTIMATE
SUPREME
PREMIUM
MEMBER
Joined
Feb 18, 2025
Messages
646
Reaction score
1,454
Parsing files and preparing data in bash

The third block of CTF routine is file processing. Download the archive, unpack, find among the hundreds of nested files the one where the flag is hidden. By hand, it takes forever. The Linux command line for CTF solves the task in seconds via the pipelines.
Search for flags in the file system

Classic: somewhere on the machine a file with a flag of format is lying flag{...} or CTF{...}. Team grep -rn 'flag{' /home/ 2>/dev/null recursively looking for a string, shows the file name and line number. Redirect 2>/dev/null suppresses access errors to secure directories. Add --include='*.txt' to search only by text files: grep -rn --include='*.txt' 'flag{' / 2>/dev/null.

To search by attributes – find. Files changed in the last hour: find / -type f -mmin -60 2>/dev/null. SUID files are a classic privilege-raising vector, Atomic Red Team for T1059.004 has a “Harvest SUID executable files” test: find / -perm -4000 -type f 2>/dev/null. Files with non-standard extensions: find . -name '*.enc' -o -name '*.b64' -o -name '*.hex'.
Package scripting for CTF file preparation

Competitions often give an archive with nested files or a series of URLs for download. The script below automates the full cycle: pulls the files down the list, determines the type of each, unpacks the archives, and searches for flags.

#!/bin/bash
DIR="ctf_loot"; mkdir -p "$DIR" && cd "$DIR"
while read -r url; do
f=$(basename "$url"); curl -sO "$url"
file "$f" | grep -qi 'zip' && unzip -qo "$f" 2>/dev/null
file "$f" | grep -qiE 'gzip|tar' && tar xf "$f" 2>/dev/null
done < ../urls.txt
grep -rn 'flag{' . 2>/dev/null
find . -name '*.txt' -exec grep -l 'CTF{' {} \;
echo "[+] Обработка завершена, файлы в $DIR/"

Script reads urls.txt line, downloads through curl -sO (flag -O saves the file with the original name). Team file determines the type of content by magical bytes, and grep -qi searching for archive markers without register accounting. If the file is ZIP, there is a unpacking through unzip, if tar/gzip — through tar xf. The final lines are on all the extracted files, looking for patterns of flags of both formats. Conceptually, it is close to the T1119 (Automated Collection) – automatic data collection. T1119 in MITRE ATT & CK formally covers different platforms, but in Atomic Red Team tests for it are implemented only under Windows (command_prompt, powershell), so direct validation on Linux through Atomic tests is not available.
Grep, awk and sed conveyors for CTF tasks

Parsing files with bash utilities are Linux command-line bread for CTF. The patterns I use at every competition:

Pull all IP addresses from the text file: grep -oE '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' file.txt | sort -u. The regular is looking for four groups of digits through the point, sort -u removes duplicates.

Open ports from nmap output to line via comma: grep '^[0-9]' nmap.txt | cut -d'/' -f1 | tr '\n' ',' | sed 's/,$//'. Chain of four utilities: grep takes lines of ports, cut cut off /tcp, tr replaces translations of lines into commas, sed removes the final comma. Result: 22,80,443 – ready string for substitution into other tools.

Decoding base64: base64 -d encoded.txt > decoded.txt. A few coded lines: while read -r line; do echo "$line" | base64 -d; done < encoded.txt. Hex data: xxd -r -p hex.txt > binary.out.

Frequency word analysis (useful for crypto tasks): cat file.txt | tr ' ' '\n' | sort | uniq -c | sort -rn | head -20. The conveyor breaks the text into words, sorts, considers repetitions and outputs the top 20 on the decline.
Bash one-liners for pentesters: working cheat sheet

Not every task should be made in a full-fledged script. Bash one-liners for pentesters often solve the task faster.

Remaking subdomains via DNS: while read -r sub; do host "$sub.target.com" | grep -v 'NXDOMAIN' && echo ">> $sub"; done < subdomains.txt – makes a DNS query for each line from the file and filters non-existent records.

HTTP URL list codes: while read -r url; do code=$(curl -so /dev/null -w '%{http_code}' "$url"); echo "$code $url"; done < urls.txt | grep -v '^404' – outputs the response code for each address, weeding out 404. Flag -w '%{http_code}' makes curl show only status.

Generation of PIN codes: seq -w 0000 9999 > pins.txt – creates a wordlist of all 4-digit PINs with leading zeros. No flag -w meaning 0042 will turn into 42, and this is a different password.

Page Change Monitoring: while true; do curl -s http://target/page | md5sum; sleep 5; done – every 5 seconds considers the hash of the contents. If the line has changed, the update page. Helps catch dynamic flags and rotation of keys on CTF.

Extracting strings from binary: for f in *; do echo "=== $f ==="; strings "$f" | grep -iE 'flag|key|pass'; done – passes through all files in the directory, extracts readable lines and filters potentially significant.

Fast ping-sweep network: for ip in 192.168.1.{1..254}; do (ping -c 1 -W 1 "$ip" &>/dev/null && echo "$ip up") & done; wait – checks each IP in the subnet in parallel with the timeout of 1 second. & before done launches every ping in the background, wait Waiting for the completion of all processes.

Extraction of email addresses: grep -oE '[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}' dump.txt | sort -u – regex for email, result without duplicates. Useful in the analysis of leaks and database dumps.
Five Beginner Mistakes in Bash Scripts for CTF

Forgotten chmod +x. Created a file, launched ./script.sh – received Permission denied. Solution: chmod +x script.sh or launch through bash script.sh. The second mistake is even more common than the first.

Missed quotes around variables. Construction if [ $var == "test" ] will fall if $var contains a space or a blank – bash will set the arguments incorrectly. Always turn: if [ "$var" == "test" ]. On CTFs, variables often contain special symbols from parsing - without quotes, the script breaks unpredictably, and debugging takes longer than writing.

for instead of while read for large files. Construction for line in $(cat bigfile.txt) loads the entire file into memory and breaks down the spaces, not the lines. Wordlist for 14 million records (rockyou.txt) will kill the shell. The right option: while read -r line; do ... done < file.txt – line reading without memory consumption.

No dependency check. Script causes nmap, and nmap is not installed - at the output, empty files and broken filters are further along the chain. Add to the beginning: command -v nmap >/dev/null || { echo "nmap не найден"; exit 1; }. Two lines that save an hour of debugging.

Hardly prescribed values instead of arguments. Script with TARGET="10.10.10.5" in the body instead of TARGET="$1" - one-time. On the CTF, one script is launched on ten machines. Use arguments ($1, $2) and defaults through ${1:-значение}. It doesn’t complicate the code – it makes it overused.
Organization of the Bash script library for CTF

After several competitions you will accumulate a dozen scripts. Organize them correctly from the beginning – a chaotic folder with files of type script2_final_v3.sh after a month turns into a dump, where nothing can be found.

Create a structure: ~/ctf/bin/ for executable scripts, ~/ctf/lib/ for general functions. File ~/ctf/lib/common.sh contains overused designs: argument check, color output, logging functions. Connecting from any script is one line: source ~/ctf/lib/common.sh.

Add ~/ctf/bin/ in the variable PATH: string export PATH="$HOME/ctf/bin:$PATH" in ~/.bashrc. After that, call scripts by name from any directory: recon.sh 10.10.10.5 instead of ~/ctf/bin/recon.sh 10.10.10.5.

Initialize the git repository through git init in ~/ctf/. Versioning allows you to roll back unsuccessful changes and transfer the entire collection to a new machine through git clone. This is especially valuable on CTF: clone the repository for a fresh virtual – and the full arsenal is available in a second.

New script template: shebang, argument check, dependency check, description comment. Four lines of preamble that are worth copying to each file. This is not formalism – in a month you will open the script and in two seconds you will remember why you need it.

For parallel work with several purposes, use tmux. tmux new-session -s ctf creates a session, Ctrl+B % divides the window vertically, Ctrl+B " - horizontally. One panel is an intelligence script, the second is log monitoring, the third is manual work. At competitions with multiple machines, one panel for each target is parallel reconnaissance without switching between terminals.

In my year and a half of participation in the CTF, I came to one observation: 80% of the tasks of the initial and middle level are solved by a dozen bash scripts and two dozen one-liners. Not because the tasks are simple – the routine in them is the same. Download, unpack, find it, send it. The difference between participants who close 12 tasks per competition and those who close 4 is not knowledge. Knowledge is roughly the same at this level. The difference is in automation. One is recruiting nmap -sV -sC -oN ... with your hands on every car and waiting. Another starts the script and after 90 seconds reads the finished results.
 
Top Bottom