Preparing the Environment for CTF Shell Scripts
Before copying scripts – two checks that save from the loss of time already in the competition.
The first is to make sure that bash is used, not dash or sh. In Debian and Ubuntu /bin/sh refers to the dash where /dev/tcp There is no and half of the designs are silently broken. Check: bash --version. On macOS by default is Bash 3.2 - Apple does not update because of the GPLv3 license. Associative arrays, wait -n and a number of designs from the scripts below will not work without brew install bash.
The second is to create a working directory in advance: mkdir -p ~/ctf/scan && cd ~/ctf/scan. All weekend files fall into one place. In the middle of the competition, looking for results all over the disc is the loss of rounds that are worth the points.
Minimum set: curl, grep, awk, sed is in any distribution. For advanced parsing is useful jq – put through apt install jq. Each script begins with a shebang #!/bin/bash and becomes executed after chmod +x script.sh. Without chmod +x - Permission denied, the first trap every newbie goes through.
Third, each serious script longer than three lines begins with set -euo pipefail. Flag -e interrupts execution on the first error, -u swearing at non-initialized variables, -o pipefail throws non-zero return code through the pipes. Without this three, the script, when you fail, quietly continues to work and gives out the trash result that you will take for a clean conclusion. I lost half an hour on one CTF because the script is without -u silently framed the empty variable in the URL — and the phased root of the site instead of the desired path.
[Applicable: CTF jeopardy, CTF attack-defense, internal pentest on Linux infrastructure]
Bash ports selection script: scanning via /dev/tcp
On CTF-sites with restrictions on nmap tools is not available. Bash is able to check TCP ports through a virtual device /dev/tcp – it is not a file on the disk, but a built-in shell mechanism. Recording in /dev/tcp/host/port initiates TCP connection: the port is open - return code 0, closed - error. In MITRE ATT&CK terminology, using bash to scan mapping on Active Scanning (T1595, Reconnaissance), and the interpreter itself, Unix Shell (T1059.004, Execution tactics). Mapping is conditional: ATT&CK describes TTPs of real adversary rather than training CTF scenarios, but understanding classification is useful for pentester reports.
Preconditions and limitations: - Works if: bash is compiled without --disable-net-redirections (default /dev/tcp included in the vast majority of assemblies; disabled only in rare minimal/hardened distributions) - Does not work if: used dash or sh; bash is assembled without network redirect support; target by firewall with DROP policy (timeout instead of RST stretches the scan to infinity) - /dev/tcp works only with TCP – you need UDP nmap -sU or nc -u -w 1 $HOST $port
The basic sequential option is the cycle for port in $(seq 1 1024) with an attempt echo >/dev/tcp/$HOST/$port in subshell. At 1024 ports with a timeout of ~1 second for each closed port takes up to 17 minutes. This is unacceptable on the CTF. Solution – parallelization:
#!/bin/bash
set -uo pipefail
HOST="${1:?Usage: $0 <target>}"; MAX_JOBS=50
scan_port() {
(echo >/dev/tcp/"$HOST"/"$1") 2>/dev/null && echo "[+] $HOST:$1 open"
}
for port in $(seq 1 1024); do
scan_port "$port" &
(( $(jobs -r | wc -l) >= MAX_JOBS )) && wait -n
done; wait
Debriefing line. set -uo pipefail – strict regime without -e, because errors on closed ports are expected behavior, and -e Here everything will kill. MAX_JOBS=50 – ceiling of simultaneous processes: 1024 background machine will survive, 65535 – most likely not, the core will be drawn to the limit on descriptors. Function scan_port turns the port check in subshell through the brackets () – isolates the closed port error from the main process. Redirect 2>/dev/null suppresses messages Connection refused. Operator && after subshell performs echo only with a successful connection. Construction jobs -r | wc -l considers active background processes, wait -n wait for the completion of any one of them, freeing the slot. The Final wait without arguments, all the remaining ones are waiting.
Alternative through xargs – for those who don’t want to manage the queue manually: seq 1 1024 | xargs -P 50 -I{} bash -c '(echo >/dev/tcp/"$1"/{}) 2>/dev/null && echo "[+] $1:{} open"' _ "$HOST". Here -P 50 – the same 50 parallel processes, xargs itself steers the queue.
When the technique is NOT working: Banner-grubbing through /dev/tcp unreliable – to determine the versions of services you need nmap -sV or manual nc -w 2 $HOST $port. With a DROP firewall policy, each closed port hangs to a system timeout (~30 seconds), which breaks the parallel strategy. It is bypassed through timeout 1 bash -c "echo >/dev/tcp/$HOST/$port", but everyone timeout generates an additional process.
Port scanning is the Port Scan category (#14 according to the AbuseIPDB classification), a standard trigger for automatic locks. On CTF it is usually not a problem, on a real pentest - consider: Sigma-rule lnx_shell_susp_commands.yml catches suspicious execution of shell commands, and the AbuseIPDB counter can increase confidence score address above the blocking threshold (recommended cutoff - 75 out of 100).
Parsing output commands bash: nmap, curl and awk
The main task on the CTF is not to run the scanner, but to quickly pull the desired one out of its output. Most guides end on grep "open" – this covers from the force a tenth of the real tasks of parsing.
Parsing nmap -oG: grepable format
The key -oG (grepable output) issues one line on the host with all open ports - the format is created specifically for conveyor processing. The standard multi-line nmap output is inconvenient: each port on a separate line, between hosts empty lines and decorative frames. Launch: nmap -sC -sV -oG scan.grep -oN scan.txt 10.10.0.0/24. Two formats at the same time are a habit that saves time. Grepable for scripts, normal for eyes.
Extract IP with open port 80: grep '80/open' scan.grep | awk '{print $2}'. Here awk '{print $2}' outputs the second line field — IP address, the first field in the format -oG This word Host:.
Get a list of ports of a specific host: grep '10.10.0.5' scan.grep | grep -oE '[0-9]+/open' | cut -d'/' -f1. Regularity [0-9]+/open through grep -oE (extended regulars) removes the port number before /open, cut -d'/' -f1 cuts everything off after the slash. I use -oE instead of -oP, because BSD grep on macOS does not support Perl-compatible regulars — -oE works everywhere.
Script for mass processing of greepable-output:
#!/bin/bash
# Парсит nmap -oG -> ip:
while read -r line; do
ip=$(echo "$line" | awk '{print $2}')
ports=$(echo "$line" | grep -oE '[0-9]+/open' \
| cut -d'/' -f1 | tr '\n' ',')
[ -n "$ports" ] && echo "$ip -> ${ports%,}"
done < <(grep '/open/' "$1")
Parsing: while read -r line reads the whole lines, the flag -r disables the interpretation of reverse slashes - without it, a string with paths like /etc/passwd lose the slashes. Process substitution < <(grep ...) transfers only lines with open ports to the cycle, skipping comments and empty. tr '\n' ',' glues the ports through the comma. Construction ${ports%,} removes the final comma through pattern removal – without it, the output ends on 22,80,443, with a hanging tail.
Also useful: withdraw all hosts with a specific service. grep -i 'ssh' scan.grep | awk '{print $2}' return the IP of all the machines where nmap found the SSH. Combination with sorting: grep '/open/' scan.grep | awk '{print $2}' | sort -u gives a deduplicated list of all living hosts.
Parsing HTTP responses curl
On CTF, you often need to process dozens of HTTP answers: find a page with a different length (a hint to the vulnerability), pull the token out of the title, filter by status.
Basic pattern – curl -s -o /dev/null -w "%{http_code} %{size_download}" – returns HTTP code and response size in bytes without body. Turning into a cycle with a wordlist: while read -r dir; do code=$(curl -s -o /dev/null -w "%{http_code}" "http://target/$dir"); [ "$code" != "404" ] && echo "$code $dir"; done < wordlist.txt. Simple directory fuzzer from one line. For serious phaseding is better ffuf or gobuster, but when they are not, the cycle with curl saves.
Critical Beginner Mistake – Gaps in [ "$code" != "404" ]. Square brackets — team challenge test. Gaps around brackets and operator are mandatory. Recording ["$code" without a gap – [: command not found. The most frequent question from beginners and the most irritating syntactic nuance of Bash. Everyone goes through it, and every time you want to knock on the table.
For JSON-answers is indispensable jq: curl -s http://target/api/user | jq -r '.token'. Without jq – through grep -oE '"token":"[^"]+"' | cut -d'"' -f4, but the design is fragile: breaks on nested quotes or non-standard gaps in JSON. To extract headlines: curl -sI http://target | grep -i 'set-cookie' – flag -I sends a HEAD request and returns only titles.
A few patterns sed, which on the CTF are constantly useful. Removing blank lines: sed '/^$/d' output.txt. Extraction of value after password=: sed -n 's/.*password[=:][[:space:]]*\([^ ]*\).*/\1/p' config.txt. Output lines from 10 to 20 from the log: sed -n '10,20p' access.log. Breakdown of the line by divider: echo "hash:salt:user" | sed 's/:/\n/g' – three separate lines.
Search for flags by regex template: grep and regular expressions for CTF
In the terminology of MITRE ATT&CK, this intersection of File and Directory Discovery (T1083, Discovery Tactics) and Unix Shell (T1059.004) — the framework describes TTPs of real-world attackers rather than CTF tasks, but knowledge of mapping is useful for pentester reports. Targeted data on CTFs – flags.
Flags on different platforms use predictable formats. According to Capture The Flag Cheatsheet, typical patterns are: flag{...}, FLAG{...}, CTF{...}, HTB{...}, THM{...}. Typical storage areas: /root/root.txt, home directories of users, configs, environment variables (through env), HTTP headers and cookies, database dumps. Knowing the formats and locations, you can write a universal search engine:
#!/bin/bash
PATTERN='(flag|FLAG|CTF|HTB|THM|\{[A-Za-z0-9_\-]+\}'
TARGET="${1:-.}"
echo "[*] :"
grep -rIoE "$PATTERN" "$TARGET" 2>/dev/null | head -50
echo "[*] ( strings):"
find "$TARGET" -type f -exec strings {} + 2>/dev/null \
| grep -oE "$PATTERN" | head -20
echo "[*] :"
env | grep -oE "$PATTERN"
Debrief. PATTERN store regex for six CTF formats through | (alternation in ERE). Figure brackets shielded \{ and \}, because in extended regulars { – quantifier. Inside the brackets [A-Za-z0-9_\-] – permissible symbols of the flag body. Construction ${1:-.} uses the first argument or current directory as a default – convenient when a lazy call without parameters. Flag -r makes grep recursive, -I passes binary files (otherwise grep will choke and spat out the trash), -o only brings out the coincidence, -E includes extended regulars. head -50 limits the output - on a CTF machine with thousands of files without a limit, the terminal will fly into an endless scroll.
Block with find ... -exec strings {} + processes binary files: utility strings extracts the sequences of printed symbols, and grep filters by pattern. Exactly through strings most often find flags in compiled binary and ELF files – a standard approach from the HackTheBox and TryHackMe cheats: strings /path/to/binary | grep -E 'flag|CTF|HTB'.
Preconditions and limitations: - Works if: have read-access to target directories, GNU grep is installed - Does not work if: files in UTF-16 encoding — grep will not see ASCII strings without prior iconv -f UTF-16 -t UTF-8 - Does not work if: the flag is encrypted, obfused, or broken down in pieces in different files (and then a very different story begins)
Advanced templates: base64, hex and obfuscation
Base64-wrapped flags are a frequent reception on jeopardy. Approach: first find lines similar to base64 (20+ characters from base64-alphabet with possible = at the end), then drive each through the decoder. In one line: grep -rIoE '[A-Za-z0-9+/]{20,}={0,2}' "$TARGET" 2>/dev/null | while read -r b; do echo "$b" | base64 -d 2>/dev/null | grep -oE "$PATTERN"; done. The first grep finds candidates, the cycle decodes and filters.
Hex-coded flags: grep -rIoE '([0-9a-fA-F]{2}){10,}' "$TARGET" 2>/dev/null | while read -r h; do echo "$h" | xxd -r -p 2>/dev/null | grep -oE "$PATTERN"; done. Here xxd -r -p converts hex into binary data, grep looking for a flag as a result.
To search in HTTP responses: curl -s http://target/ | grep -oE '<!--.*-->' | grep -oE "$PATTERN" – extracts HTML comments and searches for a flag in them. Option for Headings: curl -sI http://target/ | grep -oE "$PATTERN" The flag may lie in a custom title. On one CTF I found the flag in the title X-Secret-Flag The task was designed for those who do not look at the headlines. And then in the same spirit..
Bash one-liner for CTF: ready selection
One-liners are the main instrument at competitions when there is no time for a full-fledged script. Below are the ones I use on each CTF.
Mass ping subnets: for i in $(seq 1 254); do ping -c 1 -W 1 10.10.0.$i &>/dev/null && echo "10.10.0.$i alive" & done; wait. -c 1 One ICMP package, -W 1 - timeout second. Ampersand & parallelist checking all hosts, wait waiting for completion.
Extracting unique IPs from the log: grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}' access.log | sort -u. Regularity ([0-9]{1,3}\.){3}[0-9]{1,3} match IPv4 addresses, sort -u removes duplicates.
Quick selection of subdomains: while read -r sub; do host "$sub.target.com" | grep -q "has address" && echo "$sub.target.com"; done < subdomains.txt. Team host makes a DNS request, grep -q silently checks for a response.
Decoding all base64 lines from the file: grep -oE '[A-Za-z0-9+/]{4,}={0,2}' dump.txt | while read -r b; do decoded=$(echo "$b" | base64 -d 2>/dev/null) && echo "$decoded"; done. Useful on jeopardy when analyzing dumps.
Monitoring changes on the web page (attack-defense): while true; do curl -s http://target/page | md5sum; sleep 5; done. The hash is recalculated every 5 seconds – once it has changed, someone patched the service or exploited the vulnerability.
Mass checking HTTP statuses from the URL list: while read -r url; do echo "$(curl -s -o /dev/null -w '%{http_code}' "$url") $url"; done < urls.txt. Result – Table 200 http://target/admin, 403 http://target/backup, 301 http://target/old.
Search for SUID binary to escalate privileges: find / -perm -4000 -type f 2>/dev/null. The standard first step is to find binary with a SUID-bit and check them through GTFOBins (gtfobins.github.io). This is also the technique of Unix Shell (T1059.004) - Atomic Red Team includes the test "Harvest SUID executable files" with such a team.
Rake of bash scripts: IFS, quotes and race conditions
Most of the bugs in CTF scripts are not in logic, but in the features of the syntax bash. Three categories of errors break scripts most often.
Quotes and word splitting. The variable without double quotes passes through word splitting and globbing. File called flag file.txt. Team cat $file (without quotes) interprets the gap as a divider: bash sees cat flag and file.txt – two non-existent files. Right: cat "$file". The rule is to always wrap variables in double quotes, except when word splitting is needed intentionally, for example, when iteration for port in $ports_list.
IFS and non-standard dividers. Variable IFS (Internal Field Separator) determines by which characters bash breaks strings into words. By default – space, tabulation, translation of the line. If parsing CSV or output with a non-standard divider, overdeter IFS locally: IFS=':' read -r user pass hash <<< "$line". This design will break the string admin
assword123:md5hash on three variables. The global change in IFS without recovery is the cause of unexplained bugs, when the script works fine on some data and breaks down on others. Always restore: OLD_IFS="$IFS"; IFS=','; ...; IFS="$OLD_IFS", or use IFS only in context read. I once killed half an hour on the debug of the script, which broke on the third pass of the cycle - it turned out that the IFS re-recorded in the first iteration and everything went awry.
Race conditions at parallel recording. When multiple background processes are written into a single file through >>, the records can stir. Two processes simultaneously refine the result in results.txt – get a line [+] 10[+] 10.10.0.5:80 open.10.0.3:22 open, where two conclusions are glued together in porridge. Solutions two: either flock -x results.lock bash -c "echo '[+] $HOST:$port open' >> results.txt" for exclusive blocking, or collect output through stdout - each background process writes in stdout, wait everyone is waiting, and redirecting the entire block collects the result after completion.
Verification of arguments. Construction [ -z "${1:-}" ] && echo "Usage: $0 <target>" && exit 1 checks for the first argument. Recording ${1:-} returns an empty string if $1 not specified – without this set -u The script will fall with a mistake unbound variable Even before the tip is withdrawn.
Pentester Routine Automation: Functions for Reuse
On the CTF, the same set of actions is repeated from task to task. Instead of copying single-ins – take the routine into functions and download through source ~/ctf/functions.sh.
Quick scan function: qscan() { nmap -sC -sV -oG "$HOME/ctf/$1.grep" -oN "$HOME/ctf/$1.txt" "$1"; }. After downloading through source calling qscan 10.10.0.5 – the results are automatically saved with the name of the goal, you do not need to remember the keys and paths every time.
Flag Search Function: fflag() { grep -rIoE '(flag|FLAG|CTF|HTB|THM)\{[A-Za-z0-9_-]+\}' "${1:-.}" 2>/dev/null; }. Challenge: fflag /tmp/challenge – one word instead of a line with a regular.
Before copying scripts – two checks that save from the loss of time already in the competition.
The first is to make sure that bash is used, not dash or sh. In Debian and Ubuntu /bin/sh refers to the dash where /dev/tcp There is no and half of the designs are silently broken. Check: bash --version. On macOS by default is Bash 3.2 - Apple does not update because of the GPLv3 license. Associative arrays, wait -n and a number of designs from the scripts below will not work without brew install bash.
The second is to create a working directory in advance: mkdir -p ~/ctf/scan && cd ~/ctf/scan. All weekend files fall into one place. In the middle of the competition, looking for results all over the disc is the loss of rounds that are worth the points.
Minimum set: curl, grep, awk, sed is in any distribution. For advanced parsing is useful jq – put through apt install jq. Each script begins with a shebang #!/bin/bash and becomes executed after chmod +x script.sh. Without chmod +x - Permission denied, the first trap every newbie goes through.
Third, each serious script longer than three lines begins with set -euo pipefail. Flag -e interrupts execution on the first error, -u swearing at non-initialized variables, -o pipefail throws non-zero return code through the pipes. Without this three, the script, when you fail, quietly continues to work and gives out the trash result that you will take for a clean conclusion. I lost half an hour on one CTF because the script is without -u silently framed the empty variable in the URL — and the phased root of the site instead of the desired path.
[Applicable: CTF jeopardy, CTF attack-defense, internal pentest on Linux infrastructure]
Bash ports selection script: scanning via /dev/tcp
On CTF-sites with restrictions on nmap tools is not available. Bash is able to check TCP ports through a virtual device /dev/tcp – it is not a file on the disk, but a built-in shell mechanism. Recording in /dev/tcp/host/port initiates TCP connection: the port is open - return code 0, closed - error. In MITRE ATT&CK terminology, using bash to scan mapping on Active Scanning (T1595, Reconnaissance), and the interpreter itself, Unix Shell (T1059.004, Execution tactics). Mapping is conditional: ATT&CK describes TTPs of real adversary rather than training CTF scenarios, but understanding classification is useful for pentester reports.
Preconditions and limitations: - Works if: bash is compiled without --disable-net-redirections (default /dev/tcp included in the vast majority of assemblies; disabled only in rare minimal/hardened distributions) - Does not work if: used dash or sh; bash is assembled without network redirect support; target by firewall with DROP policy (timeout instead of RST stretches the scan to infinity) - /dev/tcp works only with TCP – you need UDP nmap -sU or nc -u -w 1 $HOST $port
The basic sequential option is the cycle for port in $(seq 1 1024) with an attempt echo >/dev/tcp/$HOST/$port in subshell. At 1024 ports with a timeout of ~1 second for each closed port takes up to 17 minutes. This is unacceptable on the CTF. Solution – parallelization:
#!/bin/bash
set -uo pipefail
HOST="${1:?Usage: $0 <target>}"; MAX_JOBS=50
scan_port() {
(echo >/dev/tcp/"$HOST"/"$1") 2>/dev/null && echo "[+] $HOST:$1 open"
}
for port in $(seq 1 1024); do
scan_port "$port" &
(( $(jobs -r | wc -l) >= MAX_JOBS )) && wait -n
done; wait
Debriefing line. set -uo pipefail – strict regime without -e, because errors on closed ports are expected behavior, and -e Here everything will kill. MAX_JOBS=50 – ceiling of simultaneous processes: 1024 background machine will survive, 65535 – most likely not, the core will be drawn to the limit on descriptors. Function scan_port turns the port check in subshell through the brackets () – isolates the closed port error from the main process. Redirect 2>/dev/null suppresses messages Connection refused. Operator && after subshell performs echo only with a successful connection. Construction jobs -r | wc -l considers active background processes, wait -n wait for the completion of any one of them, freeing the slot. The Final wait without arguments, all the remaining ones are waiting.
Alternative through xargs – for those who don’t want to manage the queue manually: seq 1 1024 | xargs -P 50 -I{} bash -c '(echo >/dev/tcp/"$1"/{}) 2>/dev/null && echo "[+] $1:{} open"' _ "$HOST". Here -P 50 – the same 50 parallel processes, xargs itself steers the queue.
When the technique is NOT working: Banner-grubbing through /dev/tcp unreliable – to determine the versions of services you need nmap -sV or manual nc -w 2 $HOST $port. With a DROP firewall policy, each closed port hangs to a system timeout (~30 seconds), which breaks the parallel strategy. It is bypassed through timeout 1 bash -c "echo >/dev/tcp/$HOST/$port", but everyone timeout generates an additional process.
Port scanning is the Port Scan category (#14 according to the AbuseIPDB classification), a standard trigger for automatic locks. On CTF it is usually not a problem, on a real pentest - consider: Sigma-rule lnx_shell_susp_commands.yml catches suspicious execution of shell commands, and the AbuseIPDB counter can increase confidence score address above the blocking threshold (recommended cutoff - 75 out of 100).
Parsing output commands bash: nmap, curl and awk
The main task on the CTF is not to run the scanner, but to quickly pull the desired one out of its output. Most guides end on grep "open" – this covers from the force a tenth of the real tasks of parsing.
Parsing nmap -oG: grepable format
The key -oG (grepable output) issues one line on the host with all open ports - the format is created specifically for conveyor processing. The standard multi-line nmap output is inconvenient: each port on a separate line, between hosts empty lines and decorative frames. Launch: nmap -sC -sV -oG scan.grep -oN scan.txt 10.10.0.0/24. Two formats at the same time are a habit that saves time. Grepable for scripts, normal for eyes.
Extract IP with open port 80: grep '80/open' scan.grep | awk '{print $2}'. Here awk '{print $2}' outputs the second line field — IP address, the first field in the format -oG This word Host:.
Get a list of ports of a specific host: grep '10.10.0.5' scan.grep | grep -oE '[0-9]+/open' | cut -d'/' -f1. Regularity [0-9]+/open through grep -oE (extended regulars) removes the port number before /open, cut -d'/' -f1 cuts everything off after the slash. I use -oE instead of -oP, because BSD grep on macOS does not support Perl-compatible regulars — -oE works everywhere.
Script for mass processing of greepable-output:
#!/bin/bash
# Парсит nmap -oG -> ip:
while read -r line; do
ip=$(echo "$line" | awk '{print $2}')
ports=$(echo "$line" | grep -oE '[0-9]+/open' \
| cut -d'/' -f1 | tr '\n' ',')
[ -n "$ports" ] && echo "$ip -> ${ports%,}"
done < <(grep '/open/' "$1")
Parsing: while read -r line reads the whole lines, the flag -r disables the interpretation of reverse slashes - without it, a string with paths like /etc/passwd lose the slashes. Process substitution < <(grep ...) transfers only lines with open ports to the cycle, skipping comments and empty. tr '\n' ',' glues the ports through the comma. Construction ${ports%,} removes the final comma through pattern removal – without it, the output ends on 22,80,443, with a hanging tail.
Also useful: withdraw all hosts with a specific service. grep -i 'ssh' scan.grep | awk '{print $2}' return the IP of all the machines where nmap found the SSH. Combination with sorting: grep '/open/' scan.grep | awk '{print $2}' | sort -u gives a deduplicated list of all living hosts.
Parsing HTTP responses curl
On CTF, you often need to process dozens of HTTP answers: find a page with a different length (a hint to the vulnerability), pull the token out of the title, filter by status.
Basic pattern – curl -s -o /dev/null -w "%{http_code} %{size_download}" – returns HTTP code and response size in bytes without body. Turning into a cycle with a wordlist: while read -r dir; do code=$(curl -s -o /dev/null -w "%{http_code}" "http://target/$dir"); [ "$code" != "404" ] && echo "$code $dir"; done < wordlist.txt. Simple directory fuzzer from one line. For serious phaseding is better ffuf or gobuster, but when they are not, the cycle with curl saves.
Critical Beginner Mistake – Gaps in [ "$code" != "404" ]. Square brackets — team challenge test. Gaps around brackets and operator are mandatory. Recording ["$code" without a gap – [: command not found. The most frequent question from beginners and the most irritating syntactic nuance of Bash. Everyone goes through it, and every time you want to knock on the table.
For JSON-answers is indispensable jq: curl -s http://target/api/user | jq -r '.token'. Without jq – through grep -oE '"token":"[^"]+"' | cut -d'"' -f4, but the design is fragile: breaks on nested quotes or non-standard gaps in JSON. To extract headlines: curl -sI http://target | grep -i 'set-cookie' – flag -I sends a HEAD request and returns only titles.
A few patterns sed, which on the CTF are constantly useful. Removing blank lines: sed '/^$/d' output.txt. Extraction of value after password=: sed -n 's/.*password[=:][[:space:]]*\([^ ]*\).*/\1/p' config.txt. Output lines from 10 to 20 from the log: sed -n '10,20p' access.log. Breakdown of the line by divider: echo "hash:salt:user" | sed 's/:/\n/g' – three separate lines.
Search for flags by regex template: grep and regular expressions for CTF
In the terminology of MITRE ATT&CK, this intersection of File and Directory Discovery (T1083, Discovery Tactics) and Unix Shell (T1059.004) — the framework describes TTPs of real-world attackers rather than CTF tasks, but knowledge of mapping is useful for pentester reports. Targeted data on CTFs – flags.
Flags on different platforms use predictable formats. According to Capture The Flag Cheatsheet, typical patterns are: flag{...}, FLAG{...}, CTF{...}, HTB{...}, THM{...}. Typical storage areas: /root/root.txt, home directories of users, configs, environment variables (through env), HTTP headers and cookies, database dumps. Knowing the formats and locations, you can write a universal search engine:
#!/bin/bash
PATTERN='(flag|FLAG|CTF|HTB|THM|\{[A-Za-z0-9_\-]+\}'
TARGET="${1:-.}"
echo "[*] :"
grep -rIoE "$PATTERN" "$TARGET" 2>/dev/null | head -50
echo "[*] ( strings):"
find "$TARGET" -type f -exec strings {} + 2>/dev/null \
| grep -oE "$PATTERN" | head -20
echo "[*] :"
env | grep -oE "$PATTERN"
Debrief. PATTERN store regex for six CTF formats through | (alternation in ERE). Figure brackets shielded \{ and \}, because in extended regulars { – quantifier. Inside the brackets [A-Za-z0-9_\-] – permissible symbols of the flag body. Construction ${1:-.} uses the first argument or current directory as a default – convenient when a lazy call without parameters. Flag -r makes grep recursive, -I passes binary files (otherwise grep will choke and spat out the trash), -o only brings out the coincidence, -E includes extended regulars. head -50 limits the output - on a CTF machine with thousands of files without a limit, the terminal will fly into an endless scroll.
Block with find ... -exec strings {} + processes binary files: utility strings extracts the sequences of printed symbols, and grep filters by pattern. Exactly through strings most often find flags in compiled binary and ELF files – a standard approach from the HackTheBox and TryHackMe cheats: strings /path/to/binary | grep -E 'flag|CTF|HTB'.
Preconditions and limitations: - Works if: have read-access to target directories, GNU grep is installed - Does not work if: files in UTF-16 encoding — grep will not see ASCII strings without prior iconv -f UTF-16 -t UTF-8 - Does not work if: the flag is encrypted, obfused, or broken down in pieces in different files (and then a very different story begins)
Advanced templates: base64, hex and obfuscation
Base64-wrapped flags are a frequent reception on jeopardy. Approach: first find lines similar to base64 (20+ characters from base64-alphabet with possible = at the end), then drive each through the decoder. In one line: grep -rIoE '[A-Za-z0-9+/]{20,}={0,2}' "$TARGET" 2>/dev/null | while read -r b; do echo "$b" | base64 -d 2>/dev/null | grep -oE "$PATTERN"; done. The first grep finds candidates, the cycle decodes and filters.
Hex-coded flags: grep -rIoE '([0-9a-fA-F]{2}){10,}' "$TARGET" 2>/dev/null | while read -r h; do echo "$h" | xxd -r -p 2>/dev/null | grep -oE "$PATTERN"; done. Here xxd -r -p converts hex into binary data, grep looking for a flag as a result.
To search in HTTP responses: curl -s http://target/ | grep -oE '<!--.*-->' | grep -oE "$PATTERN" – extracts HTML comments and searches for a flag in them. Option for Headings: curl -sI http://target/ | grep -oE "$PATTERN" The flag may lie in a custom title. On one CTF I found the flag in the title X-Secret-Flag The task was designed for those who do not look at the headlines. And then in the same spirit..
Bash one-liner for CTF: ready selection
One-liners are the main instrument at competitions when there is no time for a full-fledged script. Below are the ones I use on each CTF.
Mass ping subnets: for i in $(seq 1 254); do ping -c 1 -W 1 10.10.0.$i &>/dev/null && echo "10.10.0.$i alive" & done; wait. -c 1 One ICMP package, -W 1 - timeout second. Ampersand & parallelist checking all hosts, wait waiting for completion.
Extracting unique IPs from the log: grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}' access.log | sort -u. Regularity ([0-9]{1,3}\.){3}[0-9]{1,3} match IPv4 addresses, sort -u removes duplicates.
Quick selection of subdomains: while read -r sub; do host "$sub.target.com" | grep -q "has address" && echo "$sub.target.com"; done < subdomains.txt. Team host makes a DNS request, grep -q silently checks for a response.
Decoding all base64 lines from the file: grep -oE '[A-Za-z0-9+/]{4,}={0,2}' dump.txt | while read -r b; do decoded=$(echo "$b" | base64 -d 2>/dev/null) && echo "$decoded"; done. Useful on jeopardy when analyzing dumps.
Monitoring changes on the web page (attack-defense): while true; do curl -s http://target/page | md5sum; sleep 5; done. The hash is recalculated every 5 seconds – once it has changed, someone patched the service or exploited the vulnerability.
Mass checking HTTP statuses from the URL list: while read -r url; do echo "$(curl -s -o /dev/null -w '%{http_code}' "$url") $url"; done < urls.txt. Result – Table 200 http://target/admin, 403 http://target/backup, 301 http://target/old.
Search for SUID binary to escalate privileges: find / -perm -4000 -type f 2>/dev/null. The standard first step is to find binary with a SUID-bit and check them through GTFOBins (gtfobins.github.io). This is also the technique of Unix Shell (T1059.004) - Atomic Red Team includes the test "Harvest SUID executable files" with such a team.
Rake of bash scripts: IFS, quotes and race conditions
Most of the bugs in CTF scripts are not in logic, but in the features of the syntax bash. Three categories of errors break scripts most often.
Quotes and word splitting. The variable without double quotes passes through word splitting and globbing. File called flag file.txt. Team cat $file (without quotes) interprets the gap as a divider: bash sees cat flag and file.txt – two non-existent files. Right: cat "$file". The rule is to always wrap variables in double quotes, except when word splitting is needed intentionally, for example, when iteration for port in $ports_list.
IFS and non-standard dividers. Variable IFS (Internal Field Separator) determines by which characters bash breaks strings into words. By default – space, tabulation, translation of the line. If parsing CSV or output with a non-standard divider, overdeter IFS locally: IFS=':' read -r user pass hash <<< "$line". This design will break the string admin
assword123:md5hash on three variables. The global change in IFS without recovery is the cause of unexplained bugs, when the script works fine on some data and breaks down on others. Always restore: OLD_IFS="$IFS"; IFS=','; ...; IFS="$OLD_IFS", or use IFS only in context read. I once killed half an hour on the debug of the script, which broke on the third pass of the cycle - it turned out that the IFS re-recorded in the first iteration and everything went awry.Race conditions at parallel recording. When multiple background processes are written into a single file through >>, the records can stir. Two processes simultaneously refine the result in results.txt – get a line [+] 10[+] 10.10.0.5:80 open.10.0.3:22 open, where two conclusions are glued together in porridge. Solutions two: either flock -x results.lock bash -c "echo '[+] $HOST:$port open' >> results.txt" for exclusive blocking, or collect output through stdout - each background process writes in stdout, wait everyone is waiting, and redirecting the entire block collects the result after completion.
Verification of arguments. Construction [ -z "${1:-}" ] && echo "Usage: $0 <target>" && exit 1 checks for the first argument. Recording ${1:-} returns an empty string if $1 not specified – without this set -u The script will fall with a mistake unbound variable Even before the tip is withdrawn.
Pentester Routine Automation: Functions for Reuse
On the CTF, the same set of actions is repeated from task to task. Instead of copying single-ins – take the routine into functions and download through source ~/ctf/functions.sh.
Quick scan function: qscan() { nmap -sC -sV -oG "$HOME/ctf/$1.grep" -oN "$HOME/ctf/$1.txt" "$1"; }. After downloading through source calling qscan 10.10.0.5 – the results are automatically saved with the name of the goal, you do not need to remember the keys and paths every time.
Flag Search Function: fflag() { grep -rIoE '(flag|FLAG|CTF|HTB|THM)\{[A-Za-z0-9_-]+\}' "${1:-.}" 2>/dev/null; }. Challenge: fflag /tmp/challenge – one word instead of a line with a regular.
