On the forensic-task CTF got a file of 2 MB without extension. file returned the succinct "data". Binwalk was silent. I open the hex editor - the first eight bytes are scored by zeros. Someone rubbed them deliberately. The definition of file type by magictes manually took three minutes: line IHDR on displacement 0x0C Unequivocally pointed to the PNG with the killed headline. Two edits in the hex editor – and the restored image contained the flag.
The situation is typical for CTF and for real foresex. The extension is removed, replaced or not at all. The file system is destroyed. The only reliable method of identification is the analysis of binary file formats by signatures directly in the hex dump.
What is magic bytes and why extension doesn't mean anything
Magic bytes is a fixed sequence of bytes at the beginning of a file that uniquely identifies the format. Windows relies on extension: rename .exe in .jpg – the conductor will show the picture icon and will not even choke. Linux team file works differently - she reads the heder of the file and checks the first bytes with the database of known signatures.
Note: common marker of the start of JPEG — FF D8 FF, the third byte (APPn) varies. The JFIF/EXIF format is defined by the line JFIF\0 or Exif\0 within the respective segment of APP0/APP1.
| JPEG/EXIF | FF D8 FF E1 | .... | 0 | FF D9 || GIF89a | 47 49 46 38 39 61 | GIF89a | 0 | 00 3B || PDF | 25 50 44 46 2D | %PDF- | 0 | %%EOF || ZIP | 50 4B 03 04 | PK.. | 0 | 50 4B 05 06 || RAR4 | 52 61 72 21 1A 07 00 | Rar!... | 0 | No || RAR5 | 52 61 72 21 1A 07 01 00 | Rar!... | 0 | No || 7z | 37 7A BC AF 27 1C | 7z... | 0 | No || ELF | 7F 45 4C 46 | . ELF | 0 | No || PE (EXE/DLL) | 4D 5A | MZ | 0 | No || SQLite | 53 51 4C 69 74 65 20 66 6F 72 6D 61 74 20 33 00 | SQLite format 3 | 0 | No || GZIP | 1F 8B | .. | 0 | No || BMP | 42 4D | BM | 0 | No || FLAC | 66 4C 61 43 | fLaC | 0 | No |
Formats with a well-known footer (PNG, JPEG, ZIP, GIF, PDF) allow you to perform signature-based file recovery - carving on the pair "header + footer". Formats without footer require the maximum size when carving or disassembling the internal structure of the binary format.
PE-file signature 4D 5A - just two bytes. Weak: a random coincidence is likely. In practice file and binwalk further check the title PE\0\0 by displacement from the DOS-stab (e_lfanew), which makes the final identification reliable. SQLite Signature (SQLite format 3\0) — 16 bytes of read ASCII: there are no long, unique, false positives. Belkasoft in the documentation on carving directly says: a good signature should be quite long and constant - otherwise the tools will flood you false positive.
Quick Identification: file, binwalk, and hex editor for forensics
Utility file – first line
file mystery_file First thing you start. She reads magic bytes and gives something like PNG image data, 800 x 600, 8-bit/color RGBA. Option file -b --mime-type mystery_file will return only MIME-type - convenient for scripts. If file returns "data" - the file is damaged, obfused or contains a non-standard format. Here begins a real work.
binwalk — analysis of files with nested data
Binwalk searches for all known signatures throughout the file content, not just in the beginning. For steganography and forensics CTF tasks, where another is hidden inside one file is an indispensable thing.
binwalk mystery_file will show a list of the found signatures with their offsets. binwalk -e mystery_file automatically cut all found objects into a separate directory. Typical CTF scenario: inside the PNG picture on the offset 0x1A3F0 A ZIP archive is discovered. Binwalk finds signature 50 4B 03 04, cuts the archive, inside is a text file with a flag. Classic genre.
Another useful thing is the analysis of entropy through binwalk -E mystery_file. If a portion of a file shows an entropy close to 1.0, the data is encrypted or compressed. It helps to detect a payload, obfuscated through Encrypted/Encoded File (T1027.013, Defense Evasion by MITRE ATT&CK).
Hex editor – when automation is powerless
When file and binwalk Silent, open the file directly. For a quick viewing – xxd mystery_file | head -20 in the terminal. For full-fledged work – HxD (Windows), 010 Editor (cross-platform) or hexedit in the terminal. Here’s what a valid PNG header in a hex dump looks like:
Offset 00 01 02 03 04 05 06 07 ASCII
00000000 89 50 4E 47 0D 0A 1A 0A .PNG....
00000008 00 00 00 0D 49 48 44 52 ....IHDR
00000010 00 00 03 20 00 00 02 58 ... ...X
The first 8 bytes are PNG signatures. Byte 89 prevents the interpretation of the file as text. The pair 0D 0A (CR LF) and single 0A (LF) - damage indicators when transmitted in text mode: if they have changed, the file is damaged. Next is the IHDR chank: 00 00 00 0D – data length (13 bytes), 49 48 44 52 – type of chanka in ASCII. A byte parsing of a binary format structure is the only way to determine the type when magic bytes are partially damaged.
File Carving forensics: recovering files from raw data
File carving is a technique for extracting files from a disk image or memory dump without relying on the file system. The tool scans the flow of bytes, looks for the famous header and footer, carves everything between them. NIST Computer Forensics Tools Catalog describes it as "searching for and reconstructing files based on content, than rather file system metadata."
sharing and scalpel for-based file discovery
Foremost is a classic file carving. foremost -i disk.dd -o recovered/ scans the image of the disk and puts the found files on the folders: jpg/, png/, pdf/, zip/. The confirmation configuration lives in /etc/foremost.conf – there you can add custom rules.
Scalpel is a fork with a more flexible setting. Config /etc/scalpel/scalpel.conf contains the commented rules for dozens of formats. Comment on the desired lines and specify the header, footer and maximum file size. Example string for JPEG: jpg y 20000000 \xff\xd8\xff\xe0 \xff\xd9 (in the header, you should not fix the bytes of the length of the APP0 segment - cut off the valid JPEGs with another length) - where y indicates case-sensitive search, 20000000 – limit of 20 MB.
PhotoRec works with hundreds of formats "out of the box" and does not require manual customization. For mass data recovery after image damage is the most. But for custom or obfuscated formats, it's less flexible.
When file carving is not working
Carving has clear limitations, and it is better to know about them in advance:
SSD with TRIM enabled - the controller physically resets the blocks after removal. There is simply no data for recovery.
Full-disc encryption – without a key, the content looks like random noise. Signatures are not detected, carving is useless.
Strong fragmentation – if the file is broken into dozens of non-contiguous clusters, the header/footer carving will only assemble the first fragment. Belkasoft confirms: contiguous data is restored well, and fragmented with a simple signature-based approach - practically not.
Restoration of damaged files: repair of headers manually
The recovery of damaged files in the CTF is usually reduced to one of three scenarios: the first N bytes are erased, the signature is replaced, or the control chunk is damaged. The algorithm is the same for any format.
Step 1 – define the format. Look for characteristic lines inside the file: IHDR/IDAT/IEND indicate PNG, JFIF/Exif – on JPEG, PK - on ZIP. Team strings mystery_file | grep -iE 'ihdr|jfif|pk|%pdf' speeds up the search.
Step 2 – compare the first bytes with the reference signature from the table above.
Step 3 – correct damaged bytes in the hex editor. In HxD or hexedit go to the desired offset, enter the correct values, save.
A specific example is a file where file says “data” but strings shows IHDR:
До: 00 00 00 00 00 00 00 00 00 00 00 0D 49 48 44 52
После: 89 50 4E 47 0D 0A 1A 0A 00 00 00 0D 49 48 44 52
Eight zero bytes are replaced by a standard PNG signature. After saving the file opens in any viewer. On CTF CyberSkyline, a similar tack was solved in the same way: JPEG with the latest byte of signatures (0D instead of 01) did not open until it was brought to the standard FF D8 FF E0 00 10 4A 46 49 46 00 01.
For JPEG, the situation is similar: if there is a string inside the file JFIF – the first bytes should be FF D8 FF E0. For ZIP look for PK and restore the four-byte header 50 4B 03 04.
Automation of signature check in Python
When there are dozens of files, manual verification is impractical. Minimum script for determining file type by magic bytes:
import sys
SIGS = {b'\x89PNG\r\n\x1a\n': 'PNG', b'\xff\xd8\xff': 'JPEG',
b'%PDF': 'PDF', b'PK\x03\x04': 'ZIP', b'\x7fELF': 'ELF',
b'MZ': 'PE', b'Rar!\x1a\x07': 'RAR', b'SQLite format 3\x00': 'SQLite'}
with open(sys.argv[1], 'rb') as f:
header = f.read(16)
for sig, fmt in SIGS.items():
if header[:len(sig)] == sig: print(fmt); break
else: print(f'Unknown: {header[:8].hex(" ")}')
The script reads the first 16 bytes and compares with known signatures. If there are no matches, outputs hex as a starting point for manual analysis. Dictionary SIGS expands to the desired number of formats; in a serious project it is worth putting it in the JSON file.
How Attackers Mask File Type: Anti-Forensic by MITRE ATT&CK
In real forensics and on advanced CTFs, the files are not just corrupted – they are deliberately obfuscated. Several techniques from MITRE ATT&CK that are worth recognizing:
Masquerade File Type (T1036.008, Defense Evasion) – replacement extension or magic bytes. The PE file is disguised as a document. The definition of the file type by magic bytes exposes the substitution: inside the “picture” the title is detected 4D 5A. Beautiful phantom, and inside - exe'shnik.
Obfuscated Files or Information (T1027, Defense Evasion) – obfuscation of content. Encrypted/Encoded File (T1027.013): payload encoded by XOR or Base64, standard signatures do not detect. When carving, such files are missed - you need an analysis of entropy through binwalk -E.
Binary Padding (T1009, Defense Evasion) – adding trash cans to the file. Changes the hash and can shift the internal structure. Binwalk is doing: it searches for signatures throughout the file, not just in the first bytes.
When analyzing a suspicious file from an incident, check all three vectors sequentially: magic bytes, entropy, internal structure.
Practical checklist: from unknown file to result
file mystery_file – if the answer is not “data”, the format is defined.
binwalk mystery_file – if nested signatures are found, extract through binwalk -e.
xxd mystery_file | head -20 – compare the first bytes with the signature table.
strings mystery_file | grep -iE 'ihdr|jfif|pk|pdf' – if magic bytes are corrupted, look for characteristic lines inside the file.
Fix the header in the hex editor by setting the right magic bytes.
Check the result – open the file in the corresponding application or parser.
For disc images (.dd, .raw) steps 1-2 are replaced by foremost -i image.dd -o output/ – the tool itself will scan the entire image and spread the recovered files on the directories.
Most of the mid-level forensic CTFs are solved by this checklist in 5-10 minutes. Complex dumps add XOR-encryption of the title, custom formats or intentional fragmentation - but the basic approach remains the same: find a signature, understand the structure of the binary format, restore.
I often hear that for forensics-tasks it is enough to know three commands: file, binwalk -e and foremost. On the attachment tags, yes, it works. At the DEF CON or HackTheBox competition, no longer. The authors of the Tasks read the same tutorials and deliberately make files that break the standard pipeline: custom by magic, XOR of the first 16 bytes, swapped by CRC in PNG rattles. If there is no skill to open the hex-dump and read the structure of the binary format, the first non-standard task becomes a wall.
The situation is typical for CTF and for real foresex. The extension is removed, replaced or not at all. The file system is destroyed. The only reliable method of identification is the analysis of binary file formats by signatures directly in the hex dump.
What is magic bytes and why extension doesn't mean anything
Magic bytes is a fixed sequence of bytes at the beginning of a file that uniquely identifies the format. Windows relies on extension: rename .exe in .jpg – the conductor will show the picture icon and will not even choke. Linux team file works differently - she reads the heder of the file and checks the first bytes with the database of known signatures.
Note: common marker of the start of JPEG — FF D8 FF, the third byte (APPn) varies. The JFIF/EXIF format is defined by the line JFIF\0 or Exif\0 within the respective segment of APP0/APP1.
| JPEG/EXIF | FF D8 FF E1 | .... | 0 | FF D9 || GIF89a | 47 49 46 38 39 61 | GIF89a | 0 | 00 3B || PDF | 25 50 44 46 2D | %PDF- | 0 | %%EOF || ZIP | 50 4B 03 04 | PK.. | 0 | 50 4B 05 06 || RAR4 | 52 61 72 21 1A 07 00 | Rar!... | 0 | No || RAR5 | 52 61 72 21 1A 07 01 00 | Rar!... | 0 | No || 7z | 37 7A BC AF 27 1C | 7z... | 0 | No || ELF | 7F 45 4C 46 | . ELF | 0 | No || PE (EXE/DLL) | 4D 5A | MZ | 0 | No || SQLite | 53 51 4C 69 74 65 20 66 6F 72 6D 61 74 20 33 00 | SQLite format 3 | 0 | No || GZIP | 1F 8B | .. | 0 | No || BMP | 42 4D | BM | 0 | No || FLAC | 66 4C 61 43 | fLaC | 0 | No |
Formats with a well-known footer (PNG, JPEG, ZIP, GIF, PDF) allow you to perform signature-based file recovery - carving on the pair "header + footer". Formats without footer require the maximum size when carving or disassembling the internal structure of the binary format.
PE-file signature 4D 5A - just two bytes. Weak: a random coincidence is likely. In practice file and binwalk further check the title PE\0\0 by displacement from the DOS-stab (e_lfanew), which makes the final identification reliable. SQLite Signature (SQLite format 3\0) — 16 bytes of read ASCII: there are no long, unique, false positives. Belkasoft in the documentation on carving directly says: a good signature should be quite long and constant - otherwise the tools will flood you false positive.
Quick Identification: file, binwalk, and hex editor for forensics
Utility file – first line
file mystery_file First thing you start. She reads magic bytes and gives something like PNG image data, 800 x 600, 8-bit/color RGBA. Option file -b --mime-type mystery_file will return only MIME-type - convenient for scripts. If file returns "data" - the file is damaged, obfused or contains a non-standard format. Here begins a real work.
binwalk — analysis of files with nested data
Binwalk searches for all known signatures throughout the file content, not just in the beginning. For steganography and forensics CTF tasks, where another is hidden inside one file is an indispensable thing.
binwalk mystery_file will show a list of the found signatures with their offsets. binwalk -e mystery_file automatically cut all found objects into a separate directory. Typical CTF scenario: inside the PNG picture on the offset 0x1A3F0 A ZIP archive is discovered. Binwalk finds signature 50 4B 03 04, cuts the archive, inside is a text file with a flag. Classic genre.
Another useful thing is the analysis of entropy through binwalk -E mystery_file. If a portion of a file shows an entropy close to 1.0, the data is encrypted or compressed. It helps to detect a payload, obfuscated through Encrypted/Encoded File (T1027.013, Defense Evasion by MITRE ATT&CK).
Hex editor – when automation is powerless
When file and binwalk Silent, open the file directly. For a quick viewing – xxd mystery_file | head -20 in the terminal. For full-fledged work – HxD (Windows), 010 Editor (cross-platform) or hexedit in the terminal. Here’s what a valid PNG header in a hex dump looks like:
Offset 00 01 02 03 04 05 06 07 ASCII
00000000 89 50 4E 47 0D 0A 1A 0A .PNG....
00000008 00 00 00 0D 49 48 44 52 ....IHDR
00000010 00 00 03 20 00 00 02 58 ... ...X
The first 8 bytes are PNG signatures. Byte 89 prevents the interpretation of the file as text. The pair 0D 0A (CR LF) and single 0A (LF) - damage indicators when transmitted in text mode: if they have changed, the file is damaged. Next is the IHDR chank: 00 00 00 0D – data length (13 bytes), 49 48 44 52 – type of chanka in ASCII. A byte parsing of a binary format structure is the only way to determine the type when magic bytes are partially damaged.
File Carving forensics: recovering files from raw data
File carving is a technique for extracting files from a disk image or memory dump without relying on the file system. The tool scans the flow of bytes, looks for the famous header and footer, carves everything between them. NIST Computer Forensics Tools Catalog describes it as "searching for and reconstructing files based on content, than rather file system metadata."
sharing and scalpel for-based file discovery
Foremost is a classic file carving. foremost -i disk.dd -o recovered/ scans the image of the disk and puts the found files on the folders: jpg/, png/, pdf/, zip/. The confirmation configuration lives in /etc/foremost.conf – there you can add custom rules.
Scalpel is a fork with a more flexible setting. Config /etc/scalpel/scalpel.conf contains the commented rules for dozens of formats. Comment on the desired lines and specify the header, footer and maximum file size. Example string for JPEG: jpg y 20000000 \xff\xd8\xff\xe0 \xff\xd9 (in the header, you should not fix the bytes of the length of the APP0 segment - cut off the valid JPEGs with another length) - where y indicates case-sensitive search, 20000000 – limit of 20 MB.
PhotoRec works with hundreds of formats "out of the box" and does not require manual customization. For mass data recovery after image damage is the most. But for custom or obfuscated formats, it's less flexible.
When file carving is not working
Carving has clear limitations, and it is better to know about them in advance:
SSD with TRIM enabled - the controller physically resets the blocks after removal. There is simply no data for recovery.
Full-disc encryption – without a key, the content looks like random noise. Signatures are not detected, carving is useless.
Strong fragmentation – if the file is broken into dozens of non-contiguous clusters, the header/footer carving will only assemble the first fragment. Belkasoft confirms: contiguous data is restored well, and fragmented with a simple signature-based approach - practically not.
Restoration of damaged files: repair of headers manually
The recovery of damaged files in the CTF is usually reduced to one of three scenarios: the first N bytes are erased, the signature is replaced, or the control chunk is damaged. The algorithm is the same for any format.
Step 1 – define the format. Look for characteristic lines inside the file: IHDR/IDAT/IEND indicate PNG, JFIF/Exif – on JPEG, PK - on ZIP. Team strings mystery_file | grep -iE 'ihdr|jfif|pk|%pdf' speeds up the search.
Step 2 – compare the first bytes with the reference signature from the table above.
Step 3 – correct damaged bytes in the hex editor. In HxD or hexedit go to the desired offset, enter the correct values, save.
A specific example is a file where file says “data” but strings shows IHDR:
До: 00 00 00 00 00 00 00 00 00 00 00 0D 49 48 44 52
После: 89 50 4E 47 0D 0A 1A 0A 00 00 00 0D 49 48 44 52
Eight zero bytes are replaced by a standard PNG signature. After saving the file opens in any viewer. On CTF CyberSkyline, a similar tack was solved in the same way: JPEG with the latest byte of signatures (0D instead of 01) did not open until it was brought to the standard FF D8 FF E0 00 10 4A 46 49 46 00 01.
For JPEG, the situation is similar: if there is a string inside the file JFIF – the first bytes should be FF D8 FF E0. For ZIP look for PK and restore the four-byte header 50 4B 03 04.
Automation of signature check in Python
When there are dozens of files, manual verification is impractical. Minimum script for determining file type by magic bytes:
import sys
SIGS = {b'\x89PNG\r\n\x1a\n': 'PNG', b'\xff\xd8\xff': 'JPEG',
b'%PDF': 'PDF', b'PK\x03\x04': 'ZIP', b'\x7fELF': 'ELF',
b'MZ': 'PE', b'Rar!\x1a\x07': 'RAR', b'SQLite format 3\x00': 'SQLite'}
with open(sys.argv[1], 'rb') as f:
header = f.read(16)
for sig, fmt in SIGS.items():
if header[:len(sig)] == sig: print(fmt); break
else: print(f'Unknown: {header[:8].hex(" ")}')
The script reads the first 16 bytes and compares with known signatures. If there are no matches, outputs hex as a starting point for manual analysis. Dictionary SIGS expands to the desired number of formats; in a serious project it is worth putting it in the JSON file.
How Attackers Mask File Type: Anti-Forensic by MITRE ATT&CK
In real forensics and on advanced CTFs, the files are not just corrupted – they are deliberately obfuscated. Several techniques from MITRE ATT&CK that are worth recognizing:
Masquerade File Type (T1036.008, Defense Evasion) – replacement extension or magic bytes. The PE file is disguised as a document. The definition of the file type by magic bytes exposes the substitution: inside the “picture” the title is detected 4D 5A. Beautiful phantom, and inside - exe'shnik.
Obfuscated Files or Information (T1027, Defense Evasion) – obfuscation of content. Encrypted/Encoded File (T1027.013): payload encoded by XOR or Base64, standard signatures do not detect. When carving, such files are missed - you need an analysis of entropy through binwalk -E.
Binary Padding (T1009, Defense Evasion) – adding trash cans to the file. Changes the hash and can shift the internal structure. Binwalk is doing: it searches for signatures throughout the file, not just in the first bytes.
When analyzing a suspicious file from an incident, check all three vectors sequentially: magic bytes, entropy, internal structure.
Practical checklist: from unknown file to result
file mystery_file – if the answer is not “data”, the format is defined.
binwalk mystery_file – if nested signatures are found, extract through binwalk -e.
xxd mystery_file | head -20 – compare the first bytes with the signature table.
strings mystery_file | grep -iE 'ihdr|jfif|pk|pdf' – if magic bytes are corrupted, look for characteristic lines inside the file.
Fix the header in the hex editor by setting the right magic bytes.
Check the result – open the file in the corresponding application or parser.
For disc images (.dd, .raw) steps 1-2 are replaced by foremost -i image.dd -o output/ – the tool itself will scan the entire image and spread the recovered files on the directories.
Most of the mid-level forensic CTFs are solved by this checklist in 5-10 minutes. Complex dumps add XOR-encryption of the title, custom formats or intentional fragmentation - but the basic approach remains the same: find a signature, understand the structure of the binary format, restore.
I often hear that for forensics-tasks it is enough to know three commands: file, binwalk -e and foremost. On the attachment tags, yes, it works. At the DEF CON or HackTheBox competition, no longer. The authors of the Tasks read the same tutorials and deliberately make files that break the standard pipeline: custom by magic, XOR of the first 16 bytes, swapped by CRC in PNG rattles. If there is no skill to open the hex-dump and read the structure of the binary format, the first non-standard task becomes a wall.
