Chapter 15: Malware Analysis

Chapter 15: Malware Analysis#

“To defeat malware, you must first understand it.” malware analysis doctrine

Learning Objectives#

After completing this chapter, you will be able to:

Classify malware by type and describe the behavior and impact of each category.
Set up a safe, isolated analysis environment.
Perform static analysis to extract strings, imports, and metadata without executing a sample.
Perform basic dynamic analysis by observing system and network behavior during execution.
Identify common malware persistence mechanisms and their detection signatures.
Describe common evasion and anti-analysis techniques used by modern malware.
Produce a structured malware analysis report.
Explain how threat intelligence and YARA rules capture malware knowledge.

Key Terms#

Malware: malicious software designed to damage, disrupt, or gain unauthorized access.
Virus: malware that attaches to a legitimate file and spreads when that file is executed.
Worm: self-replicating malware that spreads across networks without user action.
Trojan: malware disguised as a legitimate program.
Ransomware: malware that encrypts victim files and demands payment for the key.
RAT: Remote Access Trojan; provides the attacker with remote control.
Rootkit: malware that hides its presence from the operating system and security tools.
Botnet: a network of compromised machines (bots) under central attacker control.
Dropper / Loader: malware that downloads or decrypts additional payloads.
Static analysis: analyzing a sample without executing it.
Dynamic analysis: executing a sample in a controlled environment and observing its behavior.
Sandbox: an isolated virtual environment for safe malware execution.
YARA: a pattern-matching language for malware identification.
IOC: Indicator of Compromise; artefact useful for detection or threat hunting.

15.1 Malware Taxonomy#

Viruses and Worms#

Viruses require a host file and spread when the infected file is executed. File infectors, macro viruses (embedded in Office documents), and boot-sector viruses are subcategories. Viruses were the dominant threat category before ubiquitous internet connectivity.

Worms are self-contained and propagate autonomously across networks by exploiting vulnerabilities or default credentials. The Morris Worm (1988) demonstrated catastrophic worm propagation on the early internet; WannaCry (2017) used the EternalBlue exploit (CVE-2017-0144) to spread across unpatched Windows networks worldwide, encrypting files as it went.

Trojans and RATs#

Trojans masquerade as legitimate software: a cracked game, a fake codec, a malicious email attachment. The user executes the Trojan voluntarily. A RAT (Remote Access Trojan) is a Trojan that establishes a persistent, attacker-controlled remote session, giving full control over the infected host: keylogging, screen capture, camera access, file management, and command execution.

Ransomware#

Ransomware encrypts victim files and demands a ransom for the decryption key. Modern ransomware campaigns combine encryption with exfiltration (double extortion) and sometimes threats of DDoS against the victim (triple extortion). The ransomware-as-a-service (RaaS) model allows operators to license the ransomware and infrastructure, taking a percentage of ransom payments, enabling technically unsophisticated actors to run sophisticated campaigns.

Ransomware Attack Chain#

Initial access via phishing, exploitation, or compromised credentials.
Privilege escalation to domain administrator.
Lateral movement to spread across the network.
Data exfiltration before encryption (double extortion).
Disable backups and shadow copies.
Deploy and execute ransomware payload.
Ransom note delivered.

Rootkits#

Rootkits operate at a high privilege level (kernel, hypervisor, firmware) and hide their presence by intercepting and modifying OS calls. A kernel rootkit that hooks the process-listing syscall can make itself invisible to ps, tasklist, and most security tools. Detection requires inspection from a known-good context: a memory forensics tool loaded into a clean VM, or a bootable forensic toolkit that bypasses the compromised OS.

Botnets and C2#

A botnet is a collection of compromised hosts (bots or zombies) under centralized attacker control. The bots communicate with a Command and Control (C2) server to receive instructions: send spam, participate in DDoS, mine cryptocurrency, or propagate ransomware. Modern C2 frameworks use HTTPS to blend with legitimate traffic, fast-flux DNS to rotate C2 IP addresses rapidly, and domain generation algorithms (DGA) to create thousands of possible C2 domains, making blocking impractical without threat intelligence.

15.2 Analysis Environment Setup#

Safe Lab Requirements#

Malware analysis requires a completely isolated environment where the sample cannot reach the internet, infect the analyst’s machine, or escape to the network. Requirements:

Network isolation: no connection to production networks; analysis network firewalled to a controlled fake internet or completely air-gapped.
Snapshot capability: take a clean snapshot before analysis; revert after each sample.
Controlled internet simulation: INetSim or Remnux tools simulate DNS, HTTP, SMTP, and IRC to allow malware to behave normally without reaching the real internet.
Host-only networking: the VM can reach the analyst’s host but nothing beyond.

REMnux and FlareVM#

REMnux (Linux) and FlareVM (Windows) are curated distributions pre-loaded with malware analysis tools. REMnux includes Wireshark, Volatility, Ghidra, YARA, INetSim, and hundreds of Python analysis libraries. FlareVM adds Windows-native tools: x64dbg, IDA Free, PE Studio, PEiD, and the Sysinternals Suite.

15.3 Static Analysis#

File Identification#

Before executing a sample, identify it from metadata alone:

Magic bytes: the first bytes of a file identify its format. PE files start with MZ (0x4D5A); ELF files with \x7fELF; ZIP/JAR/DOCX with PK. Malware often uses misleading extensions.
Hash lookup: compute the SHA-256 hash and query VirusTotal, MalwareBazaar, and Hybrid Analysis. A known hash gives immediate classification and prior analysis.
File size and entropy: high entropy (close to 8 bits/byte) indicates encryption or packing. Legitimate executables have lower entropy in their code section.

PE Analysis#

PE (Portable Executable) is the Windows executable format. Key fields:

Imports (IAT): functions imported from DLLs. CreateRemoteThread + VirtualAllocEx + WriteProcessMemory suggests process injection. RegSetValueEx suggests persistence. CryptEncrypt + FindFirstFile suggests ransomware.
Sections: .text (code), .data (initialized data), .rsrc (resources). A PE with only one section, a packed name (.upx0), or mismatched virtual-size/raw-size is likely packed.
Exports: functions the PE exports. Malware DLLs may export a minimal API for the loader.
Strings: extractable ASCII and Unicode strings often include URLs, registry keys, mutex names, error messages, and command-and-control infrastructure.

YARA Rules#

YARA matches patterns in files to classify malware families:

rule RansomwareCandidateStrings {
    strings:
        $enc1 = "Your files have been encrypted" nocase
        $enc2 = ".onion" nocase
        $crypt = { 43 72 79 70 74 41 63 71 75 69 72 65 43 6F 6E 74 65 78 74 }
    condition:
        2 of them
}

YARA rules can match on strings, byte patterns, hex sequences, and regular expressions. The condition block combines matches with boolean logic.

Reverse Engineering Malware with Ghidra#

When strings and signatures are not enough, an analyst disassembles and decompiles the sample to read its logic directly. Always do this on an isolated analysis VM with no production network access (Section 15.2). A practical workflow follows.

Triage first. Before opening a disassembler, record the file hash, run file, list imports and sections (PE or ELF headers), and extract strings. Imports such as CryptAcquireContext, WinHttpOpen, or CreateRemoteThread already hint at encryption, network, or process-injection behavior.
Import into Ghidra. Create a project, import the binary, and let auto-analysis run. Ghidra disassembles the code and lifts it into its P-Code intermediate representation, which powers a built-in decompiler that shows readable C-like pseudocode beside the assembly.
Find the interesting code. Start from what you already know: double-click a suspicious string or an imported API in the Symbol Tree or Imports view to jump to every place it is used. Following CryptAcquireContext or a ransom-note string usually lands you in the encryption routine.
Read and annotate. In the Decompiler window, rename functions and variables (for example FUN_00401abc becomes encrypt_files), add comments, and define data types as you understand them. Good naming turns machine code into a readable narrative and is the heart of reverse engineering.
Trace behavior. Follow calls to map the program: configuration parsing, persistence (registry keys, scheduled tasks), command-and-control URLs, and the cryptographic scheme. Ghidra’s cross-references (“References to”) show how functions connect.
Confirm dynamically. Static reading tells you what the code can do; confirm it by detonating the sample in the sandbox (Section 15.4) or stepping through a debugger, watching the API calls and network traffic you predicted statically.

Choosing a Reverse-Engineering Tool#

Ghidra is a comprehensive, free option, but several decompilers each have distinct strengths:

Tool	Best for	Notes
Ghidra (NSA)	General cross-architecture analysis	Free and open source; P-Code IR; disassembler plus a native decompiler
IDA Pro (Hex-Rays)	Complex, commercial-grade analysis	Industry standard; polished pseudocode; Lumina signature matching; commercial
Binary Ninja	Scripting and workflow	Clean UI and a strong API; commercial, with a free cloud tier
JADX	Android applications	Lifts the Dalvik bytecode in APKs into readable Java source; open source
Cutter (rizin)	Lightweight free GUI	Open-source alternative; can use the Ghidra decompiler through a plugin

Match the tool to the target: a Windows PE or ARM firmware image suits Ghidra, IDA Pro, or Binary Ninja, while an Android APK is usually fastest to read in JADX. Reverse-engineer only samples you are authorized to analyze, and only inside an isolated lab.

15.4 Dynamic Analysis#

Behavioral Monitoring Tools#

During dynamic analysis, the analyst monitors:

Process activity: Process Monitor (ProcMon) logs file system, registry, and process/thread activity. New process spawning, registry Run key writes, and file creation in temp directories are suspicious.
Network activity: Wireshark captures all traffic; INetSim provides fake services for malware to interact with. DNS queries reveal C2 domains; HTTP requests reveal payloads.
API calls: API Monitor or a debugger traces every Windows API call the malware makes.

Common Malware Behaviors to Watch#

Behavior	What to observe	Significance
Persistence	Registry Run keys, Scheduled Tasks	Survival across reboots
Privilege escalation	UAC prompts, token impersonation	Gaining higher access
Process injection	CreateRemoteThread, WriteProcessMemory	Hiding in legitimate processes
C2 communication	Outbound HTTPS, DNS DGA queries	Command receipt, data exfiltration
Anti-VM check	CPUID, VMware registry keys, sleep	Evasion of sandbox analysis
Encryption	High CPU, file extensions changed	Ransomware payload execution

15.5 Anti-Analysis and Evasion Techniques#

VM and Sandbox Detection#

Many malware samples check whether they are running in a virtual machine or automated sandbox and behave differently (or do nothing) if detected. Techniques include: checking for VMware registry keys, querying CPUID for hypervisor bit, checking the number of running processes (sandboxes often have few), checking the disk size (sandboxes often have small disks), and sleeping for long periods to outlast automated analysis timeouts.

Packers and Obfuscators#

A packer compresses and encrypts the original payload, decrypting it at runtime. The static analysis sees only the decryption stub, not the real payload. Unpacking may require: running the sample until it unpacks itself (then dumping the decrypted PE from memory), using a known unpacker for common packers (UPX), or manual unpacking in a debugger.

Fileless Malware#

Fileless malware executes entirely in memory via legitimate system binaries (Living off the Land binaries, LOLBins): PowerShell, WMI, mshta, regsvr32, certutil. No file is written to disk, evading file-based detection. Detection requires memory forensics or behavioral monitoring of the LOLBin activity.

15.6 Malware Analysis Report Structure#

A structured analysis report includes:

Executive summary: one paragraph, family, capabilities, severity.
Sample metadata: file name, size, type, MD5/SHA-256, compile time.
Static analysis: imports, strings, sections, packer, YARA matches.
Dynamic analysis: persistence mechanisms, C2 communication, file activity.
IOCs: all extracted indicators (IPs, domains, hashes, file paths, registry keys).
MITRE ATT&CK mapping: technique IDs for each observed behavior.
Recommendations: detection rules (YARA, Sigma), remediation steps.

Worm / virus / Trojan: self-propagating, host-file-attached, and disguised malware.
Ransomware / RaaS / double extortion: file-encrypting extortion malware, sold as a service, that also leaks stolen data.
RAT / botnet / spyware / cryptominer / wiper / logic bomb: malware action categories.
Fileless malware: memory-resident malware abusing built-in tools to evade signatures.

15.7 Antivirus and Antimalware Defenses#

Having dissected how malware is built and analyzed, we turn to the defenses that try to stop it, the antivirus/antimalware layer. NIST’s Computer Security Resource Center defines malware as “a program that is inserted into a system, usually covertly, with the intent of compromising the confidentiality, integrity, or availability of the victim’s data, applications, or operating system, or otherwise annoying or disrupting the victim.” Malware is classified by how it propagates and what actions it performs, spanning attack kits, viruses, worms, and rootkits (the taxonomy of this chapter), and defenders face tens of thousands of new samples every day, which is why static signatures alone cannot keep up.

Antivirus technology has evolved through four generations: simple scanners that matched known signatures, heuristic engines that judged suspicious structure, activity/anomaly-based detection that watched behavior, and today’s next-generation (NextGen) AV that blends behavioral analytics and machine learning. Antimalware is best understood as the last line of defense: no single method is good enough by itself, NextGen capabilities are now necessary, and real-time scanning is essential rather than periodic sweeps.

Behaviorally, malware betrays itself through actions an antimalware engine can watch for: writing to restricted locations such as the registry or startup folders, modifying executables, opening/deleting/editing files in suspicious patterns, writing to the boot sector, and creating or injecting macros into documents. Two practical cautions close the topic: not all antivirus is created equal, so evaluate products before deploying, and while AV can be costly for an organization it remains a necessity, the final safety net beneath the firewalls, IDS/IPS, and detection methods of Chapters 11 and 12. The detection methods themselves (signature, heuristic, and anomaly/ML) are exactly those analyzed in Chapter 12, applied here at the endpoint.

Knowledge Check

According to NIST, what intent defines a program as malware?
List the four generations of antivirus technology in order.
Name three behaviors an antimalware engine watches for to identify malicious activity.

Answers: (1) Being inserted into a system, usually covertly, to compromise the confidentiality, integrity, or availability of the victim’s data, applications, or OS (or otherwise disrupt the victim). (2) Simple scanners, heuristics, activity/anomaly-based, and next-generation (NextGen) AV. (3) Any three of: writing to the registry or startup files, modifying executables, suspicious file open/delete/edit, writing to the boot sector, and creating/injecting document macros.

The Anti-* Family: Beyond Antivirus#

Antivirus is the best-known endpoint defense, but in practice it is one member of a family of complementary “anti-” controls, each tuned to a different threat, and a layered defense (defense-in-depth) runs several at once. Modern endpoint protection platforms (EPP) and endpoint detection and response (EDR) suites typically bundle most of them.

Antivirus / antimalware detect and remove malicious executables using the signature, heuristic, and anomaly methods of Chapter 12. The terms are now used almost interchangeably, “antimalware” simply emphasizing the broader modern scope (ransomware, trojans, worms, and more) beyond classic file viruses.
Anti-spyware targets software that covertly collects information, keyloggers, tracking cookies, info- stealers, and stalkerware, watching for the telltale behaviors of data capture and exfiltration rather than destruction.
Anti-phishing defends the human attack vector of Chapter 4: email and web filters that score messages and links, block known malicious and look-alike (typosquatted) domains, warn on credential-harvesting pages, and enforce sender-authentication checks (SPF, DKIM, DMARC). Because most intrusions begin with phishing, this control is disproportionately valuable.
Rootkit detectors hunt the hardest class of malware, rootkits that subvert the operating system (or below it) to hide their own presence (Chapter analysis of stealth malware). Because a kernel-level rootkit can lie to any tool running on the same system, detectors use techniques a compromised OS cannot easily fake: cross-view analysis (comparing what the OS reports against a lower-level enumeration to spot hidden files, processes, or registry keys), integrity checking against known-good baselines (Tripwire/AIDE-style), behavior monitoring, and offline or boot-time scanning from trusted media. Tools such as GMER, chkrootkit, and rkhunter illustrate the approach.

The unifying lesson is the precision-versus-recall trade-off seen throughout the book: every one of these tools balances false positives against false negatives, none is sufficient alone, and the strongest posture layers them with the network controls of Chapters 11 and 12, the detection methods that power them, and user awareness, so that what one layer misses another can catch.

15.8 A Field Guide to Malware Types#

The taxonomy section above introduced classification by propagation and action; here we name the families an analyst meets, because identifying the type guides both analysis and response. The classic distinction is by how it spreads: a virus attaches to a host file and runs when that file is opened; a worm spreads by itself across a network with no user action (the property that makes worms so explosive); and a Trojan masquerades as benign software the user installs willingly. Layered on top is what it does:

Ransomware encrypts the victim’s files (or whole disks) and demands payment, now usually with double extortion (also stealing data and threatening to leak it) and sold as Ransomware-as-a-Service (RaaS).
Rootkits subvert the operating system (or below it) to hide, the stealth class the rootkit detectors of this chapter target.
Remote Access Trojans (RATs) give an attacker interactive control of a host.
Botnets enlist many compromised machines under a command-and-control (C2) server for DDoS, spam, or fraud (Chapters 3 and 11).
Spyware/infostealers and keyloggers covertly harvest data and credentials.
Cryptominers steal compute to mine cryptocurrency (“cryptojacking”).
Wipers destroy data outright (sometimes disguised as ransomware), and logic bombs trigger on a condition.
Fileless malware lives only in memory and abuses built-in tools (PowerShell, WMI), leaving little on disk and defeating signature scanners, which is why memory forensics (Chapter 13) and behavior-based detection (Chapter 12) matter.

Most real malware is blended, a phishing Trojan that drops a RAT that deploys ransomware, so these labels describe capabilities a single sample often combines.

15.9 The Malware Lifecycle and a Ransomware Deep Dive#

Malware rarely acts in one step; it follows a lifecycle that mirrors the kill chain of Chapter 12 and the post-exploitation of Chapter 9: delivery (phishing, drive-by, or exploiting a public service), execution and installation, persistence, command and control, privilege escalation and lateral movement, and finally actions on objectives (encrypt, exfiltrate, destroy). Ransomware is the most consequential modern example and repays a closer look. A typical intrusion gains initial access (often through phishing or an exposed service, Chapter 8), escalates and spreads over hours, exfiltrates data first (for double extortion), disables backups and security tools, and only then detonates encryption across the estate. The encryption itself uses sound cryptography against the victim: a fast symmetric cipher (AES) per file with the file keys wrapped under the attacker’s public key (Chapter 2), so victims cannot recover files without the attacker’s private key.

        flowchart LR
    A[Phishing / exposed service] --> B[Execute + persist] --> C[Escalate + spread]
    C --> D[Exfiltrate data] --> E[Disable backups + tools] --> F[Encrypt everything] --> G[Ransom + leak threat]

The defenses follow directly from the lifecycle and from earlier chapters: phishing-resistant authentication and user training (Chapter 4), patching and least privilege to limit spread (Chapters 1 and 9), EDR/behavioral detection to catch the staging (Chapter 12), and, decisively, tested, offline/immutable backups so that encryption is a recoverable outage rather than a catastrophe (the durability and resiliency of Chapter 17). Paying ransoms is discouraged, funds the ecosystem, may be legally restricted, and does not guarantee recovery.

Notable Ransomware Strains: LockBit 3.0 and Rorschach#

Two strains illustrate how modern ransomware industrializes the lifecycle above. LockBit 3.0 (also called LockBit Black), which appeared in 2022, was one of the most prolific ransomware-as-a-service (RaaS) operations of the early 2020s. It ran an affiliate program, used AES with elliptic-curve key wrapping, added cryptocurrency options such as Zcash, and even advertised a “bug bounty” inviting researchers to report flaws in its own code. Its builder leaked in September 2022, letting other criminals create custom variants. In February 2024 an international law-enforcement action named Operation Cronos, led by the UK National Crime Agency and the U.S. FBI with partners, seized LockBit’s infrastructure, servers, and accounts and disrupted the operation, later naming its lead operator.

Rorschach (also tracked as BabLock), reported by Check Point Research in 2023, shows how fast and stealthy encryption has become. At disclosure it was the fastest encryptor publicly measured, achieved through partial (intermittent) encryption of each file, efficient multithreading, and a hybrid scheme combining the curve25519 key exchange with the HC-128 stream cipher. It was delivered by DLL side-loading that abused a digitally signed component of a legitimate security product to load its malicious loader, used direct system calls to evade monitoring, and could self-propagate across a Windows domain, skipping machines configured with Commonwealth of Independent States (CIS) languages.

Free Recovery: The No More Ransom Project#

Before paying anyone, victims should check whether the files can be recovered for free. No More Ransom (https://www.nomoreransom.org), launched in 2016 by Europol’s European Cybercrime Centre, the Dutch National Police, and cybersecurity companies, is a global initiative that hosts hundreds of free decryption tools. Its Crypto Sheriff tool lets a victim upload the ransom note or a sample encrypted file to identify the exact strain and, if a decryptor exists, download it at no cost. The portal also links victims to report the attack to law enforcement in their region, which aids investigations and supports the incident-response and breach-notification duties of Chapter 14. Checking No More Ransom is a standard early step in ransomware response, alongside isolating affected systems and restoring from tested, offline backups, and it reinforces the chapter’s central point: paying is a last resort, not a recovery plan.

15.10 Software Reverse Engineering in Depth#

The analysis techniques earlier in this chapter are applications of a broader discipline: software reverse engineering (SRE), the process of recovering a program’s design and behavior from its compiled form when no source code is available. SRE is used defensively to understand malware, find vulnerabilities, check for license or patent violations, and interoperate with closed formats, and offensively to develop exploits. This section consolidates the discipline, filling in the low-level foundations that the malware workflow assumes. Because reverse engineering is a laboratory skill, every subsection below names the tools and the hands-on exercise that builds it; all of it must be practiced only on software you own or are authorized to analyze, inside the isolated environment of Section 15.2.

The Reverse-Engineering Workflow#

Reverse engineering alternates between two complementary modes, introduced in Sections 15.3 and 15.4. Static analysis examines the binary without running it: identifying the file type and architecture, extracting strings and imports, and disassembling the machine code. Dynamic analysis runs the binary under control, watching its behavior in a debugger and a sandbox. Static analysis is complete (it sees all code paths) but defeated by packing and obfuscation; dynamic analysis sees the real behavior (including unpacked code in memory) but only the paths that actually execute. Skilled analysts iterate: static triage suggests where to set breakpoints, and dynamic execution reveals code that static tools could not reach, which is then re-examined statically.

An x86 and x86-64 Assembly Primer#

Disassemblers reconstruct assembly, so reading it is the core SRE skill. A CPU executes a stream of instructions that move data between registers and memory and perform arithmetic and control flow. On x86-64 the general-purpose registers are RAX, RBX, RCX, RDX, RSI, RDI, RBP, and RSP, plus R8 through R15; RIP is the instruction pointer (the address of the next instruction), and RFLAGS holds condition bits set by comparisons. Each 64-bit register has 32-, 16-, and 8-bit sub-registers (RAX, EAX, AX, AL), a frequent source of confusion when reading disassembly. The 32-bit x86 predecessor uses the same registers with an E prefix (EAX, ESP) and only eight of them.

Category	Representative instructions	Purpose
Data movement	`mov`, `lea`, `push`, `pop`	Load, store, and stack manipulation
Arithmetic and logic	`add`, `sub`, `xor`, `and`, `shl`	Computation; `xor reg, reg` is the idiomatic zeroing
Comparison	`cmp`, `test`	Set flags for a following conditional jump
Control flow	`jmp`, `je`, `jne`, `call`, `ret`	Unconditional and conditional branches, function calls

Two conventions must be internalized. First, syntax: Intel syntax writes mov dst, src while AT&T syntax writes mov src, dst with % and $ sigils; disassemblers can show either, and mixing them up inverts operands. Second, the calling convention, which fixes how arguments pass to functions. On 32-bit x86, cdecl and stdcall push arguments onto the stack (differing in who cleans up), while fastcall uses registers. On x86-64 there are two dominant conventions: the System V ABI (Linux and macOS) passes the first integer arguments in RDI, RSI, RDX, RCX, R8, and R9, whereas the Microsoft x64 convention passes them in RCX, RDX, R8, and R9. Both return the result in RAX. Recognizing the convention is what lets an analyst label the arguments to an API call in a disassembly. Lab: disassemble a small C program you compiled yourself and match each source statement to its assembly, then repeat with optimizations enabled to see how the compiler transforms it.

The PE Format and the Windows API#

Section 15.3 introduced the Portable Executable (PE) format; reverse engineering leans on it heavily. The PE headers describe how Windows loads the file: the sections (.text for code, .data and .rdata for data, .rsrc for resources), the entry point, and, critically, the Import Address Table (IAT), the list of functions the program imports from system DLLs. The IAT is the analyst’s fastest route to intent, because a binary importing CreateRemoteThread, VirtualAllocEx, InternetOpenUrl, or CryptEncrypt announces its capabilities before a single instruction is read. Programs interact with the operating system through the Windows API (the documented Win32 layer such as kernel32.dll and advapi32.dll) and, beneath it, the native API in ntdll.dll, whose thin wrappers issue the actual system calls; malware often calls the native API directly, or resolves API addresses dynamically with GetProcAddress, precisely to keep an empty, uninformative IAT. Tools: PE-bear, PEview, CFF Explorer, and Ghidra’s headers view.

Disassemblers, Decompilers, and Debuggers#

Three tool classes do the work. A disassembler translates machine code back to assembly; a decompiler goes further, reconstructing C-like pseudocode that is far faster to read. The open-source Ghidra (from the NSA) and the commercial IDA Pro both provide interactive disassembly and decompilation, cross-references, function graphs, and scripting; Binary Ninja and radare2 (with its Cutter front end) are common alternatives. A debugger runs the binary and lets the analyst pause at breakpoints, step one instruction at a time, inspect registers and memory, and modify state; x64dbg and OllyDbg are standard on Windows, WinDbg for kernel work, and GDB on Linux. The Ghidra walkthrough at the end of this chapter is the recommended starting lab.

Code Injection: DLL Injection and Process Injection#

A recurring reverse-engineering target is code that runs inside another process, both to hide and to gain the victim process’s privileges and trust. The classic technique is DLL injection: the attacker opens a handle to the target with OpenProcess, allocates memory inside it with VirtualAllocEx, writes the path of a malicious DLL there with WriteProcessMemory, and then calls CreateRemoteThread (or QueueUserAPC) with the address of LoadLibrary, so the target itself loads the DLL. Reflective DLL injection removes the disk artifact by mapping the DLL from memory with a custom loader, leaving nothing for LoadLibrary to open. Related methods include SetWindowsHookEx (injecting via a message hook), the AppInit_DLLs registry value, and APC injection (queuing an asynchronous procedure call to a thread). A distinct family replaces a process’s contents rather than adding to them: process hollowing (also called RunPE) starts a legitimate process in a suspended state, unmaps its original image, writes a malicious image in its place, rewrites the thread’s entry point, and resumes it, so a trusted-looking process such as svchost.exe is actually running attacker code. These covert launching methods are why analysts correlate process trees, memory-resident code, and API-call sequences (Section 15.4) rather than trusting a process name.

Obfuscation and Deobfuscation#

Authors obscure code to slow analysis. Obfuscation transforms a program to preserve behavior while destroying readability: renaming and control-flow flattening, inserting junk code, encrypting strings and decrypting them at runtime, and virtualization obfuscation that compiles the real logic to a custom bytecode run by an embedded interpreter. Deobfuscation reverses this. String decryption is often recovered by locating the decryption routine and either scripting it or letting the program decrypt the strings under a debugger and dumping them; control-flow flattening is undone by reconstructing the original dispatch order, sometimes with symbolic-execution tools; and constant folding and dead-code elimination clean up junk. The general principle is that dynamic analysis defeats most obfuscation, because the code must eventually deobfuscate itself in memory to run, and that is the moment to capture it. Lab: recover the plaintext strings from a sample that decrypts them at runtime, first statically by reimplementing the routine, then dynamically by breakpointing after the decrypt call.

Anti-Analysis: Anti-Disassembly, Anti-Debugging, and Anti-VM#

Malware actively resists the tools above, and recognizing these tricks is a core skill.

Anti-disassembly exploits the fact that linear disassemblers decode instructions sequentially. Inserting a byte that begins a multi-byte instruction, then jumping past it, desynchronizes the disassembler so it misreads the following real instructions; opaque predicates (a conditional that is always true or always false but not obviously so) and overlapping instructions have the same effect. Recursive-descent disassemblers such as those in Ghidra and IDA, which follow control flow rather than reading linearly, resist these, and forcing the tool to re-analyze from the true instruction boundary fixes the display.

Anti-debugging detects or disrupts a debugger. Direct checks include the IsDebuggerPresent and CheckRemoteDebuggerPresent APIs and reading the BeingDebugged byte or NtGlobalFlag in the Process Environment Block; timing checks use RDTSC or GetTickCount to notice the delay that single-stepping introduces; and structural tricks scan for software breakpoints (the 0xCC byte) or abuse the trap flag and hardware debug registers. Analysts defeat these by patching the check to always report “no debugger,” using debugger plugins such as ScyllaHide that hide these indicators, or stepping around the check dynamically.

Anti-VM and sandbox evasion detects an analysis environment: the CPUID hypervisor-present bit, VMware or VirtualBox registry keys, driver files and MAC-address prefixes, low resource counts, absence of user activity, or long sleeps that outlast an automated sandbox’s timeout. Countermeasures are to harden the analysis VM so it looks like a real host, and to fast-forward or patch out sleeps.

Packing and Unpacking#

A packer compresses or encrypts a program and prepends a small stub that reconstructs the original code in memory at runtime, which both shrinks the file and hides its contents from static analysis; UPX is the common benign example, and many families use custom or commercial protectors. A packed PE betrays itself through high entropy in its code section, very few imports, unusual section names (.upx0), and a large gap between a section’s virtual and raw sizes. Unpacking means letting the stub run until it transfers control to the reconstructed code, at the original entry point (OEP), then dumping the process image from memory and rebuilding its import table with a tool such as Scylla. UPX unpacks with upx -d; custom packers require the manual run-to-OEP-and-dump approach under a debugger. Unpacking is usually the first step before any real static analysis of a protected sample.

Shellcode, Buffer Overflows, and Network Attacks#

Reverse engineering connects directly to exploitation. Shellcode is a compact, position-independent payload written to run once control is hijacked; because it cannot rely on a normal loader, it resolves the APIs it needs at runtime by walking the loaded-module list in the PEB, a pattern the analyst learns to recognize. Shellcode is typically delivered by a buffer overflow or a related memory-corruption bug, and Chapter 9 develops the mechanics, from stack smashing to return-oriented programming, along with the mitigations (ASLR, DEP, stack canaries) that shape modern exploits. Reverse engineers also analyze a sample’s network behavior, capturing its traffic (Chapter 3) to recover command-and-control protocols, domain-generation algorithms, and the ARP-spoofing, man-in-the-middle, or denial-of-service modules it may carry, then writing detection signatures (Chapter 12) from what they find. Reverse engineering is thus the connective tissue of this book: it turns an opaque binary into the concrete indicators, exploit understanding, and defenses that the surrounding chapters put to work.

Exercises#

Given a disassembly that begins mov rcx, ...; mov rdx, ...; call CreateFileW, which calling convention is in use and what are the first two arguments to the call?
A PE has one section named .upx1, an entropy of 7.9 in its code section, and only three imports (LoadLibraryA, GetProcAddress, VirtualAlloc). What do you conclude, and what is your next step?
Explain why dynamic analysis often recovers obfuscated strings that static analysis cannot, and name the single best moment to capture them.
Name three ways a program can detect that it is running under a debugger, and one way an analyst defeats each.
Put these DLL-injection API calls in order and state what each does: CreateRemoteThread, OpenProcess, WriteProcessMemory, VirtualAllocEx.
What distinguishes process hollowing from ordinary DLL injection, and why does it evade name-based detection?

Answer Key#

The Microsoft x64 convention (arguments in RCX, RDX, R8, R9); the first two arguments are the values placed in RCX and RDX, here the file name pointer and the desired access.
The sample is packed with UPX (packed section name, near-maximal entropy, and an import table reduced to the loader primitives). Unpack it first, with upx -d or by running to the original entry point and dumping, before attempting static analysis.
The code must deobfuscate itself in memory before it can run, so the plaintext exists at runtime even when it never appears in the file; the best capture point is immediately after the decryption routine returns.
Detection: IsDebuggerPresent/PEB BeingDebugged, timing via RDTSC, and scanning for the 0xCC breakpoint byte. Defeats: patch the check to return false, use a hiding plugin such as ScyllaHide, and set hardware rather than software breakpoints (or step past the scan).
OpenProcess (get a handle to the target), VirtualAllocEx (allocate memory in it), WriteProcessMemory (write the DLL path), CreateRemoteThread (start a thread at LoadLibrary to load the DLL).
Hollowing replaces the entire image of a legitimate, suspended process with malicious code rather than adding a DLL, so a trusted process name such as svchost.exe runs attacker code, defeating any detection that trusts the process name.

15.11 An End-to-End Analysis Walkthrough#

The techniques of Sections 15.3 through 15.10 come together in a single ordered workflow. Given an unknown sample in the isolated lab of Section 15.2, an analyst proceeds from cheapest to most expensive, stopping as soon as the question is answered.

        flowchart TB
    A[Triage: hash, file type, VirusTotal] --> B[Static: strings, imports, PE sections, entropy]
    B --> C{Packed?}
    C -->|yes| U[Unpack: run to OEP, dump]
    C -->|no| D[Disassemble / decompile in Ghidra]
    U --> D
    D --> E[Dynamic: run in sandbox, watch API, files, registry, network]
    E --> F[Extract IOCs and write YARA + network signatures]
    F --> G[Report: capabilities, IOCs, severity]

The steps are: compute a hash and identify the file type (file, TrID), and check the hash against threat intelligence without uploading the sample if it is sensitive; run static triage for strings, imports, section names, and entropy (Section 15.3), which reveals capability and whether the sample is packed; if packed, unpack to the original entry point and dump (Section 15.10); disassemble and decompile in Ghidra to understand the logic; then detonate in a sandbox (Section 15.4) to observe the real behavior that static analysis cannot reach, especially for obfuscated or fileless samples. Finally, extract indicators of compromise and turn them into detections. The discipline is to let each cheap step guide the expensive one.

15.12 Extracting Indicators of Compromise#

The deliverable of malware analysis is not a story but detections: the concrete indicators of compromise (IOCs) that let the rest of the organization find and block the threat (Chapter 12). IOCs include file hashes, embedded IP addresses and domains (command-and-control), URLs, mutex names, registry keys, and file paths. The lab below pulls the common network and hash indicators out of a sample’s raw bytes, the first automated step of report generation.

# Extract simple IOCs from a (benign, synthetic) sample's bytes: hashes, IPv4, URLs, domains.
import hashlib, re

sample = (b"MZ......this is a stand-in for a binary......"
          b"http://malicious.example/gate.php  "
          b"C2 host: 203.0.113.66 backup 198.51.100.9 "
          b"mutex: Global\\r4nS0m_lock  ")

def extract_iocs(data: bytes):
    text = data.decode("latin-1")
    ipv4 = re.findall(r"\b(?:\d{1,3}\.){3}\d{1,3}\b", text)
    urls = re.findall(r"https?://[^\s\"']+", text)
    domains = re.findall(r"\b(?:[a-z0-9-]+\.)+[a-z]{2,}\b", text, re.I)
    return {
        "md5":    hashlib.md5(data).hexdigest(),
        "sha256": hashlib.sha256(data).hexdigest(),
        "ipv4":   sorted(set(ipv4)),
        "urls":   sorted(set(urls)),
        "domains": sorted(set(d for d in domains if not re.match(r"\d+\.\d+", d))),
    }

for key, val in extract_iocs(sample).items():
    print(f"{key:8}: {val}")

Raw extraction over-collects (harmless strings look like domains), so an analyst validates candidates against context and threat intelligence before publishing them. The validated indicators become a YARA rule for files (Section 12.7), Snort or Suricata rules for the network callbacks, and Sigma rules for the host behaviors, closing the loop from a single analyzed sample to organization-wide detection. Because attackers change hashes and domains cheaply but change behavior rarely, the most durable indicators are behavioral, the API-call sequences and parent-child process relationships of Sections 15.4 and 13.13, which is the reasoning behind the pyramid of pain: hashes are trivial for an attacker to alter, while tactics, techniques, and procedures are costly, so detections built on behavior age far better than detections built on literals.

Exercises#

Order these analysis steps from first to last and justify the order: decompile in Ghidra, compute a hash, detonate in a sandbox, extract strings.
A sample’s import table contains only LoadLibraryA, GetProcAddress, and VirtualAlloc, and its .text section has entropy 7.9. What do you conclude and what do you do next?
Why are behavioral indicators considered more durable than file-hash indicators?
Give one file-level, one network-level, and one host-log-level detection you would produce from a sample that beacons to bad.example over HTTPS and drops C:\Users\Public\svc.exe.

Answer Key#

Compute a hash (cheapest, enables intel lookup), extract strings (static triage), detonate in a sandbox (observe real behavior), decompile in Ghidra (deepest, most expensive); cheap steps scope and guide the expensive ones.
The sample is packed (loader-only imports plus near-maximal entropy); unpack it to the original entry point and dump before attempting real static analysis (Section 15.10).
Attackers change hashes, domains, and IPs cheaply, but changing behavior (tactics and techniques) is costly, so behavioral detections keep working across samples and campaigns.
File: a YARA rule matching svc.exe’s distinctive strings or code; network: a Suricata rule or DNS/TLS-SNI block for bad.example; host log: a Sigma rule for a process writing and then executing from C:\Users\Public\.

15.13 Ransomware Internals and the Criminal Economy#

Ransomware deserves a closer look because it dominates the modern threat landscape and because its internals are a compact lesson in applied cryptography (Chapter 2). Effective ransomware uses hybrid encryption: it generates a random symmetric key (AES) for each victim, encrypts the files with it for speed, then encrypts that symmetric key with the attacker’s public key (RSA or elliptic-curve). Only the attacker’s private key can recover the symmetric key, so victims cannot decrypt even by reverse-engineering the sample. Poorly built families break this in instructive ways: reusing a key across victims, using a predictable random-number generator (Chapter 2), or leaving the key in memory (recoverable by the memory forensics of Section 13.13), which is exactly how free decryptors on the No More Ransom project come to exist.

The behavior is as important as the cryptography. Before encrypting, modern ransomware deletes volume shadow copies and backups to prevent recovery, and most now practice double extortion: it exfiltrates data first, then encrypts, so victims who restore from backup are still coerced by the threat to publish. The delivery is increasingly human-operated, an intrusion (Chapter 9) that spends days conducting reconnaissance and lateral movement before detonating across the domain at once. Underpinning all of this is ransomware-as-a-service (RaaS), a criminal business model in which operators lease the malware and infrastructure to affiliates for a cut of the proceeds, which is why the same family appears in unrelated attacks and why disrupting one operator (as with the LockBit and law-enforcement actions of this chapter’s News in Focus) ripples across many campaigns.

15.14 Rootkits, Bootkits, and Firmware Malware#

The most evasive malware hides by subverting the layer that is supposed to observe it. A rootkit operates at a privileged level to conceal files, processes, and connections from the operating system and from security tools; a user-mode rootkit hooks library calls, while a far stealthier kernel-mode rootkit manipulates kernel structures directly, so that tools relying on the kernel’s own answers are blind. A bootkit goes lower still, infecting the boot process (the Master Boot Record or the UEFI firmware) so it loads before the operating system and its defenses exist, persisting even across a full disk reinstall. Firmware implants in the UEFI or in device controllers are the deepest and hardest to remediate. The defenses are architectural: Secure Boot and a measured boot chain anchored in a TPM (Section 11.15) verify each stage before running it, and detection increasingly relies on out-of-band checks and memory forensics rather than on asking a possibly-subverted operating system. The lesson is the trust-anchor principle: a monitor can only be trusted if it sits below what it monitors, which is why hardware roots of trust matter.

15.15 Mobile and Cross-Platform Malware#

Malware follows users onto phones. Android, with its open app model, sees the most: trojanized apps (especially from outside the official store), banking trojans that abuse the accessibility service to overlay fake login screens and steal one-time codes, and stalkerware. iOS is harder to infect because of its locked-down model, so attacks against it concentrate on high-value targets through expensive zero-click exploit chains (the mercenary-spyware pattern of NSO Group’s Pegasus, which reached targets with no user interaction). Across both platforms the recurring risks are excessive permission grants, SMS and messaging phishing (smishing, Chapter 4) as the delivery vector, and supply-chain compromise of legitimate apps and SDKs. The defensive story mirrors the desktop: keep the platform patched, install only from trusted sources, review permissions, and, for high-risk users, adopt the hardened lockdown modes the platforms now provide.

Exercises#

Explain, using hybrid encryption, why a victim usually cannot decrypt ransomware even after fully reverse-engineering the sample, and name two implementation mistakes that have produced free decryptors.
What is double extortion, and why does it defeat a backup-only recovery strategy?
Why can a security tool that runs as an ordinary program fail to detect a kernel-mode rootkit, and what architectural defense addresses this?
Account for the difference in malware prevalence between Android and iOS, and name the delivery vector common to both.

Answer Key#

Files are encrypted with a per-victim AES key that is itself encrypted with the attacker’s public key, so recovery needs the attacker’s private key, which is not in the sample; free decryptors arise when a family reuses keys, uses a weak or predictable random-number generator, or leaves the key recoverable in memory.
The attacker exfiltrates data before encrypting and threatens to publish it, so restoring from backup removes the encryption leverage but not the threat of disclosure.
A kernel-mode rootkit manipulates the kernel structures a user-mode tool queries, so the tool receives falsified answers; the defense is a trust anchor below the malware, such as Secure Boot and TPM-measured boot, plus out-of-band or memory-based verification.
Android’s open installation model and sideloading permit far more trojanized apps, while iOS’s locked-down model forces costly zero-click chains against select targets; SMS and messaging phishing (smishing) is the common delivery vector.

15.16 Fileless and Living-off-the-Land Malware#

Modern malware increasingly avoids writing files to disk, because files are what antivirus and forensics find. Fileless malware runs entirely in memory, injected into a legitimate process (Section 15.10) or executed from a script that never touches the file system, so a disk scan finds nothing and only the memory forensics of Section 13.13 records the truth. Closely related is the living-off-the-land technique of abusing tools that are already present and trusted on the system, the LOLBins, so that malicious activity is indistinguishable at a glance from routine administration. PowerShell downloads and executes code in memory; certutil and bitsadmin fetch payloads; wmic and regsvr32 execute code; scheduled tasks and WMI subscriptions persist without a dropped binary. Because there is no malicious file to signature, detection shifts entirely to behavior (Chapter 12): the parent-child relationship of an Office app spawning PowerShell, an encoded command line, a script reaching out to the internet, all of which are the host-telemetry detections of Section 12.10. Fileless and living-off-the-land techniques are the reason the industry moved from file-scanning antivirus to behavior-based EDR.

15.17 Worms, Botnets, and Supply-Chain Malware#

Malware that spreads on its own or through trusted channels causes the largest incidents. A worm propagates without user action by exploiting a vulnerability, as WannaCry did through the EternalBlue SMB flaw (this chapter’s News in Focus) and as NotPetya did to cause billions in damage; the defense is the patch and segmentation discipline of Sections 11.16 and 11.18, because a worm’s blast radius is set by how flat and unpatched the network is. A botnet is a network of compromised machines under a common command-and-control (Section 9.20), rented out for distributed denial-of-service (the Mirai attack of Chapter 3), spam, and credential stuffing; taking down a botnet is as much a legal and coordination problem (the No More Ransom and law-enforcement actions of this chapter) as a technical one. Supply-chain malware is the most insidious, compromising trusted software before it reaches the victim: the SolarWinds intrusion (Chapter 1) inserted a backdoor into a signed software update that thousands of organizations installed, and the XZ Utils backdoor nearly planted a covert door into a core open-source library used across the internet. These defeat the instinct to trust signed, official software, which is why software bill-of-materials, build-integrity verification, and behavioral monitoring even of trusted vendors (Section 5.12) have become essential.

15.18 Threat Intelligence and Malware Families#

Analysis produces its full value only when connected to the wider landscape. Threat intelligence tracks malware families (related samples sharing code and behavior, such as Emotet, TrickBot, and LockBit) and the threat actors who use them, so a newly analyzed sample can be matched to a known family, and its likely next actions anticipated during an incident (Section 14.15). Naming is famously inconsistent, the same family carries different names across vendors, which is why analysts anchor on behavior and code overlap rather than labels, and organize detections on the pyramid of pain of Section 15.12. The practical loop is that this chapter’s analysis feeds Chapter 12’s detections and Chapter 14’s response: a sample is analyzed, its family and indicators identified, detections written, and the intelligence shared (through frameworks such as MITRE ATT&CK and information-sharing communities) so that one organization’s analysis protects many. Malware analysis is therefore not an end in itself but the engine that supplies the indicators, detections, and context the rest of a security program runs on.

Exercises#

Explain why fileless malware defeats traditional file-scanning antivirus, and name the detection approach and forensic technique that still catch it.
Give two examples of living-off-the-land binaries and state what makes their malicious use hard to detect.
Contrast a worm and a botnet by how each spreads and what each is used for, and name the control that most limits a worm’s blast radius.
Why is supply-chain malware especially dangerous, and name two defenses that address it.

Answer Key#

It runs in memory with no file to scan; behavior-based EDR (Section 12.10) detects the malicious process activity, and memory forensics (Section 13.13) recovers the in-memory code and indicators.
PowerShell, certutil, bitsadmin, wmic, or regsvr32 (any two); they are legitimate, trusted, pre-installed tools, so their use blends with normal administration and produces no malicious file to signature.
A worm self-propagates by exploiting a vulnerability with no user action, used to spread payloads rapidly; a botnet is a set of compromised hosts under common command-and-control, rented for DDoS, spam, and credential stuffing; patching and network segmentation most limit a worm’s spread.
It arrives through trusted, often signed, software so victims install it willingly and widely; defenses include software bill-of-materials and build-integrity verification, plus behavioral monitoring even of trusted vendor software.

15.19 Case Study: Dissecting a Ransomware Sample#

The techniques of this chapter come together on a single sample. Follow the analysis of an unknown Windows executable delivered by phishing, worked in the isolated lab of Section 15.2, applying the workflow of Section 15.11.

Triage. The analyst computes the file’s hashes and checks threat intelligence: the SHA-256 is unknown, but file and the DOS header confirm a 64-bit PE. First impressions matter, and an unknown hash means the sample must be analyzed rather than merely looked up.

Static triage. Strings reveal little plaintext, and the import table lists only LoadLibraryA, GetProcAddress, and VirtualAlloc; the .text section shows entropy near 7.9. The verdict is immediate (Section 15.10): the sample is packed. The analyst unpacks it by running to the original entry point in a debugger and dumping the reconstructed image, then rebuilding the imports.

Static analysis of the unpacked code. Now the strings and imports are meaningful. The analyst sees CryptAcquireContext, CryptGenRandom, and CryptEncrypt (a cryptographic payload), vssadmin, FindFirstFile/FindNextFile (file enumeration), and a hardcoded HTTPS URL. Ghidra’s decompiler shows the logic: enumerate files, generate a random AES key, encrypt each file, then encrypt the AES key with an embedded RSA public key, the hybrid-encryption ransomware of Section 15.13.

Dynamic analysis. Detonated in the sandbox with monitoring (Section 15.4), the sample deletes volume shadow copies (vssadmin delete shadows), begins renaming files with a new extension, contacts the hardcoded URL to report the infection, and drops a ransom note. The behavioral trace confirms and enriches the static findings, and captures the network indicator.

IOC extraction and reporting. The analyst extracts the indicators (Section 15.12): the file hashes, the C2 URL and its resolved address, the new file extension and ransom-note filename, and the mutex the sample creates to avoid double-infection. These become a YARA rule for the file, a Suricata rule and DNS block for the callback, and Sigma rules for the shadow-copy deletion and mass-rename behavior (Section 12.7), which are handed to the SOC (Chapter 12) and the incident-response team (Chapter 14). The report (Section 15.6) states the capability (hybrid-encryption ransomware with shadow-copy deletion and C2 check-in), the severity, and the full indicator set.

What the analysis enabled. Because the sample used a per-victim AES key wrapped with the attacker’s RSA public key, decryption without the attacker’s private key is infeasible (Section 15.13), so the response prioritizes containment and backup restoration over decryption (Chapter 14). Had the family reused a key or leaked it in memory, the memory forensics of Section 13.13 might have recovered it, the exact circumstance behind the free decryptors on the No More Ransom project. This single analysis thus drove detection, response, and recovery decisions across three chapters, which is the entire point of malware analysis.

Exercises#

During static triage the sample has three loader-only imports and code-section entropy of 7.9. State the conclusion and the required next step before further static analysis.
From the unpacked imports (CryptGenRandom, CryptEncrypt, vssadmin, FindFirstFile), infer the sample’s capability before running it.
The sample encrypts each victim’s files with a random AES key wrapped by an embedded RSA public key. Explain why this prevents decryption from the sample alone, and the one circumstance that could still yield a decryptor.
List the four detections you would produce from this sample and the data source each targets.

Answer Key#

The sample is packed (loader-only imports plus near-maximal entropy); unpack it by running to the original entry point and dumping, and rebuild the imports, before any real static analysis.
It is ransomware: it enumerates files (FindFirstFile), generates keys and encrypts (CryptGenRandom, CryptEncrypt), and deletes backups (vssadmin) to prevent recovery.
Files are encrypted with a per-victim AES key that is itself encrypted with the attacker’s RSA public key, so recovery needs the attacker’s private key, which is not in the sample; a decryptor is possible only if the family reused keys, used a weak random-number generator, or left the key recoverable in memory.
A YARA rule for the file (on-disk), a Suricata rule and DNS block for the C2 callback (network), and Sigma rules for shadow-copy deletion and mass file rename (host logs/EDR).

15.20 A YARA Rule Cookbook#

YARA (Section 12.7) is the analyst’s primary tool for turning one analyzed sample into detection for a whole family, so writing good rules is a core skill. The recipes below illustrate the main techniques, from literal strings to structural conditions.

// 1. Detect by distinctive strings (simple, but brittle to string changes)
rule Ransom_Note_Generic {
    strings:
        $a = "Your files have been encrypted" nocase
        $b = ".onion"                      // Tor payment site
        $c = "bitcoin" nocase
    condition:
        2 of them
}

// 2. Detect a packer/loader by capability rather than exact strings
rule Loader_Behavior {
    strings:
        $api1 = "VirtualAlloc"
        $api2 = "LoadLibraryA"
        $api3 = "GetProcAddress"
    condition:
        all of them and filesize < 200KB
}

// 3. Use the PE module for structural detection (robust to string edits)
import "pe"
rule High_Entropy_Packed_PE {
    condition:
        pe.is_pe and
        for any section in pe.sections : ( math.entropy(section.raw_data_offset, section.raw_data_size) > 7.5 )
}

// 4. Detect a specific family by a byte pattern in its code (hex, with wildcards)
rule Family_X_Stub {
    strings:
        $decoder = { 8B 45 ?? 33 45 ?? 89 45 ?? EB }   // ?? = any byte
    condition:
        $decoder
}

The craft is balancing sensitivity against specificity: literal-string rules are easy to write but an attacker changes strings cheaply (the base of the pyramid of pain, Section 15.12), while rules on code structure, byte patterns, and PE features are harder to evade but require deeper analysis. Good rules also include meta fields (author, date, reference) and are tested against a corpus of known-good files to avoid false positives, the same detection-engineering discipline as Chapter 12. A single well-crafted YARA rule can detect an entire malware family across an organization’s files and memory, which is the leverage that makes malware analysis worthwhile.

15.21 Building and Operating an Analysis Lab#

Everything in this chapter depends on a safe place to work, so a malware-analysis course begins by building one. The requirements are isolation and reversibility. Analysis runs in virtual machines that are snapshotted before each detonation and reverted afterward, so a sample cannot persist between runs. The lab network is isolated from production and the internet, or routed through a controlled fake-internet service (INetSim or FakeNet-NG) that answers the malware’s DNS, HTTP, and other requests so its behavior can be observed without letting it reach real command-and-control or spread. The analysis VM is provisioned with the toolset of Sections 15.3, 15.4, and 15.10: a disassembler and decompiler (Ghidra), a debugger (x64dbg), PE and string tools, a network sniffer (Wireshark), and behavioral monitors (Procmon, API Monitor). Two cautions define professional practice. First, sophisticated malware detects virtual machines and sandboxes (Section 15.10) and behaves benignly to evade analysis, so a hardened, realistic-looking VM and, for stubborn samples, bare-metal analysis may be needed. Second, some samples are worms or have destructive payloads, so the isolation is not optional hygiene but the barrier between analyzing malware and releasing it. The distribution known as REMnux (Linux) and the FLARE VM toolset (Windows) package these tools and are the standard starting points. With this environment in place, the workflow of Section 15.11 can be run safely on real samples, which is where the discipline is actually learned.

Exercises#

Contrast a YARA rule that matches on literal strings with one that matches on PE section entropy, in terms of how easily an attacker evades each.
Why should a YARA rule be tested against a corpus of known-good files before deployment?
Give the two core requirements of a malware-analysis lab and the specific mechanism that provides each.
Why might a hardened or bare-metal environment be necessary despite the convenience of a snapshotting VM?

Answer Key#

A literal-string rule is trivially evaded by changing the strings (bottom of the pyramid of pain); an entropy or structural rule keys on a property the attacker cannot change without altering how the malware is built, so it is far harder to evade.
To measure and minimize false positives; a rule that also matches benign files would flood the SOC with noise and could quarantine legitimate software, so it must be validated against known-good data first.
Isolation (an air-gapped or fake-internet lab network via INetSim/FakeNet so the sample cannot reach real infrastructure or spread) and reversibility (VM snapshots reverted after each detonation so nothing persists).
Advanced malware detects virtualization and sandboxes and behaves benignly to evade analysis, so a hardened, realistic VM or bare-metal analysis is needed to observe the true behavior.

15.22 Reading x86-64 Disassembly#

Static analysis (Section 15.3) and the reverse-engineering course of Section 15.10 both depend on one skill: looking at assembly and recovering what the original code did. This is learnable because compilers translate high-level constructs into recurring assembly patterns. Once you know the patterns, disassembly reads almost like source. The register and calling-convention background is in Section 9.27; here we map source constructs to what you will actually see.

Variables and arithmetic. Local variables live in registers or at fixed offsets from the frame pointer. x = a + b compiles to loads and an add:

mov  eax, DWORD PTR [rbp-0x4]   ; eax = a
add  eax, DWORD PTR [rbp-0x8]   ; eax = a + b
mov  DWORD PTR [rbp-0xc], eax   ; x = eax

Conditionals. An if becomes a comparison followed by a conditional jump. cmp sets flags; jcc (such as je, jne, jg, jle) branches on them:

cmp  DWORD PTR [rbp-0x4], 0xa   ; compare x with 10
jle  .else_branch              ; if x <= 10, jump
...                            ; then-branch (x > 10)
.else_branch:
...

Loops. A for or while is a body plus a compare-and-jump-back at the bottom. Recognizing the backward jump target is how you spot a loop in an unfamiliar function.

Function calls. Arguments are loaded into rdi, rsi, rdx, ... (System V) or rcx, rdx, r8, r9 (Windows), then call transfers control and the result comes back in rax. This pattern is how an analyst reads an API call in malware:

mov  rdi, rax                  ; 1st arg: handle
lea  rsi, [rip+0x2f10]         ; 2nd arg: pointer to a string constant
mov  edx, 0x104                ; 3rd arg: length
call GetModuleFileNameA        ; Windows API call

Even without a decompiler you can read that as “call GetModuleFileNameA(handle, buffer, 0x104),” which tells you the sample is finding its own path on disk. The strings loaded by lea reg, [rip+disp] are how you connect code to the string and import artifacts of Sections 15.3 and 15.4.

Structures and arrays. Field and element access appears as base-plus-offset or base-plus-index-times-scale addressing: [rax+0x8] reads the field at offset 8 of the object in rax; [rax+rcx*4] reads element rcx of an array of 4-byte items. A chain of [reg+offset] dereferences is often a walk through a linked structure, which is exactly what shellcode does when it walks the PEB to resolve APIs (Section 9.25).

Why the decompiler is not enough. Ghidra and IDA produce C-like pseudocode that speeds this up enormously, but decompilers guess, and malware deliberately confuses them (Section 15.23). The analyst who can drop to the disassembly and confirm what the bytes really do is the one who does not get fooled, which is why reading assembly remains the load-bearing skill of the discipline.

Exercises#

What assembly pattern marks an if statement, and which instruction actually performs the branch?
Given mov rdi, rax then lea rsi, [rip+0x2f10] then call SomeApi, what are the first two arguments to the call on a Linux target?
How does the addressing form [rax+rcx*4] correspond to a source-level construct?
Why should an analyst be able to read raw disassembly even when a decompiler is available?

Answer Key#

A cmp (or test) that sets flags, followed by a conditional jump (jcc such as je, jne, jg); the jcc performs the branch.
rax (moved into rdi) is the first argument, and the string constant at [rip+0x2f10] (loaded into rsi) is the second argument.
It is indexed array access: base rax plus index rcx scaled by 4, that is, element rcx of an array of 4-byte elements.
Decompilers infer and can be deliberately misled by anti-analysis tricks; dropping to the disassembly lets the analyst confirm the actual behavior of the bytes and avoid being deceived.

15.23 Windows Internals for Malware Analysts#

Most malware targets Windows, so analysis requires a working model of the operating system it abuses. This section gives the minimum internals needed to read the behavior of a Windows sample and connect it to the detection artifacts of Chapter 12 and the forensics of Chapter 13.

Processes, threads, and virtual memory. A process is a container: a private virtual address space, a handle table, and one or more threads that actually execute code. Malware frequently creates a benign process and then injects code into it (Section 15.6), so the process you see running is not always the code that is running. Each process has a Process Environment Block (PEB) describing its loaded modules; shellcode and reflective loaders walk the PEB to find libraries without calling the loader, which is why the PEB is both an attacker tool and an analyst landmark (Section 9.25).

Handles and objects. Files, registry keys, mutexes, events, and processes are kernel objects accessed through numeric handles. Two behaviors matter for analysis: malware often creates a named mutex to avoid infecting a machine twice, and that mutex name is a high-quality host indicator; and handles to other processes (OpenProcess) followed by memory writes (WriteProcessMemory) and remote thread creation (CreateRemoteThread) are the classic injection signature.

The registry. The Windows registry is a hierarchical configuration database and a primary persistence and configuration surface for malware:

Registry area	Why malware uses it
`...\CurrentVersion\Run` and `RunOnce`	Auto-start a payload at login (persistence)
`...\Services`	Install a malicious service or driver
`...\Winlogon\Shell`, `Userinit`	Hijack the logon path
`...\Image File Execution Options`	Hijack a legitimate program’s launch (debugger key)
App-specific keys	Store configuration, encryption keys, or campaign IDs

These are the same keys the forensic analyst examines in Section 13.28, so learning them once serves both offense-analysis and defense.

Services and drivers. A Windows service runs in the background, often as SYSTEM, and can start at boot, which makes it attractive for persistence and for loading kernel drivers (the mechanism behind the rootkits of Section 15.14). Service creation (sc create, event ID 7045) is a heavily monitored detection point (Section 12.10).

The loader and APIs. When a program starts, the Windows loader maps the executable and its imported DLLs and fills the Import Address Table (Section 15.4). Malware that wants to hide its intentions resolves APIs at runtime by hashing export names (Section 9.25) so the import table looks empty, an evasion the analyst counters by watching the actual calls at run time (Section 15.25). Holding this model in mind, an analyst can look at a sequence of API calls and reconstruct the sample’s plan: allocate memory, write code, start a thread, contact a server, install a Run key, encrypt files. That reconstruction is the point of the whole exercise.

Exercises#

Why is the process you see running not necessarily the code that is running?
Give two reasons malware creates a named mutex, and why the name is useful to a defender.
Name three registry locations malware uses for persistence and what each achieves.
How does resolving APIs by hashing export names help malware, and how does an analyst defeat it?

Answer Key#

Because code injection (Section 15.6) places malicious code into the address space of a separate, often legitimate, process, so the executing code and the host process differ.
To avoid reinfecting a machine (a single-instance check) and to coordinate components; the specific mutex name is a stable host-based indicator of compromise a defender can hunt for.
Any three of: CurrentVersion\Run/RunOnce (auto-start at login), Services (install a service or driver), Winlogon\Shell/Userinit (hijack the logon path), and Image File Execution Options (hijack a program’s launch).
It leaves the import table empty so static analysis cannot see which APIs are used; the analyst runs the sample under a debugger or monitor (Section 15.25) and observes the calls as they happen.

15.24 Obfuscation and Deobfuscation, Worked#

Section 15.10 listed obfuscation techniques; here we work through the common ones and how an analyst reverses them, because deobfuscation is where static analysis is won or lost.

String encryption. Rather than store "http://evil.example/c2" in plaintext (which the strings tool would reveal, Section 15.3), malware stores it encrypted and decrypts it at use. In disassembly this appears as a small routine, a loop that XORs or otherwise transforms a blob just before the string is used:

lea  rsi, [rip+0x3a20]      ; pointer to the encrypted blob
xor  ecx, ecx              ; i = 0
.decrypt_loop:
mov  al, BYTE PTR [rsi+rcx]
xor  al, 0x5a              ; single-byte XOR key 0x5a
mov  BYTE PTR [rsi+rcx], al
inc  ecx
cmp  ecx, 0x18             ; length 24
jl   .decrypt_loop

The analyst recovers the string either statically, by extracting the blob and the key and reproducing the XOR, or dynamically, by setting a breakpoint after the loop and reading the now-decrypted buffer (Section 15.25). A short script does the static route:

blob = bytes.fromhex("2a3e2b2e7e...")   # the 24 bytes from the binary
print(bytes(b ^ 0x5a for b in blob).decode())   # apply the recovered key

API hashing. Instead of naming an API, malware stores a hash and, at runtime, walks each DLL’s export table hashing names until one matches (Section 9.25). To deobfuscate, the analyst identifies the hash algorithm in the resolver, then precomputes the hash of every known export name to build a lookup table that maps each stored hash back to its function, revealing the malware’s true imports.

Control-flow flattening and opaque predicates. Advanced obfuscators dismantle the natural block structure of a function into a dispatcher-plus-state-machine (flattening), or insert branches whose outcome is always the same but not obviously so (opaque predicates), to confuse decompilers. The countermeasures are increasingly automated: symbolic-execution and deobfuscation frameworks (angr, Miasm, and Triton) reason about the possible values a variable can take and simplify away the dead branches, reconstructing the original control flow.

Packing. The most common obfuscation is packing (Section 15.5): the real code is compressed or encrypted and a stub unpacks it into memory at run time. The general unpacking strategy is dynamic: let the stub run under a debugger until it transfers control to the newly written region (the original entry point), then dump that region from memory. Section 15.25 walks that process. The theme across all four is the same: obfuscation raises the cost of static analysis, and the analyst answers either by reproducing the transformation statically or by letting the malware deobfuscate itself and catching the result at run time.

Exercises#

Why do malware authors encrypt strings, and what does the decryption routine typically look like in disassembly?
Describe the two general strategies for recovering an encrypted string.
How does an analyst reverse API hashing to recover a sample’s true imports?
What is control-flow flattening, and what class of tool helps undo it?

Answer Key#

To defeat the strings tool and static signature matching; it appears as a short transform loop (often an XOR) that runs over a data blob immediately before the string is used.
Statically, extract the blob and key from the binary and reproduce the transform in a script; dynamically, breakpoint just after the decryption loop and read the plaintext from memory.
Identify the hashing algorithm in the resolver, then hash every known export name to build a table mapping each stored hash back to the function it selects.
It rewrites a function’s natural blocks into a dispatcher-driven state machine to confuse decompilers; symbolic-execution and deobfuscation frameworks (angr, Miasm, Triton) help reconstruct the original flow.

15.25 A Dynamic Analysis Session#

Sections 15.11 and 15.21 set up the lab; this section walks the dynamic analysis of a packed Windows sample so the pieces connect into a repeatable procedure. Dynamic analysis is the counter to every static evasion in Section 15.24: whatever the malware hides on disk, it must reveal at run time to actually run.

The workflow, executed inside the isolated, snapshotted VM of Section 15.21:

1. Triage statically first (Sections 15.3-15.4):
   - hashes, file type, packer detection (Detect It Easy / DIE)
   - high section entropy suggests packing (Section 15.20 YARA)
   - near-empty import table suggests API hashing
2. Detonate under a monitor to capture behavior:
   - Process Monitor (Procmon): file, registry, process, network events
   - a fake-internet service (INetSim / FakeNet-NG) answers C2 lookups
   - note dropped files, created Run keys, mutexes, contacted domains
3. Unpack under a debugger (x64dbg) to reach the real code:
   - set a breakpoint on VirtualAlloc / VirtualProtect (the stub allocates
     RWX memory for the unpacked payload)
   - run until the stub jumps into the freshly written region: that target
     is the Original Entry Point (OEP)
   - dump the process memory at the OEP (Scylla plug-in) and rebuild the
     import table to get an analyzable, unpacked binary
4. Break on interesting APIs to observe intent directly:
   - CreateFileW / CryptEncrypt  -> ransomware file encryption
   - InternetConnect / WinHttpSendRequest -> C2 traffic
   - RegSetValueEx on a Run key   -> persistence
   - decryption-loop exit (Section 15.24) -> read recovered strings
5. Extract indicators (Section 15.12): domains, IPs, mutex names, file
   paths, registry keys, and a YARA rule (Section 15.20) for the family.

The reason this order works is that each step defeats a specific defense: static triage flags packing and API hashing so you know what to expect; behavioral monitoring reveals actions even when the code is unreadable; debugging to the OEP defeats the packer; and API breakpoints defeat string encryption and hashing by catching the values after the malware itself has decrypted them. The output, a set of indicators and a detection rule, is what turns one analyzed sample into protection for an entire organization, closing the loop with the detection engineering of Chapter 12 and the incident response of Chapter 14.

Exercises#

Why is dynamic analysis the natural counter to static-analysis evasions such as packing and string encryption?
On which API would you breakpoint to catch a packer allocating memory for its unpacked payload, and what are you looking for next?
What is the Original Entry Point, and why is dumping memory at that moment useful?
Give two API breakpoints and the malware behavior each would reveal.

Answer Key#

Because to execute, the malware must unpack and decrypt itself in memory; running it under monitoring and a debugger lets the analyst observe the revealed code, strings, and actions that static analysis cannot see.
VirtualAlloc or VirtualProtect (the stub allocates or marks executable memory); next you run until the stub transfers control into that newly written region, which is the Original Entry Point.
The OEP is the entry of the real, unpacked payload; dumping process memory there yields an unpacked image that (after rebuilding imports) can be analyzed statically like an unpacked binary.
Any two, for example: CryptEncrypt/CreateFileW revealing ransomware file encryption; WinHttpSendRequest/ InternetConnect revealing C2 network activity; RegSetValueEx on a Run key revealing persistence.

15.26 Analyzing Script and .NET Malware#

The disassembly and Windows-internals sections assumed a compiled native binary, but a large and growing share of real malware is not native at all: it is PowerShell, JavaScript, VBScript, or a .NET assembly. These require different tools and are, helpfully, often easier to analyze because they carry more of their original structure. A reverse-engineering course must cover them because they dominate the initial-access stage (Section 15.16).

Script malware (PowerShell, JScript, VBScript). The commodity infection chain is a document macro or a script file that downloads and runs a payload. Scripts are text, so the whole battle is deobfuscation, because attackers wrap the logic in layers of encoding to defeat signatures:

Common PowerShell obfuscation and how to peel it:
  -EncodedCommand <base64>   -> base64-decode to get the real script
  [Convert]::FromBase64String / -join / -replace  -> string reassembly
  IEX (Invoke-Expression)    -> the sink that runs the decoded string
Deobfuscation strategy: do NOT run it. Replace the final IEX / eval with a
print/write so the script reveals its decoded next stage instead of
executing it, or decode the layers manually. Tools: CyberChef for the
encoding layers, and script-block logging (event ID 4104, Section 12.10)
which records PowerShell as it actually executes, defeating obfuscation.

The key insight is that script malware must eventually pass a decoded string to an execution sink (IEX, eval, WScript.Shell.Run); intercepting the value at that sink, statically by editing it to print, or dynamically via logging, reveals the true payload no matter how many encoding layers wrap it. This mirrors the API-breakpoint idea of Section 15.25.

.NET malware. A great deal of modern malware (many commodity stealers and loaders) is written in C# and compiled to .NET Common Intermediate Language (CIL), not native code. CIL retains method names, class structure, and often readable logic, so .NET samples decompile back to near-source. The tools are dnSpy and ILSpy, which reconstruct C# from the assembly and, in dnSpy’s case, let the analyst set breakpoints and debug the sample as if they had the source. Obfuscators (such as ConfuserEx) rename symbols and encrypt strings to fight this, and de-obfuscators (de4dot) reverse many of them. The practical workflow: identify a sample as .NET (the PE imports mscoree.dll and Detect It Easy flags it), open it in dnSpy, run de4dot if it is obfuscated, then read the decompiled C# directly, which is far faster than native reversing.

The broader lesson is to identify the technology first. Native, .NET, PowerShell, and JavaScript malware each demand a different toolchain, and pointing native disassembly tools at a .NET binary yields noise. Recognizing the sample’s type (from its headers, imports, and strings, Sections 15.3 and 15.4) is the first decision in any analysis and often the difference between an hour and a week of work.

Exercises#

Why is deobfuscation the central task when analyzing PowerShell or JavaScript malware, and what is the general trick for recovering the payload without running it?
What is the execution sink in script malware, and why is it the ideal place to capture the true payload?
Why does .NET malware decompile back to near-source, and which tools perform this?
Why is identifying the sample’s technology the first step of analysis?

Answer Key#

Because script malware is plaintext wrapped in encoding layers to defeat signatures; the general trick is to replace the final execution call (IEX/eval) with a print/write so the decoded next stage is revealed instead of executed, or to decode the layers manually.
The sink is the call that runs a decoded string (IEX, eval, WScript.Shell.Run); capturing the value passed to it reveals the real payload regardless of how many encoding layers preceded it.
.NET compiles to CIL, which retains method names, class structure, and logic, so decompilers (dnSpy, ILSpy) reconstruct readable C#; de4dot reverses common obfuscation first.
Native, .NET, and script malware require different toolchains; using native tools on a .NET or script sample produces noise, so recognizing the type from headers, imports, and strings determines the whole approach.

15.27 Linux and ELF Malware Analysis#

Most of this chapter has centered on Windows because most commodity malware targets it, but Linux is the operating system of servers, cloud workloads, containers, and the Internet of Things, and Linux malware, cryptominers, DDoS botnets, and ransomware aimed at ESXi and cloud hosts, has grown sharply. Analysis transfers, but the format and toolchain differ, so a complete course covers them.

The ELF format. Where Windows uses PE (Section 15.4), Linux uses the Executable and Linkable Format (ELF). The analyst’s landmarks are analogous: an ELF header, program headers (segments the loader maps), and section headers (.text code, .data, .rodata strings, .dynsym/.dynamic for dynamic linking). The tooling maps one-to-one onto the concepts of Sections 15.3 and 15.4:

Task	Windows tool	Linux/ELF tool
Identify file, headers	Detect It Easy, CFF Explorer	`file`, `readelf -h`, `readelf -l`
Imported functions	Dependency Walker, IAT view	`readelf -d`, `nm -D`, `objdump -T`
Strings	strings.exe, FLOSS	`strings`, `strings -e l`
Disassembly/decompile	IDA, Ghidra, x64dbg	Ghidra, radare2/rizin, objdump -d, gdb
Syscall/behavior trace	Procmon, API Monitor	`strace`, `ltrace`, `sysdig`, Falco

Linux-specific behaviors. A Linux analyst watches for a recognizable repertoire: persistence via cron jobs, systemd services, or ~/.bashrc and /etc/rc.local; privilege escalation via SUID binaries, sudo misconfigurations, or writable /etc/passwd (Section 9.18); process and file hiding via LD_PRELOAD userland rootkits or loadable kernel modules (Section 15.14); and, for IoT botnets such as the Mirai lineage, brute-forced Telnet/SSH credentials and architecture-specific droppers (the same malware cross-compiled for ARM, MIPS, x86 to match cheap devices). Behavioral tracing is especially productive on Linux because strace records every system call a sample makes, so file, network, and process activity is visible directly, which is the Linux counterpart to the API monitoring of Section 15.25.

Containers and the cloud. Modern Linux malware increasingly targets exposed container and orchestration APIs (a misconfigured Docker or Kubernetes endpoint) to deploy cryptominers at scale, and cloud ransomware targets hypervisor hosts. Analysis extends naturally: the payload is still an ELF binary or a script, examined with the tools above, but the delivery reasoning shifts to cloud identity and exposed services (Chapter 17). The unifying lesson is that the analysis method, static triage then behavioral tracing then targeted disassembly, is platform independent; only the file format and the specific tools change. An analyst who has internalized the method retools for a new platform in an afternoon.

Exercises#

What is the ELF equivalent of the Windows PE format, and name two ELF sections and what they hold.
Give the Linux tools you would use to (a) list a binary’s imported functions and (b) trace its runtime behavior.
Name three Linux-specific persistence mechanisms an analyst should check.
Why is strace particularly valuable for Linux malware analysis?

Answer Key#

ELF (Executable and Linkable Format). Any two sections, for example: .text (code), .rodata (read-only strings/constants), .data (initialized data), or .dynsym/.dynamic (dynamic-linking information).
(a) readelf -d, nm -D, or objdump -T for imported/dynamic symbols; (b) strace (system calls) or ltrace (library calls), optionally sysdig/Falco.
Any three of: cron jobs, systemd services, ~/.bashrc/profile scripts, /etc/rc.local, or LD_PRELOAD/kernel modules.
It records every system call the sample makes, exposing file, network, and process activity directly, which is the Linux counterpart to Windows API monitoring and often reveals behavior without disassembly.

15.28 Extracting Malware Configuration at Scale#

A single analyzed sample (Section 15.25) yields indicators for one infection, but real threat intelligence comes from analyzing a family across thousands of samples, and the key to that is configuration extraction. Most modern malware families are built from a common code stub plus an embedded configuration block, the campaign- specific settings, so extracting that block turns each new sample into structured intelligence automatically.

What is in a config. A malware configuration typically holds the command-and-control servers (domains, IPs, URLs), the encryption keys or campaign identifier, the mutex name (Section 15.23), target lists, and behavior flags. Because the code is shared across a campaign, these fields are what actually distinguish one operation from another, which is why they are the intelligence prize.

How it is stored, and extracted. The config is rarely plaintext; it is encrypted or encoded exactly like the strings of Section 15.24. Extraction therefore reuses the same skills:

1. Locate the config: often a distinct high-entropy blob in a PE resource
   or a specific section, referenced by the decode routine.
2. Recover the decode logic: find the routine that transforms the blob
   (XOR, RC4, AES) and its key, by static reading (Section 15.22) or by
   breakpointing after decryption at run time (Section 15.25).
3. Reproduce it in a parser: write a script that, given any sample of the
   family, locates the blob, applies the decode, and emits the fields as
   structured data (JSON).

# Skeleton of a family config extractor (illustrative)
def extract_config(sample_bytes):
    blob = find_config_blob(sample_bytes)      # e.g., a known PE resource
    key  = derive_key(sample_bytes)            # static key or per-sample
    cfg  = rc4(blob, key)                       # reproduce the malware's decode
    return {
        "c2":       parse_c2_list(cfg),
        "campaign": parse_campaign_id(cfg),
        "mutex":    parse_mutex(cfg),
    }
# Run across a corpus to map a whole campaign's infrastructure over time.

Why scale changes the game. Running one extractor across a corpus of samples reveals the campaign’s entire infrastructure and how it evolves, feeds automated blocking (Chapter 12) and hunting, and lets defenders track an actor over months. This is how commercial and open threat-intelligence platforms (and projects such as the community configuration-extractor collections) convert a flood of samples into a map of adversary operations. The lesson that completes the chapter is that malware analysis is not only about understanding one program; its highest value is turning repeated structure into automation, so that human expertise is spent once on the family and then applied a thousand times by code.

Exercises#

Why is the configuration block, rather than the shared code, the intelligence prize in a malware family?
What fields does a malware configuration typically contain?
How does config extraction reuse the deobfuscation skills of Section 15.24?
What does running a config extractor across a large corpus of samples enable?

Answer Key#

Because the code is common across a campaign; the configuration holds the campaign-specific settings (C2, keys, campaign ID, targets) that actually distinguish and identify one operation from another.
Command-and-control servers (domains/IPs/URLs), encryption keys or a campaign identifier, the mutex name, target lists, and behavior flags.
The config is encrypted or encoded like the malware’s strings, so extraction locates the blob, recovers the decode routine and key (statically or by breakpointing after decryption), and reproduces the transform.
It maps the campaign’s full infrastructure and its evolution over time, feeding automated blocking and hunting and enabling long-term tracking of the actor, turning many samples into structured intelligence.

15.29 Case Study: The WannaCry Worm#

WannaCry, which struck in May 2017, is worth a detailed technical case study because it combined, in one documented event, several themes of this book: a leaked exploit, worm propagation, ransomware, and an accidental kill switch. The facts below are drawn from the extensive public analysis that followed the outbreak.

Propagation by exploit, not by user. Unlike the phishing-borne ransomware of Section 15.13, WannaCry spread as a worm (Section 15.17): it propagated automatically across networks with no user interaction by exploiting a vulnerability in Microsoft’s Server Message Block (SMB) version 1 protocol. The exploit, known as EternalBlue, targeted the flaw that Microsoft had patched in bulletin MS17-010 two months earlier, in March 2017. EternalBlue had been developed within a national intelligence agency and was released publicly by a group calling itself the Shadow Brokers, a textbook illustration of the n-day exploitation of Section 9.33: the patch existed, but unpatched systems were abundant, and a powerful exploit was now public.

The dual payload. On each machine it reached, WannaCry did two things: it encrypted the victim’s files with hybrid cryptography (Section 15.13) and displayed a ransom demand in Bitcoin, and it used the same SMB exploit to scan for and infect further reachable machines, both on the local network and across the internet. The worm component is what turned a single infection into a global event affecting an estimated hundreds of thousands of computers across many countries within days, including systems in the United Kingdom’s National Health Service, where it disrupted hospital operations.

The accidental kill switch. WannaCry contained a curious check: before running, it tried to contact a specific, then-unregistered domain, and if that request succeeded, it stopped. A security researcher analyzing the sample registered that domain, which caused the malware’s own check to halt the majority of the spread. The mechanism is debated (a possible anti-sandbox check, since some sandboxes answer every domain, as discussed in Section 15.24), but the effect is not: registering the domain functioned as a global off switch, a vivid lesson in how much dynamic analysis of a live sample can matter.

What it teaches. WannaCry ties the course together. Patch management (Chapters 5 and 19) would have prevented it, since the fix predated the outbreak. Network segmentation and disabling the obsolete SMBv1 (Chapter 11) would have contained it. Backups (Section 15.13) were the only reliable recovery, since paying rarely restored files. Detection of the exploit and the propagation (Chapter 12) and a rehearsed incident response (Chapter 14) were what separated organizations that recovered quickly from those that did not. One incident, correctly understood, motivates nearly every defensive discipline in this book, which is why it remains the canonical worked example of a modern worm.

Exercises#

How did WannaCry propagate, and why did that make it so much more damaging than phishing-delivered ransomware?
Explain how WannaCry is an example of n-day exploitation (Section 9.33).
What was the kill switch, and how did a researcher use it to halt the spread?
Name three defensive controls from other chapters that would have prevented or contained WannaCry.

Answer Key#

It spread as a worm using the EternalBlue SMBv1 exploit, infecting reachable machines automatically with no user interaction, so it propagated across and between networks far faster than malware requiring a user to click.
Microsoft had already patched the SMB vulnerability (MS17-010) two months before the outbreak, so WannaCry exploited a known, patched (n-day) flaw on systems that had not yet applied the update, not a zero-day.
The malware checked whether a specific hardcoded domain was reachable and stopped if it was; a researcher registered that previously unregistered domain, so the check began succeeding and the malware halted itself on most systems.
Any three of: timely patching (MS17-010), disabling SMBv1 and network segmentation, offline backups for recovery, exploit/propagation detection, and a rehearsed incident-response plan.

15.30 A Catalog of Anti-Analysis Techniques#

Malware fights back against the analyst, and Section 15.10 introduced this at a high level. Because recognizing and defeating anti-analysis is a large part of real reverse-engineering work, this section consolidates the techniques into a reference catalog organized by what each one attacks: the debugger, the virtual machine, the disassembler, or the analyst’s time.

Anti-debugging (defeat dynamic analysis under a debugger, Section 15.25):

Technique	How it works	Countermeasure
`IsDebuggerPresent` / PEB flag	Reads the Windows flag set when a debugger is attached	Patch the flag or the check to return false
Timing checks (`rdtsc`, `GetTickCount`)	Debugging slows execution; large time deltas reveal it	Patch out the check; keep the delta small
INT 3 / breakpoint scanning	Scans its own code for `0xCC` breakpoint bytes	Use hardware breakpoints instead of software ones
`NtQueryInformationProcess`	Asks the kernel about debug status directly	Hook or patch the returned value

Anti-VM and anti-sandbox (behave benignly when watched, so analysis sees nothing malicious):

Technique	How it works	Countermeasure
Artifact checks	Looks for VM drivers, MAC prefixes, registry keys	Harden the VM to remove telltale artifacts
Resource checks	Real machines have many cores, large disks, uptime	Provision a realistic-looking analysis VM
Human-interaction checks	Waits for mouse movement or a reboot	Simulate user activity; use interaction-aware sandboxes
Sleep/stalling	Delays execution past the sandbox’s time budget	Patch the sleep, or fast-forward time in the sandbox

Anti-disassembly (defeat static analysis, Section 15.22):

Technique	How it works	Countermeasure
Junk bytes / overlapping instructions	Confuse the linear disassembler’s byte alignment	Use recursive-descent disassembly; correct manually
Opaque predicates	Branches that are always taken but look conditional	Symbolic simplification (angr/Miasm), manual analysis
Control-flow flattening	Real flow hidden behind a dispatcher (Section 15.24)	Deobfuscation frameworks reconstruct the graph
Self-modifying / packed code	Real code appears only at run time	Dynamic unpacking to the OEP (Section 15.25)

The unifying insight, and the reason this catalog matters more than any single entry, is that anti-analysis is an arms race with a structural asymmetry in the analyst’s favor: to run, the malware must eventually execute its real behavior on a real target, so a sufficiently realistic dynamic environment always wins in the end (Section 15.25). Static anti-analysis raises the cost of reading the code, and dynamic anti-analysis raises the cost of running it safely, but neither can make the malware both undetectable and functional. The practical workflow is therefore to expect these techniques, identify which are present during static triage (Section 15.3), and choose the analysis approach, harden the VM, patch the check, or move to bare metal, that neutralizes them. Fluency with this catalog is what turns a sample that “does nothing when I run it” into a solved case.

Exercises#

Organize the four categories of anti-analysis by what each one attacks.
How does a timing check detect a debugger, and how is it defeated?
Why do anti-VM checks often take the form of looking for artifacts and limited resources?
Explain the structural asymmetry that favors the analyst in the anti-analysis arms race.

Answer Key#

Anti-debugging attacks dynamic analysis under a debugger; anti-VM/anti-sandbox attacks the analysis environment; anti-disassembly attacks static reading of the code; and stalling/sleep attacks the analyst’s (or sandbox’s) time budget.
Debugging slows execution, so the malware measures elapsed time (via rdtsc or GetTickCount) across a region and infers a debugger from an abnormally large delta; it is defeated by patching out the check or keeping the measured interval small.
Sandboxes and analysis VMs leave telltale artifacts (specific drivers, MAC prefixes, registry keys) and are often under-provisioned (few cores, small disk, no real user), so checking for these cheaply distinguishes an analysis environment from a real victim.
To accomplish anything, the malware must eventually execute its real behavior on a real-looking target, so a sufficiently realistic dynamic environment will observe it; the malware cannot be simultaneously fully evasive and functional.

15.31 Writing the Malware Analysis Report#

Analysis that is not communicated is wasted, so the analyst’s actual deliverable is a report, just as the incident responder’s is (Section 14.27). A malware analysis report translates hours of reversing into decisions other people can act on, and its structure is standardized so that readers, incident responders, threat-intel teams, and detection engineers, can find what they need quickly.

1. Summary (non-technical). What the sample is (family, type), what it
   does, how dangerous it is, and the one or two actions the reader should
   take. Written for a manager or responder who will read nothing else.

2. Sample identification. File names, sizes, and hashes (MD5, SHA-256,
   Section 15.3); file type; and any family/attribution assessment with a
   stated confidence level.

3. Capabilities and behavior. What the malware does: persistence
   mechanism (Section 15.23), privilege use, network/C2 behavior, payload
   (encryption, theft, propagation), and anti-analysis observed
   (Section 15.30). Mapped to MITRE ATT&CK techniques.

4. Indicators of compromise. The actionable list (Section 15.12): file
   hashes, dropped file paths, registry keys, mutex names, domains, IPs,
   and URLs -- formatted so a defender can ingest them directly.

5. Detection. YARA rules for the family (Section 15.20), network
   signatures, and behavioral detections (Chapter 12), so the analysis
   becomes protection.

6. Recommendations. Containment, eradication, and hardening guidance
   feeding incident response (Chapter 14).

Two disciplines make the report trustworthy. First, separate observation from inference: “the sample writes to the Run key” is an observation, “the sample is designed for long-term persistence” is an inference, and stating confidence (“assessed with moderate confidence”) keeps the reader correctly calibrated, the same standard as the forensic reporting of Section 13.19. Second, lead with the actionable: the indicators and detections in sections 4 and 5 are what a defender uses within the hour, so they are formatted for direct ingestion rather than buried in prose. The report closes the loop that the whole chapter has traced: a sample is triaged, analyzed statically and dynamically, its configuration and indicators extracted, and finally its findings written up so that one analyst’s work protects an entire organization. That transfer, from a single binary to durable, shared defense, is the reason malware analysis is a discipline rather than a curiosity.

Exercises#

Who are the readers of a malware analysis report, and why does that shape its structure?
Why must the report separate observation from inference, and what does stating a confidence level achieve?
Which two sections are the most time-critical for a defender, and how should they be formatted?
How does the report complete the malware-analysis workflow of this chapter?

Answer Key#

Managers/responders (who need the summary and actions), incident responders and detection engineers (who need indicators and detections), and threat-intel teams (who need capabilities and attribution); the report is layered so each audience finds its part quickly.
Because conflating what was seen with what it implies misleads readers; separating them, and stating confidence, keeps the reader correctly calibrated about what is established fact versus assessment.
The indicators of compromise and the detection content; they should be formatted for direct machine ingestion (clean lists of hashes, domains, rules) so defenders can deploy them immediately.
It transfers the analyst’s findings into a form others can act on, turning the triage, static and dynamic analysis, and indicator/config extraction of one sample into durable, shared protection for the organization.

15.32 Recognizing C++ in Disassembly#

Sections 15.22 and 15.23 covered reading C constructs in assembly; a growing share of malware is written in C++, which introduces its own recognizable patterns. Because object orientation changes how code looks at the machine level, an analyst who cannot recognize C++ constructs will misread a modern sample. This section covers the four signatures that matter.

Name mangling. C++ supports function overloading, namespaces, and classes, so the compiler encodes a function’s full signature into its symbol name, a process called name mangling. A method Foo::bar(int) appears in the binary under an encoded name rather than a plain bar, and the encoding differs by compiler: Microsoft Visual C++ produces names beginning with ? (for example ?bar@Foo@@...), while the Itanium ABI used by GCC and Clang produces names beginning with _Z (for example _ZN3Foo3barEi). Modern disassemblers such as IDA Pro and Ghidra demangle these automatically, and the presence of mangled names is itself the first clue that a binary is C++ rather than C.

The this pointer and thiscall. A C++ method receives a hidden first argument: a pointer to the object it operates on, the this pointer. Under the Microsoft x86 thiscall convention, this is passed in the ecx register rather than on the stack, so a function that reads ecx as a base pointer for member accesses (for example mov eax, [ecx+8]) is almost certainly a C++ method accessing a field at offset 8 of its object. On x86-64 the object pointer is simply the first argument in rcx (Windows) or rdi (System V), consistent with the calling conventions of Section 9.27.

Virtual function tables. The signature that most distinguishes C++ is the virtual function table (vtable). When a class has virtual functions, each object carries a hidden pointer (the vptr) to a table of function pointers; a virtual call is resolved at run time by loading the vptr, indexing into the table, and calling through it. In disassembly this appears as a double indirection, mov eax, [ecx] (load the vtable) followed by call [eax+offset] (call the function at a fixed slot), rather than a direct call to a fixed address. Spotting this indirect-call-through-a-table pattern is how an analyst recognizes polymorphism and maps out a class’s methods.

Objects, constructors, and RTTI. Object creation typically appears as an allocation (new, which calls an allocator) immediately followed by a call that initializes the memory and sets the vptr, that call is the constructor. A matching destructor runs on cleanup. Where the compiler emits run-time type information (RTTI), the analyst gets a gift: RTTI structures often contain the actual class names as strings, directly revealing the program’s class hierarchy. Putting these together, the analyst reconstructs objects, their fields, their methods, and their inheritance relationships, turning an opaque C++ binary back into an understandable design. The practical takeaway is that C++ is not harder to reverse than C, it is differently shaped, and the four signatures, mangled names, the ecx/this pattern, vtable indirection, and constructor-plus-RTTI, are the lenses that bring that shape into focus.

Exercises#

What is name mangling, and why does its presence immediately tell you a binary was compiled from C++?
Under the Microsoft x86 thiscall convention, how is the this pointer passed, and what assembly pattern reveals a member-variable access?
Describe the disassembly signature of a C++ virtual function call and why it differs from a normal call.
Why is run-time type information (RTTI), when present, valuable to an analyst?

Answer Key#

Name mangling encodes a function’s full signature (class, namespace, parameter types) into its symbol name to support overloading; the compiler-specific mangled forms (?... for MSVC, _Z... for the Itanium ABI) are a direct indicator that the source language was C++.
In ecx; a function using ecx as a base register for offset accesses such as mov eax, [ecx+8] is a C++ method reading a field of its object.
A virtual call loads the object’s vtable pointer and then calls through a fixed slot in that table (for example mov eax,[ecx] then call [eax+offset]), an indirect double-dereference, rather than a direct call to a fixed address, because the target is resolved at run time by object type.
RTTI structures frequently embed the real class names as strings and encode the inheritance hierarchy, so they directly reveal the program’s object design that would otherwise have to be reconstructed by hand.

15.33 Covert and Stealthy Launching#

Section 15.6 and Section 15.23 introduced code injection; because covert launching is a distinct analysis topic (and one that malware relies on to hide), this section consolidates the family of techniques by which malware runs its code inside another, trusted process so that the malicious activity does not appear as its own suspicious program. Recognizing these is central to reading a modern Windows sample.

A launcher (or loader) is malware whose job is to set up and covertly execute another piece of malware, often by extracting an embedded or downloaded payload and running it stealthily. The main covert-launching techniques, and how each appears to an analyst, are:

Technique	Mechanism	Analyst-visible signature
DLL injection	Force a remote process to load a malicious DLL (`CreateRemoteThread` calling `LoadLibrary`)	`OpenProcess`, `VirtualAllocEx`, `WriteProcessMemory`, `CreateRemoteThread` sequence
Direct injection	Write code straight into a remote process and run it, without a DLL on disk	Same API sequence, but injecting shellcode rather than a library path
Process replacement (hollowing)	Start a legitimate process suspended, unmap its memory, overwrite it with malicious code, and resume	`CreateProcess` with `CREATE_SUSPENDED`, then `NtUnmapViewOfSection`, `WriteProcessMemory`, `SetThreadContext`, `ResumeThread`
Hook injection	Insert a malicious hook into another process via Windows hooks (`SetWindowsHookEx`)	`SetWindowsHookEx` targeting keyboard or message hooks (also used by keyloggers)
APC injection	Queue an asynchronous procedure call to run code in a target thread (`QueueUserAPC`)	`QueueUserAPC` pointing at `LoadLibrary` or shellcode in the target

The unifying idea is disguise through a trusted host: after any of these, the malicious code runs under the identity of a normal process (for example svchost.exe or explorer.exe), so it inherits that process’s trust, evades naive process-name allowlists, and separates the observed activity from the file that launched it. Process replacement (hollowing) is the most deceptive of the set, because the target appears in the process list as a legitimate program from the correct path while its memory has been entirely replaced. The defensive response follows directly from the signatures above: endpoint detection (Section 12.10) watches for the tell-tale API sequences (a suspended CreateProcess followed by remote memory writes, or CreateRemoteThread into an unrelated process), and memory forensics (Section 13.13) finds injected code that has no backing file on disk. Reading these techniques is therefore both an analysis skill and the basis for detecting them.

Exercises#

What is a launcher (loader), and why is covert launching valuable to malware?
Describe process replacement (process hollowing) and why it is especially deceptive.
Give the characteristic Windows API sequence that suggests remote DLL injection.
How do the analyst-visible signatures of covert launching become the basis for detection?

Answer Key#

A launcher is malware that sets up and stealthily executes another payload (often extracted or downloaded); covert launching lets the payload run inside a trusted process, inheriting its identity and separating the malicious activity from the file that started it.
The malware starts a legitimate process suspended, unmaps its original memory, writes malicious code in its place, fixes the thread context, and resumes it; it is deceptive because the process appears in the task list as the legitimate program from its correct path while actually running attacker code.
OpenProcess, then VirtualAllocEx and WriteProcessMemory to place the DLL path or code, then CreateRemoteThread (often invoking LoadLibrary) in the target process.
The same API call sequences and memory patterns that identify each technique are what endpoint telemetry and memory forensics monitor for (for example a suspended CreateProcess plus remote writes, or injected code with no file on disk), turning analysis signatures into detections.

15.34 Malware-Focused Network Signatures#

Section 15.12 covered extracting indicators of compromise, and Chapter 12 covered detection engineering broadly; this section addresses the specific reverse-engineering skill of producing robust network signatures from malware analysis, the point where offense-analysis and defense meet. The goal is a signature that reliably catches a malware family’s traffic while resisting the attacker’s attempts to change it.

The key insight is that analysis-driven signatures beat surface-derived ones. A defender who only captures a sample’s traffic and writes a signature on whatever strings appear (the snort-the-obvious approach) produces a brittle rule that the attacker defeats by changing a user-agent or a URL path. An analyst who has reverse engineered the malware understands which parts of the traffic are essential to the malware’s operation (a fixed protocol structure, a specific encoding, a hardcoded key, an authentication token the server checks) and targets those, producing a signature that survives cosmetic changes. This is why deep analysis produces better detection than traffic capture alone.

Several principles guide the work. First, prefer the stable over the incidental: key on the parts of the protocol the malware cannot easily change without breaking its own command-and-control, not on a banner it can edit freely. Second, understand the attacker’s perspective: the adversary can see published signatures and will test malware against antivirus and public rules (Section 15.30), so a signature that reveals too much of what the defender knows can be studied and evaded, a real tension in signature sharing. Third, capture safely: traffic must be generated in the isolated lab of Section 15.21 with a fake-internet service, never by letting the sample reach real command-and-control. Fourth, balance specificity against generality: too specific and it misses variants, too broad and it floods the analyst with false positives (Section 12.16). The output, typically a Snort or Suricata rule (Section 12.7) plus host indicators, is then validated against both malicious and benign traffic before deployment. The unifying lesson is that the best network detection is a product of analysis, not a substitute for it: understanding how the malware actually communicates is what lets the defender write a rule the attacker cannot cheaply evade.

Exercises#

Why does a network signature derived from deep analysis outperform one written from captured traffic alone?
What kinds of traffic features should a robust signature key on, and which should it avoid?
Why is the attacker’s ability to see published signatures a consideration when designing and sharing them?
How must malware traffic be generated for signature development, and why?

Answer Key#

Because analysis reveals which parts of the traffic are essential to the malware’s operation (protocol structure, encoding, keys, tokens); keying on those produces a signature that survives cosmetic changes, whereas a rule built on incidental strings is defeated by trivial edits.
It should key on stable, operationally required features (fixed protocol format, specific encoding, hardcoded authentication values) and avoid easily changed incidentals such as user-agent strings or URL paths.
Adversaries test their malware against public antivirus and detection rules, so a widely shared signature can be studied and evaded; this creates tension between the value of sharing detection and the risk of teaching the attacker exactly what to change.
In an isolated analysis lab using a fake-internet service that answers the malware’s requests, never by allowing the sample to contact real command-and-control, so behavior can be observed without aiding the attacker or spreading the malware.

15.35 Debuggers, Disassemblers, and Kernel Debugging#

The syllabus of a reverse-engineering course is organized around specific tools, and an analyst must know which tool fits which task. Sections 15.3, 15.4, and 15.22 used these tools in passing; this section consolidates the toolchain and adds the one capability not yet covered, kernel debugging, which rootkit analysis (Section 15.14) requires.

Disassemblers and decompilers. The disassembler is the analyst’s map of a binary’s code. IDA Pro, from Hex-Rays, has long been the commercial standard and is now at version 9, with a free edition (IDA Free) for learning; its interactive navigation and Hex-Rays decompiler set the bar. Ghidra, released free and open source by the NSA, provides comparable disassembly and a decompiler and has become the common teaching tool (Section 15.22). Both perform static analysis: they show the code without running it.

User-mode debuggers. A debugger runs the binary and lets the analyst pause at breakpoints, step one instruction at a time, inspect and modify registers and memory, and watch behavior unfold, the dynamic counterpart to the disassembler (Section 15.25). On Windows, OllyDbg was the classic user-mode debugger, but it is 32-bit only and long unmaintained (its site has not been updated since 2014), so modern practice uses x64dbg, an open-source debugger that is the spiritual successor to OllyDbg and supports 64-bit targets. IDA Pro also includes a debugger, and GDB fills the role on Linux (Section 9.28). Debuggers are what defeat packing and string encryption, because the analyst can let the malware decrypt itself and then read the result from memory (Sections 15.24 and 15.25).

Kernel debugging. User-mode debuggers cannot see into the operating-system kernel, so analyzing a rootkit or a malicious driver (which run in kernel space, Section 15.14) requires a kernel debugger. WinDbg, Microsoft’s debugger, is the standard tool: it attaches to the Windows kernel, typically from a second machine or across a virtual-machine connection, and lets the analyst inspect kernel structures, set breakpoints in driver code, and observe the low-level hooks by which rootkits hide processes and files. Kernel debugging is more involved than user-mode work, since a mistake can halt the whole system, which is why it is done against a disposable virtual machine (Section 15.21). The organizing principle across the toolchain is the static-versus-dynamic pairing: disassemblers (IDA Pro, Ghidra) show the code at rest, user-mode debuggers (x64dbg, IDA, GDB) show it running in an application, and a kernel debugger (WinDbg) shows it running in the operating system itself. Choosing the right one for the question at hand is much of the craft.

Exercises#

Distinguish the role of a disassembler from that of a debugger in reverse engineering.
Why has x64dbg replaced OllyDbg as the standard Windows user-mode debugger for malware analysis?
Why does analyzing a rootkit require a kernel debugger rather than a user-mode debugger?
Why is kernel debugging performed against a disposable virtual machine?

Answer Key#

A disassembler (IDA Pro, Ghidra) statically shows the program’s code without running it; a debugger runs the program and lets the analyst pause, step, and inspect or modify registers and memory dynamically as it executes.
OllyDbg is 32-bit only and has been unmaintained since 2014, so it cannot handle modern 64-bit malware; x64dbg is an actively maintained open-source successor that supports 64-bit targets.
Rootkits and malicious drivers run in kernel space, which user-mode debuggers cannot inspect; a kernel debugger such as WinDbg attaches to the kernel itself to observe driver code and the low-level hooks rootkits use to hide.
Because a mistake during kernel debugging can crash or halt the entire system, and the malware runs with kernel privileges, so a disposable virtual machine contains the risk and can be reverted.

Chapter Summary#

This chapter is a practical guide to understanding malicious software. It set out a malware taxonomy and the safe analysis environment, then developed static and dynamic analysis and the anti-analysis and evasion techniques that malware uses to resist both. It described how to structure a malware analysis report, surveyed antivirus and antimalware defenses, and provided a field guide to malware types before a deep dive into the malware lifecycle and modern ransomware. The central message is that careful, isolated analysis turns an opaque sample into actionable indicators and that defenders must account for code that actively fights inspection.

Why This Matters#

Malware analysis transforms an unknown threat into a known one. Every IOC extracted, every behavior documented, and every YARA rule written benefits every organization using that intelligence. Analysis also drives detection engineering: a malware family’s unique import combinations, mutex names, or C2 patterns become Snort signatures, YARA rules, and SIEM correlation rules that detect the next infection before it spreads.

News in Focus: WannaCry and the Worm That Used a Leaked Exploit (2017)#

WannaCry, in May 2017, is the case that fused worm, ransomware, and a leaked nation-state exploit into a global incident. It spread automatically using EternalBlue, an exploit of a Server Message Block (SMBv1) vulnerability (CVE-2017-0144) that had been developed by a national intelligence agency and then leaked, and for which Microsoft had issued a patch (MS17-010) two months earlier. Unpatched Windows systems worldwide, including the UK’s National Health Service, were encrypted within hours; the spread was slowed when a researcher registered a “kill-switch” domain hard-coded in the malware. The episode distilled several of this book’s lessons at once: the danger of unpatched systems and legacy protocols, the dual-use and leakage risk of offensive tooling (Chapter 18), the destructive power of worm-like propagation, and the fact that the patch existed but had not been applied, exactly the patch-management gap the GreyNoise edge data in Chapter 8 quantified. (Figures and attributions are per public reporting on the incident.)

Knowledge Check

What property distinguishes a worm from a virus and a Trojan?
In a modern ransomware intrusion, why does the attacker exfiltrate data and disable backups before encrypting?
Why is fileless malware hard for traditional antivirus to catch, and which two techniques help?

Answers: (1) A worm self-propagates across networks with no user action; a virus needs a host file to be opened, and a Trojan needs the user to run disguised software. (2) Exfiltration enables double extortion (threatening to leak data even if files are restored), and disabling backups removes the victim’s ability to recover without paying. (3) It runs in memory and abuses legitimate built-in tools, leaving few on-disk signatures; memory forensics (Chapter 13) and behavior-based detection/EDR (Chapter 12) help.

News in Focus: Fileless and Living-off-the-Land Attacks#

The discovery of sophisticated ransomware variants using legitimate management tools (PSExec, WMI, RDP) rather than traditional malware executables challenged organizations that relied on antivirus detection. These campaigns were classified as fileless or living-off-the-land because the malicious activity was performed by binaries already present on every Windows system. Detection required behavioral analysis rather than signature matching, driving the market shift from antivirus to endpoint detection and response (EDR).

# Chapter 15 -- Static analysis simulation: PE imports, strings, entropy, YARA

import math, re
from collections import Counter

def shannon_entropy(data: bytes) -> float:
    if not data:
        return 0
    counts = Counter(data)
    total = len(data)
    return -sum((c/total)*math.log2(c/total) for c in counts.values())

# Simulated PE import analysis
suspicious_imports = {
    "CreateRemoteThread":   "Process injection (T1055)",
    "VirtualAllocEx":       "Allocate memory in remote process",
    "WriteProcessMemory":   "Write to remote process memory",
    "RegSetValueEx":        "Registry persistence (T1547)",
    "WinExec":              "Execute commands",
    "CryptEncrypt":         "Encryption -- possible ransomware",
    "FindFirstFile":        "File enumeration -- possible ransomware",
    "InternetOpenUrl":      "Network C2 communication",
    "IsDebuggerPresent":    "Anti-debugging check (T1622)",
}

benign_imports = ["GetLastError", "HeapAlloc", "CreateFile", "CloseHandle",
                  "GetModuleHandleA", "LoadLibraryA", "GetProcAddress"]

print("=== Simulated PE Import Analysis ===")
sample_imports = list(suspicious_imports.keys())[:6] + benign_imports[:4]

print(f"\n  Total imports found: {len(sample_imports)}")
print(f"  {'Import':<28} {'Suspicion'}")
print("  " + "-"*70)
for imp in sample_imports:
    note = suspicious_imports.get(imp, "Benign")
    flag = " <-- SUSPICIOUS" if imp in suspicious_imports else ""
    print(f"  {imp:<28} {note}{flag}")

# Entropy analysis
import random
random.seed(42)
packed_data   = bytes([random.randint(0,255) for _ in range(4096)])  # high entropy
unpacked_data = (b"This is a normal text string with low entropy. " * 100)[:4096]
code_section  = bytes([random.randint(0,127) for _ in range(4096)])  # medium entropy

print(f"\n=== Entropy Analysis ===")
print(f"  Packed/encrypted section : {shannon_entropy(packed_data):.2f} bits/byte  (>7.5 = packed)")
print(f"  Code section             : {shannon_entropy(code_section):.2f} bits/byte  (typical)")
print(f"  Data/strings section     : {shannon_entropy(unpacked_data):.2f} bits/byte  (normal)")

# YARA-like rule simulation
print("\n=== YARA Rule Match Simulation ===")

class YaraRule:
    def __init__(self, name, strings, condition_count):
        self.name = name
        self.strings = strings
        self.condition_count = condition_count

    def match(self, content: str) -> bool:
        matched = sum(1 for s in self.strings if s.lower() in content.lower())
        return matched >= self.condition_count

rules = [
    YaraRule("RansomwareStrings",
             ["your files have been encrypted", ".onion", "bitcoin", "decryption key"],
             condition_count=2),
    YaraRule("RemoteAccessTrojan",
             ["CreateRemoteThread", "VirtualAllocEx", "WriteProcessMemory"],
             condition_count=2),
    YaraRule("AntiAnalysis",
             ["IsDebuggerPresent", "CheckRemoteDebuggerPresent", "NtQueryInformationProcess"],
             condition_count=1),
]

samples = {
    "sample_a.exe": "CreateRemoteThread VirtualAllocEx WriteProcessMemory IsDebuggerPresent",
    "sample_b.exe": "Your files have been encrypted. Send bitcoin to .onion address for decryption key.",
    "sample_c.exe": "GetLastError HeapAlloc CreateFile CloseHandle",
}

for fname, content in samples.items():
    hits = [r.name for r in rules if r.match(content)]
    result = ", ".join(hits) if hits else "No match"
    print(f"  {fname}: {result}")

=== Simulated PE Import Analysis ===

  Total imports found: 10
  Import                       Suspicion
  ----------------------------------------------------------------------
  CreateRemoteThread           Process injection (T1055) <-- SUSPICIOUS
  VirtualAllocEx               Allocate memory in remote process <-- SUSPICIOUS
  WriteProcessMemory           Write to remote process memory <-- SUSPICIOUS
  RegSetValueEx                Registry persistence (T1547) <-- SUSPICIOUS
  WinExec                      Execute commands <-- SUSPICIOUS
  CryptEncrypt                 Encryption -- possible ransomware <-- SUSPICIOUS
  GetLastError                 Benign
  HeapAlloc                    Benign
  CreateFile                   Benign
  CloseHandle                  Benign

=== Entropy Analysis ===
  Packed/encrypted section : 7.96 bits/byte  (>7.5 = packed)
  Code section             : 6.98 bits/byte  (typical)
  Data/strings section     : 3.91 bits/byte  (normal)

=== YARA Rule Match Simulation ===
  sample_a.exe: RemoteAccessTrojan, AntiAnalysis
  sample_b.exe: RansomwareStrings
  sample_c.exe: No match

Review Questions (MCQ)#

Q1. A worm differs from a virus primarily in that a worm: A. Is harder to detect B. Does not require a host file and propagates autonomously C. Only targets Windows D. Always encrypts files

Q2. Double extortion ransomware combines encryption with: A. Phishing B. Exfiltration of data as a second ransom leverage C. DDoS D. Rootkit installation

Q3. High entropy (close to 8 bits/byte) in a PE section suggests: A. A legitimate executable B. The section is encrypted or packed C. The file is a PDF D. The file has no imports

Q4. The Windows API function pair most indicative of process injection is: A. RegSetValueEx + CreateFile B. VirtualAllocEx + CreateRemoteThread + WriteProcessMemory C. CryptEncrypt + FindFirstFile D. WinExec + GetProcAddress

Q5. YARA rules are used to: A. Decrypt malware B. Match file patterns to classify malware families C. Block network traffic D. Analyze network captures

Q6. A rootkit at the kernel level is best detected by: A. Running antivirus from within the compromised OS B. Inspection from a known-good external context (live boot, memory forensics) C. Checking the registry for Run keys D. Examining browser history

Q7. Domain Generation Algorithms (DGA) in C2 make blocking difficult because: A. They use HTTPS B. They generate thousands of potential C2 domains, making blocklists impractical C. They encrypt DNS traffic D. They run on port 443

Q8. Fileless malware evades detection by: A. Using encrypted executables B. Executing only in memory via legitimate system binaries (LOLBins) C. Disabling antivirus on startup D. Using rootkit techniques

Q9. INetSim in a malware analysis lab is used to: A. Capture network traffic B. Simulate internet services so malware can behave normally without reaching the real internet C. Unpack malware D. Write YARA rules

Q10. The ransomware attack chain step that precedes encryption is: A. Initial access B. Privilege escalation C. Disabling backups and shadow copies D. Exfiltration

Answers: Q1 B, Q2 B, Q3 B, Q4 B, Q5 B, Q6 B, Q7 B, Q8 B, Q9 B, Q10 C.

Lab Assignment#

Part A – Static analysis: Download a benign PE file (e.g., a portable app from a trusted source). Use strings (Linux) or CFF Explorer (Windows) to extract strings. Identify: any URLs, registry keys, or suspicious API imports. Compute its SHA-256 and query VirusTotal.

Part B – Sandbox submission: Submit a known benign file (or a purposely benign script you write) to any.run or Hybrid Analysis. Document the behavioral report: file activity, registry activity, network connections, and any detection verdicts. Explain what each reported behavior means.

Part C – YARA rule writing: Write three YARA rules: one for a hypothetical ransomware sample (based on the string patterns in the chapter), one for a RAT (based on import combinations), and one for an anti-debugging sample. Test each rule using the simulator above with sample strings you compose.

Part D – MITRE ATT&CK mapping: For a ransomware incident with the following behaviors (spear phishing initial access, PowerShell execution, LSASS dump, lateral movement via PsExec, Shadow Copy deletion, file encryption), map each behavior to its ATT&CK technique ID and tactic.

References#

NIST Computer Security Resource Center (CSRC) glossary: definition of malware.
Practical Computer Security (Course 3): lecture on Antivirus and Antimalware (history, detection generations, identification behaviors).
Email authentication standards SPF, DKIM, and DMARC; rootkit-detection tools (GMER, chkrootkit, rkhunter).
Microsoft Security Bulletin MS17-010; CVE-2017-0144 (EternalBlue / SMBv1). Reporting on the WannaCry outbreak (May 2017).
Sikorski, M., and Honig, A. Practical Malware Analysis. No Starch Press.
No More Ransom project (Europol European Cybercrime Centre, Dutch National Police, and partners). Crypto Sheriff and free decryption tools. https://www.nomoreransom.org
Check Point Research (2023). Rorschach: A New Sophisticated and Fast Ransomware.
UK National Crime Agency (2024). Operation Cronos: disruption of the LockBit ransomware group.
Reverse-engineering tools: Ghidra (NSA), IDA Pro (Hex-Rays), Binary Ninja, JADX, and Cutter (rizin).

Chapter 15: Malware Analysis

Contents

Chapter 15: Malware Analysis#

Learning Objectives#

Key Terms#

15.1 Malware Taxonomy#

Viruses and Worms#

Trojans and RATs#

Ransomware#

Ransomware Attack Chain#

Rootkits#

Botnets and C2#

15.2 Analysis Environment Setup#

Safe Lab Requirements#

REMnux and FlareVM#

15.3 Static Analysis#

File Identification#

PE Analysis#

YARA Rules#

Reverse Engineering Malware with Ghidra#

Choosing a Reverse-Engineering Tool#

15.4 Dynamic Analysis#

Behavioral Monitoring Tools#

Common Malware Behaviors to Watch#

15.5 Anti-Analysis and Evasion Techniques#

VM and Sandbox Detection#

Packers and Obfuscators#

Fileless Malware#

15.6 Malware Analysis Report Structure#

15.7 Antivirus and Antimalware Defenses#

The Anti-* Family: Beyond Antivirus#

15.8 A Field Guide to Malware Types#

15.9 The Malware Lifecycle and a Ransomware Deep Dive#

Notable Ransomware Strains: LockBit 3.0 and Rorschach#

Free Recovery: The No More Ransom Project#

15.10 Software Reverse Engineering in Depth#

The Reverse-Engineering Workflow#

An x86 and x86-64 Assembly Primer#

The PE Format and the Windows API#

Disassemblers, Decompilers, and Debuggers#

Code Injection: DLL Injection and Process Injection#

Obfuscation and Deobfuscation#

Anti-Analysis: Anti-Disassembly, Anti-Debugging, and Anti-VM#

Packing and Unpacking#

Shellcode, Buffer Overflows, and Network Attacks#

Exercises#

Answer Key#

15.11 An End-to-End Analysis Walkthrough#

15.12 Extracting Indicators of Compromise#

Exercises#

Answer Key#

15.13 Ransomware Internals and the Criminal Economy#

15.14 Rootkits, Bootkits, and Firmware Malware#

15.15 Mobile and Cross-Platform Malware#

Exercises#

Answer Key#

15.16 Fileless and Living-off-the-Land Malware#

15.17 Worms, Botnets, and Supply-Chain Malware#

15.18 Threat Intelligence and Malware Families#

Exercises#

Answer Key#

15.19 Case Study: Dissecting a Ransomware Sample#

Exercises#

Answer Key#

15.20 A YARA Rule Cookbook#

15.21 Building and Operating an Analysis Lab#

Exercises#

Answer Key#

15.22 Reading x86-64 Disassembly#

Exercises#

Answer Key#

15.23 Windows Internals for Malware Analysts#

Exercises#

Answer Key#

15.24 Obfuscation and Deobfuscation, Worked#

Exercises#

Answer Key#

15.25 A Dynamic Analysis Session#

Exercises#

Answer Key#