Malware Analysis for Forensics

Static Analysis, Dynamic Analysis & IOC Extraction

Section 10 of 12

Malware Analysis Overview

Definition: Process of studying malicious software to understand its functionality, origin, and impact on systems.

Why Analyze Malware?

  • Understand Attack: What did the malware do?
  • Scope Assessment: What systems were affected?
  • Attribution: Who created it?
  • IOC Extraction: Create detection signatures
  • Mitigation: Remove and prevent reinfection
  • Evidence: Document for legal proceedings

Analysis Types

  • Static Analysis: Examine without execution
  • Dynamic Analysis: Run in controlled environment
  • Behavioral Analysis: Observe actions
  • Code Analysis: Reverse engineering
  • Memory Analysis: Runtime artifacts

CRITICAL SAFETY WARNING

NEVER analyze malware on production systems! Always use isolated virtual machines with no network access or use sandbox environments specifically designed for malware analysis.

Static Analysis

Examining malware without executing it - safe initial analysis step.

Basic Static Analysis

  • File Properties: Name, size, type
  • Hash Values: MD5, SHA-1, SHA-256
  • File Type: Identify true type (file, TrID)
  • Strings: Embedded text, URLs, IPs
  • Metadata: Compile time, compiler used
  • Packer Detection: UPX, Themida, etc.

Basic Commands

# Calculate hashes

md5sum malware.exe

sha256sum malware.exe

# Extract strings

strings malware.exe > strings.txt

strings -el malware.exe >> strings.txt # Unicode

# File type

file malware.exe

# Check if packed

detect-it-easy malware.exe

# PE analysis

pefile malware.exe

Advanced Static Analysis

  • PE Structure: Headers, sections, imports
  • Imports/Exports: Functions used
  • Resources: Embedded files, icons
  • Disassembly: Assembly code review
  • Decompilation: High-level code

Static Analysis Tools

  • PEStudio - PE file analyzer
  • Detect It Easy (DIE) - Packer detection
  • pestudio / PE-bear - PE viewers
  • IDA Free - Disassembler
  • Ghidra - NSA reverse engineering tool

Dynamic Analysis

Executing malware in a controlled environment to observe behavior.

Safe Execution Environment

Requirements

  • Isolated VM: No host interaction
  • Snapshot: Easy reversion
  • Network Isolated: No internet/LAN
  • Monitoring Tools: Pre-installed
  • Fake Network: INetSim, FakeDNS

Recommended VMs

  • REMnux - Linux malware analysis
  • FLARE VM - Windows malware analysis
  • Cuckoo Sandbox - Automated analysis
  • Any.run - Cloud sandbox

What to Monitor

  • File System: Created/modified/deleted files
  • Registry: New keys and values
  • Processes: Spawned processes
  • Network: Connections, DNS queries
  • API Calls: Function calls made
  • Mutexes: Synchronization objects

Dynamic Analysis Tools

  • Process Monitor - Real-time monitoring
  • Process Hacker - Process viewer
  • Wireshark - Network capture
  • Regshot - Registry comparison
  • Fakenet-NG - Network simulation
  • API Monitor - API call tracking

Behavioral Indicators

Common Malware Behaviors

  • Persistence: Registry Run keys, scheduled tasks, services
  • Defense Evasion: Disable AV, clear logs
  • Privilege Escalation: Exploit vulnerabilities
  • Discovery: System recon, network scanning
  • Lateral Movement: Spread to other systems
  • Collection: Keylogging, screenshot capture
  • C2 Communication: Beacon to command server
  • Exfiltration: Data theft

Registry Persistence Locations

HKCU\Software\Microsoft\Windows\

CurrentVersion\Run

HKLM\Software\Microsoft\Windows\

CurrentVersion\Run

HKCU\Software\Microsoft\Windows\

CurrentVersion\RunOnce

HKLM\System\CurrentControlSet\

Services

HKCU\Software\Microsoft\Windows NT\

CurrentVersion\Winlogon\Shell

File System Artifacts

  • Dropped Files: %TEMP%, %APPDATA%
  • Startup Folders: Autorun locations
  • Modified Files: System binaries replaced
  • Deleted Files: Cover tracks

Network Indicators

  • C2 Connections: External IPs
  • DNS Queries: Suspicious domains
  • HTTP(S) POST: Data exfiltration
  • Unusual Ports: Non-standard protocols

Sandboxing & Automated Analysis

Cuckoo Sandbox

Open-source automated malware analysis system

# Submit file for analysis

cuckoo submit malware.exe

# Submit with specific options

cuckoo submit --package exe --timeout 120 malware.exe

# Submit URL

cuckoo submit --url http://malicious.com/payload.exe

# View results

cuckoo web runserver

# Generate report

cuckoo report <task_id>

Online Sandboxes

  • Any.run - Interactive sandbox
  • Hybrid Analysis - Automated analysis
  • Joe Sandbox - Deep analysis
  • VirusTotal - Multi-AV scanning
  • Triage - Hatching.io platform

Warning: Results become public!

Sandbox Outputs

  • • Behavior summary
  • • Network traffic PCAP
  • • Dropped files
  • • Memory dumps
  • • API call logs
  • • Screenshots/video
  • • YARA/Suricata matches
  • • Malware family detection

IOC (Indicator of Compromise) Extraction

Extract artifacts that can be used to detect and hunt for malware across the network.

Network IOCs

  • IP Addresses: C2 servers, download sites
  • Domain Names: Malicious hostnames
  • URLs: Full request paths
  • User-Agents: Custom HTTP headers
  • SSL Certificates: Thumbprints
  • JA3 Hashes: TLS fingerprints

Host IOCs

  • File Hashes: MD5, SHA-1, SHA-256
  • File Names: Dropped files
  • File Paths: Installation locations
  • Registry Keys: Persistence mechanisms
  • Mutexes: Unique identifiers
  • Services: Installed services

Extract IOCs with Strings

# Extract IPs

strings malware.exe | grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}'

# Extract URLs

strings malware.exe | grep -oE 'https?://[^\s]+'

# Extract domains

strings malware.exe | grep -oE '[a-zA-Z0-9\-]+\.[a-zA-Z]{2,}'

# Extract file paths

strings malware.exe | grep -E '[A-Z]:\\.+'

YARA Rules for Malware Detection

YARA is a pattern-matching tool used to identify and classify malware based on textual or binary patterns.

Sample YARA Rule

rule Ransomware_Generic

{

meta:

description = "Detects generic ransomware indicators"

author = "Forensics Team"

date = "2025-12-03"

strings:

$s1 = "Your files have been encrypted" nocase

$s2 = "Bitcoin" nocase

$s3 = ".onion" ascii

$s4 = "ransom" nocase

$api1 = "CryptEncrypt" ascii

$api2 = "CryptGenKey" ascii

condition:

(2 of ($s*)) and (1 of ($api*))

}

Run YARA Rules

# Scan single file

yara rules.yar malware.exe

# Scan directory recursively

yara -r rules.yar /path/to/scan/

# Use rule set

yara -C rules/ suspicious_file

# Fast mode

yara -f rules.yar file

YARA Resources

  • Awesome YARA (GitHub) - Rule repository
  • YaraHub - Community rules
  • YARA Forge - Rule collections
  • Florian Roth's Rules - High-quality rules

Malware Analysis Tools

IDA Pro / IDA Free

Interactive disassembler

  • • Disassembly & decompilation
  • • Graph view
  • • Debugger integration
  • • Pro / Free version

Ghidra

NSA reverse engineering tool

  • • Decompiler
  • • Multi-platform support
  • • Scripting (Python/Java)
  • • Free, open-source

x64dbg

Open-source debugger for Windows

  • • 32-bit & 64-bit debugging
  • • Plugin support
  • • Scriptable
  • • Free

PE Explorer / PEStudio

Portable executable analysis

  • • PE structure viewer
  • • Suspicious indicator detection
  • • Resource extraction
  • • PEStudio free

Process Monitor

Sysinternals real-time monitoring

  • • File system activity
  • • Registry changes
  • • Network activity
  • • Free

Volatility

Memory forensics framework

  • • Analyze memory dumps
  • • Detect injected code
  • • Extract artifacts
  • • Free, open-source

Anti-Analysis Techniques

Malware often employs techniques to evade analysis and detection.

Obfuscation Methods

  • Packing/Compression: UPX, ASPack, Themida
  • Encryption: Encrypted strings, code
  • Polymorphism: Changes appearance each iteration
  • Metamorphism: Rewrites own code
  • Code Obfuscation: Junk code, control flow flattening

VM/Sandbox Detection

  • Hardware Checks: CPU count, RAM size
  • Artifacts: VMware/VBox files, registry keys
  • Timing Attacks: Delayed execution
  • User Interaction: Waits for mouse/keyboard
  • Domain Checks: Avoids analysis domains

Bypassing Techniques

  • Unpacking: Manual or automated unpacking tools
  • String Decryption: Runtime extraction
  • VM Evasion: Modify VM artifacts, use bare metal
  • Debugger Detection: Anti-anti-debugging plugins

Bypass Tools

  • UPX: Unpack UPX-packed files
  • de4dot: .NET deobfuscator
  • ScyllaHide: Anti-anti-debugging
  • Pafish: Test sandbox detection