Mastering Malware Analysis - Second Edition

Mastering Malware Analysis - Second Edition

by Alexey Kleymenov, Amr Thabet
Released September 2022
Publisher(s): Packt Publishing
ISBN: 9781803240244

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Learn effective malware analysis tactics to prevent your systems from getting infected

Key Features

  • Investigate cyberattacks and prevent malware-related incidents from occurring in the future
  • Learn core concepts of static and dynamic malware analysis, memory forensics, decryption, and much more
  • Get practical guidance in developing efficient solutions to handle malware incidents

Book Description

New and developing technologies inevitably bring new types of malware with them, creating a huge demand for IT professionals that can keep malware at bay. With the help of this updated second edition of Mastering Malware Analysis, you'll be able to add valuable reverse-engineering skills to your CV and learn how to protect organizations in the most efficient way.

This book will familiarize you with multiple universal patterns behind different malicious software types and teach you how to analyze them using a variety of approaches.

You'll learn how to examine malware code and determine the damage it can possibly cause to systems, along with ensuring that the right prevention or remediation steps are followed. As you cover all aspects of malware analysis for Windows, Linux, macOS, and mobile platforms in detail, you'll also get to grips with obfuscation, anti-debugging, and other advanced anti-reverse-engineering techniques. The skills you acquire in this cybersecurity book will help you deal with all types of modern malware, strengthen your defenses, and prevent or promptly mitigate breaches regardless of the platforms involved.

By the end of this book, you will have learned how to efficiently analyze samples, investigate suspicious activity, and build innovative solutions to handle malware incidents.

What you will learn

  • Explore assembly languages to strengthen your reverse-engineering skills
  • Master various file formats and relevant APIs used by attackers
  • Discover attack vectors and start handling IT, OT, and IoT malware
  • Understand how to analyze samples for x86 and various RISC architectures
  • Perform static and dynamic analysis of files of various types
  • Get to grips with handling sophisticated malware cases
  • Understand real advanced attacks, covering all their stages
  • Focus on how to bypass anti-reverse-engineering techniques

Who this book is for

If you are a malware researcher, forensic analyst, IT security administrator, or anyone looking to secure against malicious software or investigate malicious code, this book is for you. This new edition is suited to all levels of knowledge, including complete beginners. Any prior exposure to programming or cybersecurity will further help to speed up your learning process.

Table of contents

  1. Mastering Malware Analysis Second Edition
  2. Contributors
  3. About the authors
  4. About the reviewer
  5. Preface
    1. Who this book is for
    2. What this book covers
    3. To get the most out of this book
    4. Download the example code files
    5. Download the color images
    6. Conventions used
    7. Get in touch
    8. Share Your Thoughts
  6. Part 1 Fundamental Theory
  7. Chapter 1: Cybercrime, APT Attacks, and Research Strategies
    1. Why malware analysis?
      1. Malware analysis in collecting threat intelligence
      2. Malware analysis in incident response
      3. Malware analysis in threat hunting
      4. Malware analysis in creating detections
    2. Exploring types of malware
      1. A short history of malware development
      2. Malware categories
      3. Naming conventions
    3. The MITRE ATT&CK framework explained
      1. Basic terminology
      2. Enterprise Matrix
    4. APT and zero-day attacks and fileless malware
      1. APT attack
      2. Zero-day attack
      3. Fileless malware
    5. Choosing your analysis strategy
      1. Understand your audience
      2. Answer your audience’s questions
      3. Define your goals
      4. Avoid unnecessary technical details
      5. Example structures
      6. Typical analysis workflow
    6. Setting up the environment
      1. Choosing the virtualization software
      2. Safety features
    7. Summary
  8. Chapter 2: A Crash Course in Assembly and Programming Basics
    1. Basics of informatics
      1. Numeral systems
      2. Basic data units and data types
      3. Bitwise operations
    2. Architectures and their assembly
      1. Registers
      2. Memory
      3. Instructions (CISC and RISC)
    3. Becoming familiar with x86 (IA-32 and x64)
      1. Registers
      2. The instruction structure
      3. The instruction set
      4. Arguments, local variables, and calling conventions (in x86 and x64)
    4. Exploring ARM assembly
      1. Basics
      2. Instruction sets
    5. Basics of MIPS
      1. Basics
      2. The instruction set
    6. Diving deep into PowerPC
      1. Basics
      2. The instruction set
    7. Covering the SuperH assembly
      1. Basics
      2. The instruction set
    8. Working with SPARC
      1. Basics
      2. The instruction set
    9. Moving from assembly to high-level programming languages
      1. Arithmetic statements
      2. If conditions
      3. While loop conditions
    10. Summary
  9. Part 2 Diving Deep into Windows Malware
  10. Chapter 3: Basic Static and Dynamic Analysis for x86/x64
    1. Working with the PE header structure
      1. Why PE?
      2. Exploring PE’s structure
      3. PE+ (x64 PE)
      4. PE header analysis tools
    2. Static and dynamic linking
      1. Static linking
      2. Dynamic linking
      3. Dynamic link libraries
      4. Application programming interface (API)
    3. Using PE header information for static analysis
      1. How to use the PE header for incident handling
      2. How to use a PE header for threat hunting
    4. PE loading and process creation
      1. Basic terminology
      2. Process creation step by step
      3. PE file loading step by step
      4. WOW64 processes
    5. Basics of dynamic analysis using OllyDbg and x64dbg
      1. Debugging tools
      2. How to analyze a sample with OllyDbg
      3. Types of breakpoints
      4. Modifying the program’s execution
      5. List strings, APIs, and cross-references
      6. Setting labels and comments
      7. Differences between OllyDbg and x64dbg
    6. Debugging malicious services
      1. What is a service?
      2. Attaching to services
    7. Essentials of behavioral analysis
      1. File operations
      2. Registry operations
      3. Process operations
      4. WinAPIs
      5. Network activity
      6. Sandboxes
    8. Summary
  11. Chapter 4: Unpacking, Decryption, and Deobfuscation
    1. Exploring packers
      1. Exploring packing and encrypting tools
    2. Identifying a packed sample
      1. Technique 1 – using static signatures
      2. Technique 2 – evaluating PE section names
      3. Technique 3 – using stub execution signs
      4. Technique 4 – detecting a small import table
    3. Automatically unpacking packed samples
      1. Technique 1 – the official unpacking process
      2. Technique 2 – using OllyScript with OllyDbg
      3. Technique 3 – using generic unpackers
      4. Technique 4 – emulation
      5. Technique 5 – memory dumps
    4. Manual unpacking techniques
      1. Technique 1 – memory breakpoint on execution
      2. Technique 2 – call stack backtracing
      3. Technique 3 – monitoring memory allocated spaces for unpacked code
      4. Technique 4 – in-place unpacking
      5. Technique 5 – searching for and transferring control to OEP
      6. Technique 6 – stack restoration-based
    5. Dumping the unpacked sample and fixing the import table
      1. Dumping the process
      2. Fixing the import table
    6. Identifying simple encryption algorithms and functions
      1. Types of encryption algorithms
      2. Basic encryption algorithms
      3. Identifying encryption functions in disassembly
      4. String search detection techniques for simple algorithms
      5. Identifying the RC4 encryption algorithm
    7. Advanced symmetric and asymmetric encryption algorithms
      1. Extracting information from Windows cryptography APIs
      2. Cryptography API: Next Generation (CNG)
    8. Applications of encryption in modern malware – Vawtrak banking Trojan
      1. String and API name encryption
      2. Network communication encryption
    9. Using IDA for decryption and unpacking
      1. IDA tips and tricks
      2. Classic and new syntax of IDA scripts
      3. Dynamic string decryption
      4. Dynamic WinAPIs resolution
    10. Summary
  12. Chapter 5: Inspecting Process Injection and API Hooking
    1. Understanding process injection
      1. What’s process injection?
      2. Why process injection?
    2. DLL injection
      1. Windows-supported DLL injection
      2. A simple DLL injection technique
    3. Diving deeper into process injection
      1. Finding the victim process
      2. Code block injection
      3. Reflective DLL injection
      4. Stuxnet secret technique – process hollowing
    4. A dynamic analysis of code injection
      1. Technique 1 – Debug it where it is
      2. Technique 2 – Attach to the targeted process
      3. Technique 3 – Dealing with process hollowing
    5. Memory forensics techniques for process injection
      1. Technique 1 – Detecting code injection and reflective DLL injection
      2. Technique 2 – Detecting process hollowing
      3. Technique 3 – Detecting process hollowing using the HollowFind plugin
    6. Understanding API hooking
      1. Why API hooking?
      2. Working with API hooking
      3. Detecting API hooking using memory forensics
    7. Exploring IAT hooking
    8. Summary
  13. Chapter 6: Bypassing Anti-Reverse Engineering Techniques
    1. Exploring debugger detection
      1. Using PEB information
      2. Using EPROCESS information
      3. Using DebugObject
      4. Using handles
      5. Using exceptions
      6. Using parent processes
    2. Handling the evasion of debugger breakpoints
      1. Detecting software breakpoints (INT3)
      2. Detecting single-stepping breakpoints using a trap flag
      3. Detecting single-stepping using timing techniques
      4. Evading hardware breakpoints
      5. Memory breakpoints
    3. Escaping the debugger
      1. Process injection
      2. TLS callbacks
      3. Windows events callbacks
      4. Attacking the debugger
    4. Understanding obfuscation and anti-disassemblers
      1. Encryption
      2. Junk code
      3. Code transportation
      4. Dynamic API calling with checksum
      5. Proxy functions and proxy argument stacking
      6. Using the COM functionality
    5. Detecting and evading behavioral analysis tools
      1. Finding the tool process
      2. Searching for the tool window
    6. Detecting sandboxes and VMs
      1. Different output between VMs and real machines
      2. Detecting virtualization processes and services
      3. Detecting virtualization through registry keys
      4. Detecting VMs using WMI
      5. Other VM detection techniques
      6. Detecting sandboxes using default settings
    7. Summary
  14. Chapter 7: Understanding Kernel-Mode Rootkits
    1. Kernel mode versus user mode
      1. Protection rings
    2. Windows internals
      1. The anatomy of Windows
      2. The execution path from user mode to kernel mode
    3. Rootkits and device drivers
      1. What is a rootkit?
      2. Types of rootkits
      3. What is a device driver?
    4. Hooking mechanisms
      1. Hooking the SYSENTER entry function
      2. Modifying SSDT in an x86 environment
      3. Modifying SSDT in an x64 environment
      4. Patching SSDT functions
      5. IRP hooking
    5. DKOM
      1. The kernel objects – EPROCESS and ETHREAD
      2. How do rootkits perform an object manipulation attack?
    6. Process injection in kernel mode
      1. Executing the inject code using APC queuing
    7. KPP in x64 systems (PatchGuard)
      1. Bypassing driver signature enforcement
      2. Bypassing PatchGuard – the Turla example
      3. Bypassing PatchGuard – GhostHook
    8. Static and dynamic analysis in kernel mode
      1. Static analysis
      2. Dynamic and behavioral analysis
      3. Setting up a testing environment
      4. Setting up the debugger
      5. Stopping at the driver's entry point
      6. Loading the driver
      7. Restoring the debugging state
    9. Summary
  15. Part 3 Examining Cross-Platform and Bytecode-Based Malware
  16. Chapter 8: Handling Exploits and Shellcode
    1. Getting familiar with vulnerabilities and exploits
      1. Types of vulnerabilities
      2. Types of exploits
    2. Cracking the shellcode
      1. What’s shellcode?
      2. Linux shellcode in x86-64
      3. Linux shellcode for ARM
      4. Windows shellcode
      5. Static and dynamic analysis of exploits
    3. Exploring bypasses for exploit mitigation technologies
      1. Data execution prevention (DEP/NX)
      2. Return-oriented programming
      3. Address space layout randomization
      4. Other mitigation technologies
    4. Analyzing Microsoft Office exploits
      1. File structures
      2. Static and dynamic analysis of MS Office exploits
    5. Studying malicious PDFs
      1. File structure
      2. Static and dynamic analysis of PDF files
    6. Summary
  17. Chapter 9: Reversing Bytecode Languages – .NET, Java, and More
    1. The basic theory of bytecode languages
      1. Object-oriented programming
      2. Inheritance
      3. Polymorphism
    2. .NET explained
      1. .NET file structure
      2. How to identify a .NET application from PE characteristics
      3. The CIL language instruction set
      4. CIL language into higher-level languages
    3. .NET malware analysis
      1. .NET analysis tools
      2. Static and dynamic analysis
      3. Dealing with obfuscation
    4. The essentials of Visual Basic
      1. File structure
      2. P-code versus native code
      3. Common p-code instructions
    5. Dissecting Visual Basic samples
      1. Static analysis
      2. Dynamic analysis
    6. The internals of Java samples
      1. File structure
      2. JVM instructions
      3. Static analysis
      4. Dynamic analysis
      5. Dealing with anti-reverse engineering solutions
    7. Analyzing compiled Python threats
      1. File structure
      2. Bytecode instructions
      3. Static analysis
      4. Dynamic analysis
    8. Summary
  18. Chapter 10: Scripts and Macros – Reversing, Deobfuscation, and Debugging
    1. Classic shell script languages
      1. Windows batch scripting
      2. Bash
    2. VBScript explained
      1. Basic syntax
      2. Static and dynamic analysis
      3. Deobfuscation
    3. VBA and Excel 4.0 (XLM) macros and more
      1. VBA macros
      2. Excel 4.0 (XLM) macros
      3. Besides macros
    4. The power of PowerShell
      1. Basic syntax
      2. Obfuscation
      3. Static and dynamic analysis
    5. Handling JavaScript
      1. Basic syntax
      2. Anti-reverse engineering tricks
      3. Static and dynamic analysis
    6. Behind C&C – even malware has its own backend
      1. Things to focus on
      2. Static and dynamic analysis
    7. Other script languages
      1. Where to start
      2. Questions to answer
    8. Summary
  19. Part 4 Looking into IoT and Other Platforms
  20. Chapter 11: Dissecting Linux and IoT Malware
    1. Explaining ELF files
      1. The ELF structure
      2. System calls
    2. Exploring common behavioral patterns
      1. Initial access and lateral movement
      2. Persistence
      3. Privilege escalation
      4. Command and control
      5. Impact
      6. Defense evasion
    3. Static and dynamic analysis of x86 (32- and 64-bit) samples
      1. Static analysis
      2. Dynamic analysis
      3. A radare2 cheat sheet
    4. Learning about Mirai, its clones, and more
      1. High-level functionality
      2. Later derivatives
      3. Other widespread families
    5. Static and dynamic analysis of RISC samples
      1. ARM
      2. MIPS
      3. PowerPC
      4. SuperH
      5. SPARC
    6. Handling other architectures
      1. What to start from
    7. Summary
  21. Chapter 12: Introduction to macOS and iOS Threats
    1. Understanding the role of the security model
      1. macOS
      2. Other technologies
      3. iOS
    2. File formats and APIs
      1. Mach-O
      2. Application bundles (.app)
      3. Installer packages (.pkg)
      4. Apple disk images (.dmg)
      5. iOS app store packages (.ipa)
      6. APIs
    3. Attack stages
      1. Jailbreaks on demand
      2. Initial access
      3. Execution and persistence
      4. Impact
      5. Other attack techniques
    4. Advanced techniques
      1. Anti-analysis and detection tricks
      2. Misusing dynamic data exchange (DDE)
      3. User hiding
      4. Using AppleScript
      5. API hijacking
      6. Other techniques
      7. Rootkits for Mac – do they exist?
    5. Static and dynamic analysis of macOS and iOS samples
      1. Static analysis
      2. Dynamic and behavioral analysis
    6. The analysis workflow
    7. Summary
  22. Chapter 13: Analyzing Android Malware Samples
    1. (Ab)using the Android internals
      1. The file hierarchy
      2. The Android security model
      3. To root or not to root?
    2. Understanding Dalvik and ART
      1. Dalvik VM (DVM)
      2. Android runtime (ART)
      3. The bytecode set
    3. File formats and APIs
      1. DEX
      2. ODEX
      3. OAT
      4. VDEX
      5. ART
      6. ELF
      7. APK
      8. APIs
    4. Malware behavior patterns
      1. Initial access
      2. Privilege escalation
      3. Persistence
      4. Impact
      5. Collection
      6. Defence evasion
    5. Static and dynamic analysis of threats
      1. Static analysis
      2. Dynamic analysis
      3. Behavioral analysis and tracing
      4. The analysis workflow
    6. Summary
  23. Index
    1. Why subscribe?
  24. Other Books You May Enjoy
    1. Packt is searching for authors like you

Product information

  • Title: Mastering Malware Analysis - Second Edition
  • Author(s): Alexey Kleymenov, Amr Thabet
  • Release date: September 2022
  • Publisher(s): Packt Publishing
  • ISBN: 9781803240244