Learning Malware Analysis

Learning Malware Analysis

by Monnappa K A
Released June 2018
Publisher(s): Packt Publishing
ISBN: 9781788392501

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Understand malware analysis and its practical implementation

About This Book
  • Explore the key concepts of malware analysis and memory forensics using real-world examples
  • Learn the art of detecting, analyzing, and investigating malware threats
  • Understand adversary tactics and techniques
Who This Book Is For

This book is for incident responders, cyber-security investigators, system administrators, malware analyst, forensic practitioners, student, or curious security professionals interested in learning malware analysis and memory forensics. Knowledge of programming languages such as C and Python is helpful but is not mandatory. If you have written few lines of code and have a basic understanding of programming concepts, you'll be able to get most out of this book.

What You Will Learn
  • Create a safe and isolated lab environment for malware analysis
  • Extract the metadata associated with malware
  • Determine malware's interaction with the system
  • Perform code analysis using IDA Pro and x64dbg
  • Reverse-engineer various malware functionalities
  • Reverse engineer and decode common encoding/encryption algorithms
  • Perform different code injection and hooking techniques
  • Investigate and hunt malware using memory forensics
In Detail

Malware analysis and memory forensics are powerful analysis and investigation techniques used in reverse engineering, digital forensics, and incident response. With adversaries becoming sophisticated and carrying out advanced malware attacks on critical infrastructures, data centers, and private and public organizations, detecting, responding to, and investigating such intrusions is critical to information security professionals. Malware analysis and memory forensics have become must-have skills to fight advanced malware, targeted attacks, and security breaches.

This book teaches you the concepts, techniques, and tools to understand the behavior and characteristics of malware through malware analysis. It also teaches you techniques to investigate and hunt malware using memory forensics.

This book introduces you to the basics of malware analysis, and then gradually progresses into the more advanced concepts of code analysis and memory forensics. It uses real-world malware samples, infected memory images, and visual diagrams to help you gain a better understanding of the subject and to equip you with the skills required to analyze, investigate, and respond to malware-related incidents.

Style and approach

The book takes the reader through all the concepts, techniques and tools to understand the behavior and characteristics of malware by using malware analysis and it also teaches the techniques to investigate and hunt malware using memory forensics.

Publisher resources

View/Submit Errata

Table of contents

  1. Title Page
  2. Copyright and Credits
    1. Learning Malware Analysis
  3. Dedication
  4. Packt Upsell
    1. Why subscribe?
    2. PacktPub.com
  5. Contributors
    1. About the author
    2. About the reviewers
    3. Packt is searching for authors like you
  6. Preface
    1. Who this book is for
    2. What this book covers
    3. To get the most out of this book
      1. Download the color images
      2. Conventions used
    4. Get in touch
      1. Reviews
  7. Introduction to Malware Analysis
    1. 1. What Is Malware?
    2. 2. What Is Malware Analysis?
    3. 3. Why Malware Analysis?
    4. 4. Types Of Malware Analysis
    5. 5. Setting Up The Lab Environment
      1. 5.1 Lab Requirements
      2. 5.2 Overview Of Lab Architecture
      3. 5.3 Setting Up And Configuring Linux VM
      4. 5.4 Setting Up And Configuring Windows VM
    6. 6. Malware Sources
    7. Summary
  8. Static Analysis
    1. 1. Determining the File Type
      1. 1.1 Identifying File Type Using Manual Method
      2. 1.2 Identifying File Type Using Tools
      3. 1.3 Determining File Type Using Python
    2. 2. Fingerprinting the Malware
      1. 2.1 Generating Cryptographic Hash Using Tools
      2. 2.2 Determining Cryptographic Hash in Python
    3. 3. Multiple Anti-Virus Scanning
      1. 3.1 Scanning the Suspect Binary with VirusTotal
      2. 3.2 Querying Hash Values Using VirusTotal Public API
    4. 4. Extracting Strings
      1. 4.1 String Extraction Using Tools
      2. 4.2 Decoding Obfuscated Strings Using FLOSS
    5. 5. Determining File Obfuscation
      1. 5.1 Packers and Cryptors
      2. 5.2 Detecting File Obfuscation Using Exeinfo PE
    6. 6. Inspecting PE Header Information
      1. 6.1 Inspecting File Dependencies and Imports
      2. 6.2  Inspecting Exports
      3. 6.3  Examining PE Section Table And Sections
      4. 6.4 Examining the Compilation Timestamp
      5. 6.5 Examining PE Resources
    7. 7. Comparing And Classifying The Malware
      1. 7.1 Classifying Malware Using Fuzzy Hashing
      2. 7.2 Classifying Malware Using Import Hash
      3. 7.3 Classifying Malware Using Section Hash
      4. 7.4 Classifying Malware Using YARA
        1. 7.4.1 Installing YARA
        2. 7.4.2 YARA Rule Basics
        3. 7.4.3 Running YARA
        4. 7.4.4 Applications of YARA
    8. Summary
  9. Dynamic Analysis
    1. 1. Lab Environment Overview
    2. 2. System And Network Monitoring
    3. 3. Dynamic Analysis (Monitoring) Tools
      1. 3.1 Process Inspection with Process Hacker
      2. 3.2 Determining System Interaction with Process Monitor
      3. 3.3 Logging System Activities Using Noriben
      4. 3.4 Capturing Network Traffic With Wireshark
      5. 3.5 Simulating Services with INetSim
    4. 4. Dynamic Analysis Steps
    5. 5. Putting it All Together: Analyzing a Malware Executable
      1. 5.1 Static Analysis of the Sample
      2. 5.2 Dynamic Analysis of the Sample
    6. 6. Dynamic-Link Library (DLL) Analysis
      1. 6.1 Why Attackers Use DLLs
      2. 6.2 Analyzing the DLL Using rundll32.exe
        1. 6.2.1 Working of rundll32.exe
        2. 6.2.2 Launching the DLL Using rundll32.exe
          1. Example 1 – Analyzing a DLL With No Exports
          2. Example 2 – Analyzing a DLL Containing Exports
          3. Example 3 – Analyzing a DLL Accepting Export Arguments
      3. 6.3 Analyzing a DLL with Process Checks
    7. Summary
  10. Assembly Language and Disassembly Primer
    1. 1. Computer Basics
      1. 1.1 Memory
        1. 1.1.1 How Data Resides In Memory
      2. 1.2 CPU
        1. 1.2.1 Machine Language
      3. 1.3 Program Basics
        1. 1.3.1 Program Compilation
        2. 1.3.2 Program On Disk
        3. 1.3.3 Program In Memory
        4. 1.3.4 Program Disassembly (From Machine code To Assembly code)
    2. 2. CPU Registers
      1. 2.1 General-Purpose Registers
      2. 2.2 Instruction Pointer (EIP)
      3. 2.3 EFLAGS Register
    3. 3. Data Transfer Instructions
      1. 3.1 Moving a Constant Into Register
      2. 3.2 Moving Values From Register To Register
      3. 3.3 Moving Values From Memory To Registers
      4. 3.4 Moving Values From Registers To Memory
      5. 3.5 Disassembly Challenge
      6. 3.6 Disassembly Solution
    4. 4. Arithmetic Operations
      1. 4.1 Disassembly Challenge
      2. 4.2 Disassembly Solution
    5. 5. Bitwise Operations
    6. 6. Branching And Conditionals
      1. 6.1 Unconditional Jumps
      2. 6.2 Conditional Jumps
      3. 6.3 If Statement
      4. 6.4 If-Else Statement
      5. 6.5 If-Elseif-Else Statement
      6. 6.6 Disassembly Challenge
      7. 6.7 Disassembly Solution
    7. 7. Loops
      1. 7.1 Disassembly Challenge
      2. 7.2 Disassembly Solution
    8. 8. Functions
      1. 8.1 Stack
      2. 8.2 Calling Function
      3. 8.3 Returning From Function
      4. 8.4 Function Parameters And Return Values
    9. 9. Arrays And Strings
      1. 9.1 Disassembly Challenge
      2. 9.2 Disassembly Solution
      3. 9.3 Strings
        1. 9.3.1 String Instructions
        2. 9.3.2 Moving From Memory To Memory (movsx)
        3. 9.3.3 Repeat Instructions (rep)
        4. 9.3.4 Storing Value From Register to Memory (stosx)
        5. 9.3.5 Loading From Memory to Register (lodsx)
        6. 9.3.6 Scanning Memory (scasx)
        7. 9.3.7 Comparing Values in Memory (cmpsx)
    10. 10. Structures
    11. 11. x64 Architecture
      1. 11.1 Analyzing 32-bit Executable On 64-bit Windows
    12. 12. Additional Resources
    13. 13. Summary
  11. Disassembly Using IDA
    1. 1. Code Analysis Tools
    2. 2. Static Code Analysis (Disassembly) Using IDA
      1. 2.1 Loading Binary in IDA
      2. 2.2 Exploring IDA Displays
        1. 2.2.1 Disassembly Window
        2. 2.2.2 Functions Window
        3. 2.2.3 Output Window
        4. 2.2.4 Hex View Window
        5. 2.2.5 Structures Window
        6. 2.2.6 Imports Window
        7. 2.2.7 Exports Window
        8. 2.2.8 Strings Window
        9. 2.2.9 Segments Window
      3. 2.3 Improving Disassembly Using IDA
        1. 2.3.1 Renaming Locations
        2. 2.3.2 Commenting in IDA
        3. 2.3.3 IDA Database
        4. 2.3.4 Formatting Operands
        5. 2.3.5 Navigating Locations
        6. 2.3.6 Cross-References
        7. 2.3.7 Listing All Cross-References
        8. 2.3.8 Proximity View And Graphs
    3. 3. Disassembling Windows API
      1. 3.1 Understanding Windows API
        1. 3.1.1 ANSI and Unicode API Functions
        2. 3.1.2 Extended API Functions
      2. 3.2 Windows API 32-Bit and 64-Bit Comparison
    4. 4. Patching Binary Using IDA
      1. 4.1 Patching Program Bytes
      2. 4.2 Patching Instructions
    5. 5. IDA Scripting and Plugins
      1. 5.1 Executing IDA Scripts
      2. 5.2 IDAPython
        1. 5.2.1 Checking The Presence Of CreateFile API
        2. 5.2.2 Code Cross-References to CreateFile Using IDAPython
      3. 5.3 IDA Plugins
    6. 6. Summary
  12. Debugging Malicious Binaries
    1. 1. General Debugging Concepts
      1. 1.1 Launching And Attaching To Processes
      2. 1.2 Controlling Process Execution
      3. 1.3 Interrupting a Program with Breakpoints
      4. 1.4 Tracing Program Execution
    2. 2. Debugging a Binary Using x64dbg
      1. 2.1 Launching a New Process in x64dbg
      2. 2.2 Attaching to an Existing Process Using x64dbg
      3. 2.3 x64dbg Debugger Interface
      4. 2.4 Controlling Process Execution Using x64dbg
      5. 2.5 Setting a Breakpoint in x64dbg
      6. 2.6 Debugging 32-bit Malware
      7. 2.7 Debugging 64-bit Malware
      8. 2.8 Debugging a Malicious DLL Using x64dbg
        1. 2.8.1 Using rundll32.exe to Debug the DLL in x64dbg
        2. 2.8.2 Debugging a DLL in a Specific Process
      9. 2.9 Tracing Execution in x64dbg
        1. 2.9.1 Instruction Tracing
        2. 2.9.2 Function Tracing
      10. 2.10 Patching in x64dbg
    3. 3. Debugging a Binary Using IDA
      1. 3.1 Launching a New Process in IDA
      2. 3.2 Attaching to an Existing Process Using IDA
      3. 3.3 IDA's Debugger Interface
      4. 3.4 Controlling Process Execution Using IDA
      5. 3.5 Setting a Breakpoint in IDA
      6. 3.6 Debugging Malware Executables
      7. 3.7 Debugging a Malicious DLL Using IDA
        1. 3.7.1 Debugging a DLL in a Specific Process
      8. 3.8 Tracing Execution Using IDA
      9. 3.9 Debugger Scripting Using IDAPython
        1. 3.9.1 Example – Determining Files Accessed by Malware
    4. 4. Debugging a .NET Application
    5. Summary
  13. Malware Functionalities and Persistence
    1. 1. Malware Functionalities
      1. 1.1 Downloader
      2. 1.2 Dropper
        1. 1.2.1 Reversing a 64-bit Dropper
      3. 1.3 Keylogger
        1. 1.3.1 Keylogger Using GetAsyncKeyState()
        2. 1.3.2 Keylogger Using SetWindowsHookEx()
      4. 1.4 Malware Replication Via Removable Media
      5. 1.5 Malware Command and Control (C2)
        1. 1.5.1 HTTP Command and Control
        2. 1.5.2 Custom Command and Control
      6. 1.6 PowerShell-Based Execution
        1. 1.6.1 PowerShell Command Basics
        2. 1.6.2 PowerShell Scripts And Execution Policy
        3. 1.6.2 Analyzing PowerShell Commands/Scripts
        4. 1.6.3 How Attackers Use PowerShell
    2. 2. Malware Persistence Methods
      1. 2.1 Running the Registry Key
      2. 2.2 Scheduled Tasks
      3. 2.3 Startup Folder
      4. 2.4 Winlogon Registry Entries
      5. 2.5 Image File Execution Options
      6. 2.6 Accessibility Programs
      7. 2.7 AppInit_DLLs
      8. 2.8 DLL Search Order Hijacking
      9. 2.9 COM hijacking
      10. 2.10 Service
    3. Summary
  14. Code Injection and Hooking
    1. 1. Virtual Memory
      1. 1.1 Process Memory Components (User Space)
      2. 1.2 Kernel Memory Contents (Kernel Space)
    2. 2. User Mode And Kernel Mode
      1. 2.1 Windows API Call Flow
    3. 3. Code Injection Techniques
      1. 3.1 Remote DLL Injection
      2. 3.2 DLL Injection Using APC (APC Injection)
      3. 3.3 DLL Injection Using SetWindowsHookEx()
      4. 3.4 DLL Injection Using The Application Compatibility Shim
        1. 3.4.1 Creating A Shim
        2. 3.4.2 Shim Artifacts
        3. 3.4.3 How Attackers Use Shims
        4. 3.4.4 Analyzing The Shim Database
      5. 3.5 Remote Executable/Shellcode Injection
      6. 3.6 Hollow Process Injection (Process Hollowing)
    4. 4. Hooking Techniques
      1. 4.1 IAT Hooking
      2. 4.2 Inline Hooking (Inline Patching)
      3. 4.3 In-memory Patching Using Shim
    5. 5. Additional Resources
    6. Summary
  15. Malware Obfuscation Techniques
    1. 1. Simple Encoding
      1. 1.1 Caesar Cipher
        1. 1.1.1 Working Of Caesar Cipher
        2. 1.1.2 Decrypting Caesar Cipher In Python
      2. 1.2 Base64 Encoding
        1. 1.2.1 Translating Data To Base64
        2. 1.2.2 Encoding And Decoding Base64
        3. 1.2.3 Decoding Custom Base64
        4. 1.2.4 Identifying Base64
      3. 1.3 XOR Encoding
        1. 1.3.1 Single Byte XOR
        2. 1.3.2 Finding XOR Key Through Brute-Force
        3. 1.3.3 NULL Ignoring XOR Encoding
        4. 1.3.4 Multi-byte XOR Encoding
        5. 1.3.5 Identifying XOR Encoding
    2. 2. Malware Encryption
      1. 2.1 Identifying Crypto Signatures Using Signsrch
      2. 2.2 Detecting Crypto Constants Using FindCrypt2
      3. 2.3 Detecting Crypto Signatures Using YARA
      4. 2.4 Decrypting In Python
    3. 3. Custom Encoding/Encryption
    4. 4. Malware Unpacking
      1. 4.1 Manual Unpacking
        1. 4.1.1 Identifying The OEP
        2. 4.1.2 Dumping Process Memory With Scylla
        3. 4.1.3 Fixing The Import Table
      2. 4.2 Automated Unpacking
    5. Summary
  16. Hunting Malware Using Memory Forensics
    1. 1. Memory Forensics Steps
    2. 2. Memory Acquisition
      1. 2.1 Memory Acquisition Using DumpIt
    3. 3. Volatility Overview
      1. 3.1 Installing Volatility
        1. 3.1.1 Volatility Standalone Executable
        2. 3.1.2 Volatility Source Package
      2. 3.2 Using Volatility
    4. 4. Enumerating Processes
      1. 4.1 Process Overview
        1. 4.1.1 Examining the _EPROCESS Structure
        2. 4.1.2 Understanding ActiveProcessLinks
      2. 4.2 Listing Processes Using psscan
        1. 4.2.1 Direct Kernel Object Manipulation (DKOM)
        2. 4.2.2 Understanding Pool Tag Scanning
      3. 4.3 Determining Process Relationships
      4. 4.4 Process Listing Using psxview
    5. 5. Listing Process Handles
    6. 6. Listing DLLs
      1. 6.1 Detecting a Hidden DLL Using ldrmodules
    7. 7. Dumping an Executable and DLL
    8. 8. Listing Network Connections and Sockets
    9. 9. Inspecting Registry
    10. 10. Investigating Service
    11. 11. Extracting Command History
    12. Summary
  17. Detecting Advanced Malware Using Memory Forensics
    1. 1. Detecting Code Injection
      1. 1.1 Getting VAD Information
      2. 1.2 Detecting Injected Code Using VAD
      3. 1.3 Dumping The Process Memory Region
      4. 1.4 Detecting Injected Code Using malfind
    2. 2. Investigating Hollow Process Injection
      1. 2.1 Hollow Process Injection Steps
      2. 2.2 Detecting Hollow Process Injection
      3. 2.3 Hollow Process Injection Variations
    3. 3. Detecting API Hooks
    4. 4. Kernel Mode Rootkits
    5. 5. Listing Kernel Modules
      1. 5.1 Listing Kernel Modules Using driverscan
    6. 6. I/O Processing
      1. 6.1 The Role Of The Device Driver
      2. 6.2 The Role Of The I/O Manager
      3. 6.3 Communicating With The Device Driver
      4. 6.4 I/O Requests To Layered Drivers
    7. 7. Displaying Device Trees
    8. 8. Detecting Kernel Space Hooking
      1. 8.1 Detecting SSDT Hooking
      2. 8.2 Detecting IDT Hooking
      3. 8.3 Identifying Inline Kernel Hooks
      4. 8.4 Detecting IRP Function Hooks
    9. 9. Kernel Callbacks And Timers
    10. Summary
  18. Other Books You May Enjoy
    1. Leave a review - let other readers know what you think

Product information

  • Title: Learning Malware Analysis
  • Author(s): Monnappa K A
  • Release date: June 2018
  • Publisher(s): Packt Publishing
  • ISBN: 9781788392501