Syscalls and Inline Hooking

We might find ourselves invoking Kernel Mode syscalls like CreateProcess or VirtualAlloc. More often than not, security solutions are wary of these calls and monitor them closely. To facilitate effective protection and offensive maneuvers, we must understand the EDR hooks and Windows internals. Understanding how Windows facilitates Kernel Mode transitions and how EDR integrates itself into that flow—allows us to engineer a bypass.

The Architecture of the Windows Call Stack

Windows operates on a system of Protection Rings. Most applications reside in User Mode (Ring 3), while core operating system functions reside in Kernel Mode (Ring 0). Precious Kernel

Why target Native APIs?

Native APIs are the gateway to the system. They allow us to:

Manage and Allocate Memory: Using functions like VirtualAlloc and VirtualAllocEx.
Perform Process Injection & Hollowing: Leveraging NtCreateUserProcess, NtGetContextThread, and NtSetContextThread.
Execute Code: Utilizing NtCreateThreadEx.

Because these functions provide the "keys to the kingdom," they are heavily scrutinized and monitored by EDRs. One primary method of evasion is through alternative syscall implementations. In this post, we will analyze how the OS handles these APIs, examine the mechanisms EDRs use to detect them, and finally, walk through how we can bypass these controls.

How Windows Handles Syscalls

Regardless of which side of the battlefield you are on, you first need to understand the terrain. In our situation, this translates to understanding how Windows handles syscalls to achieve our goals and how a normal process functions. Whenever a process calls one of these Native APIs such as VirtualAlloc in modern Windows, kernelbase.dll is invoked. The request is then passed to ntdll.dll, where the syscalls reside. The DLL itself handles the call on our behalf because our process lacks the necessary privileges to interact with the kernel directly. The diagram illustrates this flow clearly. For a deeper dive, you can read the full blog here: Endpoint Detection and Response: How Hackers Have Evolved

USER MODE (Ring 3)              |          KERNEL MODE (Ring 0)
                                       |
+------------------------------+       |
|      Running Application     |       |
+--------------+---------------+       |
               |                       |
               | (1) Calls Function    |
               v                       |
+--------------+---------------+       |
|           Kernel32           |       |
|       (VirtualAlloc)         |       |
+--------------+---------------+       |
               |                       |
               | (2) Calls Syscall     |
               v                       |
+--------------+---------------+       |       +----------------------------+
|            Ntdll             +-------------->|      Kernel / Hardware     |
|  (NtAllocateVirtualMemory)   |  (3) Syscall  | (Allocates Physical Memory)|
+------------------------------+       |       +----------------------------+
                                       |

You might ask: how does the EDR actually know when these syscalls are executed, and where does it fit into the picture? It functions by acting as a proxy between the user and the kernel through API hooking.

                USER MODE
┌─────────────────────────────────────┐
│  ┌───────────────────────────────┐  │
│  │  Application / Process        │  │
│  └───────────────┬───────────────┘  │
│                  ▼                   │
│  ┌───────────────────────────────┐  │
│  │  WinAPI call (VirtualAlloc)   │  │
│  └───────────────┬───────────────┘  │
│                  ▼                   │
│  ┌───────────────────────────────┐  │
│  │  ntdll.dll — syscall stub     │  │
│  └───────────────┬───────────────┘  │
│                  ▼                   │
│  ╔═══════════════════════════════╗  │
│  ║  EDR hook (inline patch)      ║ ◄── injected
│  ╠═══════════════════════════════╣  │
│  ║  EDR agent DLL                ║  │
│  ║   • inspect arguments         ║  │
│  ║   • collect telemetry         ║  │
│  ║   • apply detection logic     ║  │
│  ╚═══════════════╤═══════════════╝  │
│                  ▼  (resume flow)    │
│  ┌───────────────────────────────┐  │
│  │  ntdll.dll — issues syscall   │  │
│  │  (Nt → Zw transition)         │  │
│  └───────────────┬───────────────┘  │
└──────────────────┼───────────────────┘
                   ▼
          ── kernel boundary ──
                   ▼
┌─────────────────────────────────────┐
│               KERNEL MODE            │
│  ┌───────────────────────────────┐  │
│  │  Windows kernel executes      │  │
│  │  the memory / OS operation    │  │
│  └───────────────────────────────┘  │
└─────────────────────────────────────┘

In a standard, unmonitored environment, a syscall is a straightforward . The process prepares the necessary registers and then hands control over to the kernel. At the assembly level within ntdll.dll, a clean Native API (such as NtAllocateVirtualMemory) looks like this:

mov r10, rcx
mov eax, 18h            ; NtAllocateVirtualMemory
syscall                 ; Kernel Mode
ret                     ; Return

EDRs employ a smart technique: they inject code into memory so that when a process is loaded, they overwrite the first few bytes of a function with a Jump (JMP) instruction. This redirects the CPU to the EDR's own memory space (e.g., EdrAgent.dll) for inspection before the original function can execute.

jmp 0x00007FF812345678  ; 5-byte JMP instruction to EDR's inspection engine
mov r10, rcx
mov eax, 18h            ; NtAllocateVirtualMemory
syscall                 ; Kernel Mode
ret                     ; Return

Visualizing Syscall Flow in x64dbg

Let's create a simple program that allocates 1 KB of memory with Read/Write permissions using VirtualAlloc:

VirtualAlloc(
    NULL,
    1024,              // Allocate 1KB
    MEM_COMMIT,        // Commit the memory
    PAGE_READWRITE     // Set permissions to Read/Write
);

To understand the underlying mechanics, we will observe this process inside x64dbg. We'll start by setting a breakpoint at the moment the API call is initiated to see how the execution flow transitions through the Windows subsystems. Let's follow the execution flow of this function. By pressing F7 to step into the call, we can see the redirection to KernelBase.dll, as illustrated below. Within this module, we encounter a call to NtAllocateVirtualMemory, which is the underlying Native API residing in ntdll.dll. Tracing further, we arrive at the underlying function within ntdll.dll which is the lowest-level layer of the Windows user-mode environment, this is usually where user-mode functions transition into kernel-mode system calls via $syscall$ instructions. ntdll

Note: As you can see, our call isn’t being hooked by an EDR. This is because I am running on a machine with the default Microsoft Defender Antivirus which comes pre-installed on every Windows device rather than a full EDR suite with advanced behavioral monitoring capabilities. To demonstrate how an EDR would behave, we will use either OpenEDR or Microsoft Detours to observe these hooks in action and develop a method to bypass. And will be doing that in the next blog.