Calling Convention

syscall

You can think of syscalls as being like an API to the Operating System. They are how userland programs ask the kernel to perform certain actions. Specifically, shellcode usually performs a call to sys_execve (a member of the exec() syscall family). This syscall replaces the currently running program with a new one, in our case, /bin/sh. Like functions, syscalls also have a calling convention.

Calling Convention

To perform a syscall, we need to know its syscall number, then provide appropriate parameters for any arguments it requires. For x86-64 Linux, the convention is similar to regular function calls, with the addition of syscall number being passed in $rax:

• syscall number: $rax

• 1st parameter: $rdi

• 2nd parameter: $rsi

• 3rd parameter: $rdx

• 4th parameter: $r10

• 5th parameter: $r8

• 6th parameter: $9

• call int 0x80 (32-bit) or syscall (64-bit) when everything is ready.