Stack Frame

Motivation: Function Call

Before functionA calls functionB, we have to save some information of functionA in order to recover its context when it returns:

• Function arguments

• Return address (the next RIP address when functionA returns)

• Register values

In OS, this functionality is implemented using stack.

Stack

The stack is typically structured as a linear sequence of memory allocations known as stack frames. Each time a function is called, the stack will automatically allocate a new stack frame. As the function executes, it will use the given stack frame to store and operate upon its local variables. Once the function returns, this memory will automatically get released back to the stack.

The stack fulfils four main uses:

• Track the "callstack" of a program. return values are "pushed" to the stack during a call and "popped" during a ret

• Contain local variables of functions.

• Provide scratch space (to alleviate register exhaustion).

• Pass function arguments (always on x86, only for functions with "many" arguments on other architectures).

Relevant registers (amd64):

• rsp

• rbp

Relevant instructions (amd64)

• push

• pop

Calling Convention

The cdecl (which stands for C declaration) is a calling convention that originates from Microsoft's compiler for the C programming language and is used by many C compilers for the x86 architecture. In cdecl, subroutine arguments are passed on the stack.

Key Ideas:

• Arguments are pushed onto the stack from right to left.

• Return value is stored in $rax.

Also, the calling convention specifies stack alignment. Read this section to learn more:

Stack Alignment

Caller vs. Callee

Consider a vulnerable C code snippet:

int foo(int arg1, int arg2, int arg3)
{
    char buffer[64] = {0};
    gets(buffer);
    return 0;
}

Suppose the main function calls foo(), then the main function is the caller and foo() is the callee. When function call is happening, the caller is going to:

1. Push arg3 onto the stack.

2. Push arg2 onto the stack.

3. Push arg1 onto the stack.

4. Push return address (the address of the next instruction when the callee returns) onto the stack, so the CPU knows what to do next when the callee returns.

5. Push old rbp (the rbp of the caller) onto the stack, so that the caller can reconstruct its stack frame when the callee returns.

Pictorially:

|      ...       | Low address
|     buffer     |
|      ...       |
|     old rbp    |
| return address |
|     arg1       |
|     arg2       |
|     arg3       | High address

Prologue vs. Epilogue

Prologue

Epilogue

Reference

x86 calling conventionsWikipedia

x86 calling convention - Wikipedia