Calling conventions and the stack frame

When one function calls another, the two have to agree on where the arguments go and where the result comes back. That agreement is called a calling convention. Programs on x86-64 Linux and macOS follow the System V convention:

• The first six arguments go in registers, in this order: rdi, then rsi, then rdx, then rcx, then r8, then r9.
• Any arguments beyond the first six are passed on the stack.
• The result comes back in rax.

Windows has a different convention that starts with rcx, so first find out which system the program you're reading was built for.

The stack is memory that grows toward lower addresses, and each function reserves a stack frame in it for its local variables. Without optimization, a compiler usually starts a function with two instructions: push rbp saves the frame base of the function that called it, and mov rbp, rsp makes the current top of the stack the base of the new frame. After that, local variables are read at negative offsets from rbp, like [rbp-4], where the square brackets mean the value stored at that address.

     higher addresses
+----------------+
| return address |  [rbp+8]
+----------------+
| saved rbp      |  [rbp]
+----------------+
| a              |  [rbp-4]
| b              |  [rbp-8]
+----------------+
     lower addresses

• [rbp+8] holds the return address that the call instruction pushed.
• [rbp] holds the frame base of the calling function.
• [rbp-4] and [rbp-8] hold the arguments a and b, which the function copied there from edi and esi.

Here's the function behind the frame drawn above, as a compiler emits it without optimization. It copies its two arguments from the registers into its frame and then reads them back from there. The pop rbp at the end restores the previous frame base before ret jumps to the return address:

g:
  push rbp
  mov  rbp, rsp
  mov  [rbp-4], edi   ; a
  mov  [rbp-8], esi   ; b
  mov  eax, [rbp-4]   ; eax = a
  sub  eax, [rbp-8]   ; eax = a - b
  pop  rbp
  ret

Notice that it writes to [rbp-4] and [rbp-8] without first reserving space with sub rsp, N. That's because on Linux a leaf function, one that calls no other function, may use the 128 bytes below the top of the stack without reserving them. That area is called the red zone. A function that calls others reserves its frame with sub rsp, N. Windows has no red zone at all; instead, the caller reserves 32 bytes called the shadow space.