nostack option for ARM system calls

[Ioan-Cristian]

Should ARM system calls be marked nostack as here. The Rust reference mentions that nostack must not be used if data is pushed on the stack. However, on ARMv*-M architectures, the hardware automatically pushes data on stack. Furthermore, the Tock kernel does use the process stack for storing process data on exception entry by setting the SPSEL bit to 1. I doubt the reference excludes hardware-related stack operations. As a consequence, the current libtock-rs results in undefined behaviour that just happens to work for the current LLVM backend. Note that libtock-c uses the memory clobber which covers all kinds of memory accesses, including the stack.

Am I missing something?

[ppannuto]

This is a good question. I think Rust's assembly directives are a bit more nuanced than C's — the most important of which being the final bullet of this note (emphasis mine):

The set of memory locations that assembly code is allowed to read and write are the same as those allowed for an FFI function.

• If the readonly option is set, then only memory reads are allowed.
• If the nomem option is set then no reads or writes to memory are allowed.
• These rules do not apply to memory which is private to the assembly code, such as stack space allocated within it.

IIUC, this covers the concern around the exception handler using (and critically also freeing) stack space while it runs.

One thing this does not cover is questions around any necessary memory synchronization barriers. In C-land, part of the reason for the memory clobber is that it forces the compiler to emit (if-necessary) barriers before invoking the syscall. Not really an issue on most single-core M0–M4's, but as we look to bigger and more capable cores, if libtock-rs preserves the nomem on all the syscall, it might need to add explicit synchronization barriers.

More generally, the decision to put the memory clobber on all syscalls in C was done out of a bit of uncertainty / conservative-ness. It is probably not strictly necessary for all syscalls, and may harm performance in a few places, but I am not confident about where we could safely remove it.

I think there are separate concerns around buffers which have been allowed to the kernel, and in which cases the Tock kernel can update those. At minimum, my instinct is that it is probably necessary to remove nomem from the Yield family of syscalls here — but perhaps that synchronization is already handled at another level in libtock-rs.

[ppannuto]

Notes to myself as much as answers, w.r.t. the nostack directive more specifically.

The rust docs for nostack

nostack: The assembly code does not push data to the stack, or write to the stack red-zone (if supported by the target). If this option is not used then the stack pointer is guaranteed to be suitably aligned (according to the target ABI) for a function call.

And the more detailed notes on implications of nostack

Unless the nostack option is set, assembly code is allowed to use stack space below the stack pointer.

• On entry to the assembly code the stack pointer is guaranteed to be suitably aligned (according to the target ABI) for a function call.
• You are responsible for making sure you don’t overflow the stack (e.g. use stack probing to ensure you hit a guard page).
• You should adjust the stack pointer when allocating stack memory as required by the target ABI.
• The stack pointer must be restored to its original value before leaving the assembly code.

Because ARM's svc internally is treated as an exception by the hardware, it's not necessary for software to guarantee stack pointer alignment prior to invoking svc [since any other exception could happen at any time, hardware already has to have support for re-aligning the stack pointer (and then unaligining it on exception return) if needed; that's what CCR.STKALIGN tracks IIUC (i.e., did the hardware re-align the stack on exception entry)].

The rest of the required guarantees are similarly already handled by the hardware exception mechanisms for the same reason.

[Ioan-Cristian]

@ppannuto I had a typo in my original post. I am sorry for it! The first sentence is a question:

Should ARM system calls be marked nostack as here?

It should be interpreted as: why is the nostack option used for ARM system calls since the hardware touches the stack space below the stack pointer?

Unless the nostack option is set, assembly code is allowed to use stack space below the stack pointer.

I understand this sentence as following:

• nostack is used => the software/the hardware is not allowed to use any stack space below the stack pointer.
• nostack is not used => the software/the hardware is allowed to use the stack space below the stack pointer, as long as it respects the requirements you cited.

Since all ARM system calls are marked nostack, they shouldn't touch any stack space below the stack pointer. However, as previously mentioned, the ARMv*-M architectures automatically push and pop data on the stack on exception entry, including system call exceptions. As a consequence, in my opinion, the current ARM system call implementation results in undefined behaviour that just happens to work.

[ppannuto]

the software/the hardware

I don't think it's appropriate to lump these together like this (on ARM at least). The assembly code does not "use stack space below the stack pointer" — i.e., no assembly instruction explicitly touches the stack in any way.

Yes, the svc happens to cause hardware to use stack space, but hardware also (1) validates, fixes, and restores stack pointer alignment if-needed, and (2) more generally restores the stack to an 'unchanged' state when it is done.

This is conceptually no different than if any other hardware interrupt, e.g. a timer interrupt, happened to fire while inside the assembly block. My understanding is that the nostack directive only applies to software, because it would be meaningless / never-usable (without disabling interrupts every time you entered an assembly block) to make statements about what hardware does with the stack.

Similarly, the Tock kernel does assume that the userspace stack is in a sane, aligned state with contents in a known layout relative to the stack pointer — but that is safe/okay because the ARM hardware ensures that is the case on exception entry.

[Ioan-Cristian]

@ppannuto My understanding of nostack is different.

First of all, svc does touch the area under the stack pointer. I see no difference between svc and x86 call instructions. In both cases, data is pushed on the stack. In both cases, the stack is touched as a consequence of the program's instruction flow.

It doesn't matter that svc pushes data, then pops it back. The example provided by the Rust reference pushes, then pops data, similar to the behaviour of svc, but it is still marked as undefined behaviour.

The question that might arrise: why would the compiler use the stack area below the stack pointer? One situation I have managed to reproduce with gcc on a x64 machine long time ago is the following: you have a large leaf function that is called in several places. You compile the code with -fomit-frame-pointer or -fomit-leaf-frame-pointer. When optimizations are enabled, the compiler ignores the frame pointer, but also the instructions for allocating and deallocating the stack: substraction of the SP in the prologue + addition of the SP in the epilogue. Why would the compiler skip stack allocation + deallocation? For several reasons:

1. The stack pointer may not need to be updated to use the space under the stack pointer. Instead of using positive offsets for stack objets, the compile can use negative offsets.
2. Omitting stack substraction + addition saves two instructions: less code + faster execution, especially on small architectures such as ARMv*-M architectures.

I have tried to reproduce this behaviour with libtock-rs by marking syscall4() #[inline(never)] and using RUSTFLAGS="-C force-frame-pointers=no". Unfortunately, the compiler still creates a frame pointer as the force-frame-pointers=no does not guarantee that a frame pointer will not be generated. The result is the same whether the optimization level is set to z or 3.

I'll try to think of an example where svc marked as nostack leads to erroneous code generation.

Curious what @jrvanwhy thinks about this.

[jrvanwhy]

I think there are separate concerns around buffers which have been allowed to the kernel, and in which cases the Tock kernel can update those. At minimum, my instinct is that it is probably necessary to remove nomem from the Yield family of syscalls here — but perhaps that synchronization is already handled at another level in libtock-rs.

nomem is not currently set for Yield (yield1 interface, yield2 interface, ARM yield1 impl, ARM yield2 impl, RISC-V yield1 impl, RISC-V yield2 impl).

Yes, the svc happens to cause hardware to use stack space, but hardware also (1) validates, fixes, and restores stack pointer alignment if-needed, and (2) more generally restores the stack to an 'unchanged' state when it is done.

My mental model was that when the interrupt is complete, the register values and stack pointer value would be restored but the contents of the bytes below the stack pointer (those that the hardware used to save the register values) would not be restored. When the hardware pushes the register values to the stack, it is overwriting something, and I don't see how it can save the value of those bytes.

So it's not entirely 'unchanged', it's just that the compiler must guarantee that anytime an interrupt can happen, the stack is in a state where that modification is acceptable (i.e. those bytes are ignored).

You know how ARM works a lot better than me, though. Is my mental model correct or incorrect?

Also, I believe you are saying that the software interrupt mechanism (triggered by svc) does not change anything in userspace that the hardware interrupt mechanism does not, correct?

My understanding is that the nostack directive only applies to software, because it would be meaningless / never-usable (without disabling interrupts every time you entered an assembly block) to make statements about what hardware does with the stack.

The language reference can't really distinguish hardware from the assembly code. I was also under this understanding (and see some support for it in related discussions. However, I would hope to see some statement that the compiler will still leave the stack in a configuration that allows for interrupts, but I don't see that.

I think we're fine here, but I'm not 100% sure of that. If my mental model and understanding correct, then I can go upstream and ask whether nostack makes enough guarantees to allow for interrupts to execute without causing UB or faulting the process.

[ppannuto]

The example provided by the Rust reference pushes, then pops data, similar to the behaviour of svc, but it is still marked as undefined behaviour.

The example linked here uses Intel's push assembly instruction, which according to the Intel reference manual; PDF page 1256 is a software instruction that explicitly modifies the stack.

Contrast this to ARM's svc instruction, which according to the ARM reference manual, "Operation: The SVC instruction causes an exception.", i.e., the SVC instruction makes no direct assumptions about the state or behavior of the stack.

---

So it's not entirely 'unchanged', it's just that the compiler must guarantee that anytime an interrupt can happen, the stack is in a state where that modification is acceptable (i.e. those bytes are ignored).

Yes, that's exactly the key point here.

On ARM platforms, it is either the case that interrupts must be disabled before before entering any assembly block with the nostack directive set OR it is safe to use svc inside of an assembly block with the nostack directive set (because on ARM, svc behaves just like any other interrupt—e.g. a timer interrupt—that happens to fire during execution of the assembly).

I am not an expert on Intel or RISC-V platforms, and cannot speak to how nostack should (or should not) be used on those platforms.

[ppannuto]

[edit to delete] I do not really understand redzones — ignore the removed comment contents here, sorry.

[jrvanwhy]

I asked about this at https://users.rust-lang.org/t/asm-s-nostack-and-arm-interrupts/132881, hopefully we get a clearer answer there.