Re: [PATCH 1/4] coroutine: support SafeStack in ucontext backend

[Daniele Buono <dbuono@linux.vnet.ibm.com>]

Hi Stefan,
thank you so much for reviewing this.
See my answers below:

On 5/21/2020 5:44 AM, Stefan Hajnoczi wrote:
> On Wed, Apr 29, 2020 at 03:44:17PM -0400, Daniele Buono wrote:
>> diff --git a/include/qemu/coroutine_int.h b/include/qemu/coroutine_int.h
>> index bd6b0468e1..2ffd75ddbe 100644
>> --- a/include/qemu/coroutine_int.h
>> +++ b/include/qemu/coroutine_int.h
>> @@ -28,6 +28,12 @@
>>   #include "qemu/queue.h"
>>   #include "qemu/coroutine.h"
>>   
>> +#if defined(__has_feature) && __has_feature(safe_stack)
>> +#define CONFIG_SAFESTACK 1
> 
> Please perform this feature check in ./configure. That way
> CONFIG_SAFESTACK will be defined alongside all the other CONFIG_* values
> and be available to C and Makefiles via config-host.h and
> config-host.mak.
> 

Sure, will do this in v2.

>> @@ -160,6 +169,19 @@ Coroutine *qemu_coroutine_new(void)
>>       /* swapcontext() in, siglongjmp() back out */
>>       if (!sigsetjmp(old_env, 0)) {
>>           start_switch_fiber(&fake_stack_save, co->stack, co->stack_size);
>> +#ifdef CONFIG_SAFESTACK
>> +        /*
>> +         * Before we swap the context, set the new unsafe stack
>> +         * The unsafe stack grows just like the normal stack, so start from
>> +         * the last usable location of the memory area.
>> +         * NOTE: we don't have to re-set it afterwards because sigsetjmp was
>> +         * called with the original usp. Since we are not coming back with a
>> +         * swapcontext, but with a siglongjmp, when we are back here we
>> +         * already have usp restored to the valid one for this function
> 
> I don't understand this comment. __safestack_unsafe_stack_ptr is a
> thread-local variable, not a CPU register. How will siglongjmp()
> automatically restore it?
> 
Correct, setjmp/longjmp have no visibility of the unsafe stack. What I
meant is that it is not automatically restored by the longjmp itself,
but by code the compiler adds around the sigsetjmp.

Specifically, every sigsetjmp/sigjmp is intercepted by the compiler, the
current value of __safestack_unsafe_stack_ptr is saved on the normal
(safe) stack.
Right after the sigsetjmp call it is then restored.

I will change the comment to make it clearer.

In practice, this is what happens:

Original clang implementation in qemu_coroutine_new:
----
40130c:  callq  4008d0 <__sigsetjmp@plt>
401311:  cmp    $0x0,%eax
401314:  jne    40132d <qemu_coroutine_new+0x12d>
----
Clang Implementation with safestack:
----
4027a7:  mov    %rdx,-0x38(%rbp) <- Save unsafe ptr onto the safe stack
[...]
40289c:  callq  401410 <__sigsetjmp@plt>
4028a1:  mov    0x201738(%rip),%rdi        # 603fe0 
<__safestack_unsafe_stack_ptr@@Base+0x603fe0>
4028a8:  mov    -0x38(%rbp),%rbx
4028ac:  mov    %rbx,%fs:(%rdi) <- Restore the unsafe ptr
4028b0:  cmp    $0x0,%eax
4028b3:  jne    4028d9 <qemu_coroutine_new+0x179>


>> +         */
>> +        void *usp = co->unsafe_stack + co->unsafe_stack_size;
>> +        __safestack_unsafe_stack_ptr = usp;
>> +#endif
>>           swapcontext(&old_uc, &uc);
>>       }
>>