On Wed, Feb 16, 2022 at 09:40:34PM +0100, Florian Weimer wrote:
> * Stefan Hajnoczi:
> 
> > On Wed, 16 Feb 2022 at 18:14, Florian Weimer <fweimer@redhat.com> wrote:
> >>
> >> * Stefan Hajnoczi:
> >>
> >> > I've been trying to make the inline asm that gets the address of a TLS
> >> > variable for QEMU coroutines pass QEMU's GitLab CI.
> >> > https://gitlab.com/stefanha/qemu/-/blob/coroutine-tls-fix/include/qemu/coroutine-tls.h#L89
> >> >
> >> > The code isn't -fPIC-friendly (R_X86_64_TPOFF32 relocations aren't
> >> > allowed in -fPIC shared libraries) so builds fail with ./configure
> >> > --enable-modules. While I was tackling this I stumbled on this:
> >> >
> >> >   void *dst_ptr;
> >> >   asm volatile("" : "=r"(dst_ptr) : "0"(&tls_var))
> >> >
> >> > What's nice about it:
> >> > - It's portable, there are no arch-specific assembly instructions.
> >> > - It works for both -fPIC and non-PIC.
> >> >
> >> > However, I wonder if the compiler might reuse a register that already
> >> > contains the address. Then we'd have the coroutine problem again when
> >> > qemu_coroutine_yield() is called between the earlier address calculation
> >> > and the asm volatile statement.
> >> >
> >> > Thoughts?
> >>
> >> Sorry, I don't see why this isn't equivalent to a plain &tls_var.
> >> What exactly are you trying to achieve?
> >
> > &tls_var, except forcing the compiler to calculate the address from scratch.
> 
> I think you can compute
> 
>   (void *) &tls_var - __builtin_thread_pointer ();
> 
> to get the offset.  On many targets, GCC folds away the thread pointer
> load, but that doesn't change the outcome.  Then it boils down to
> getting access to the thread pointer, and you can get that behind a
> compiler barrier (in a separate function).

Interesting, this is something we haven't tried yet. It sounds like it
can be implemented in C without architecture- or ELF-specific inline
assembly.

> But going against ABI and toolchain in this way is really no long-term
> solution.  You need to switch to stackless co-routines, or we need to
> provide proper ABI-level support for this.  Today it's the thread
> pointer, tomorrow it's the shadow stack pointer, and the day after that,
> it's the SafeStack pointer.  And further down the road, it's some thread
> state for garbage collection support.  Or something like that.

Yes, understood :(. This does feel like solving an undefined behavior
problem by adding more undefined behavior on top!

Stackless coroutines have been tried in the past using Continuation
Passing C (https://github.com/kerneis/cpc). Ideally we'd use a solution
built into the compiler though. I'm concerned that CPC might not be
supported or available everywhere QEMU needs to run now and in the
future.

I took a quick look at C++20 coroutines since they are available in
compilers but the primitives look hard to use even from C++, let alone
from C.

If you have any suggestions for stackless coroutine implementations,
please let me know!

Stefan