Re: [PATCH RFC RFT 2/5] fork: Add shadow stack support to clone3()

From: Deepak Gupta <debug@rivosinc.com>
To: "Szabolcs.Nagy@arm.com" <Szabolcs.Nagy@arm.com>
Cc: Mark Brown <broonie@kernel.org>,
	"Edgecombe, Rick P" <rick.p.edgecombe@intel.com>,
	"dietmar.eggemann@arm.com" <dietmar.eggemann@arm.com>,
	"keescook@chromium.org" <keescook@chromium.org>,
	"brauner@kernel.org" <brauner@kernel.org>,
	"shuah@kernel.org" <shuah@kernel.org>,
	"mgorman@suse.de" <mgorman@suse.de>,
	"dave.hansen@linux.intel.com" <dave.hansen@linux.intel.com>,
	"fweimer@redhat.com" <fweimer@redhat.com>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"vincent.guittot@linaro.org" <vincent.guittot@linaro.org>,
	"hjl.tools@gmail.com" <hjl.tools@gmail.com>,
	"rostedt@goodmis.org" <rostedt@goodmis.org>,
	"mingo@redhat.com" <mingo@redhat.com>,
	"tglx@linutronix.de" <tglx@linutronix.de>,
	"vschneid@redhat.com" <vschneid@redhat.com>,
	"catalin.marinas@arm.com" <catalin.marinas@arm.com>,
	"bristot@redhat.com" <bristot@redhat.com>,
	"will@kernel.org" <will@kernel.org>,
	"hpa@zytor.com" <hpa@zytor.com>,
	"peterz@infradead.org" <peterz@infradead.org>,
	"jannh@google.com" <jannh@google.com>,
	"bp@alien8.de" <bp@alien8.de>,
	"bsegall@google.com" <bsegall@google.com>,
	"linux-kselftest@vger.kernel.org"
	<linux-kselftest@vger.kernel.org>,
	"linux-api@vger.kernel.org" <linux-api@vger.kernel.org>,
	"x86@kernel.org" <x86@kernel.org>,
	"juri.lelli@redhat.com" <juri.lelli@redhat.com>
Subject: Re: [PATCH RFC RFT 2/5] fork: Add shadow stack support to clone3()
Date: Fri, 27 Oct 2023 16:24:18 -0700	[thread overview]
Message-ID: <ZTxGovqKdhA5hYMz@debug.ba.rivosinc.com> (raw)
In-Reply-To: <ZTuj565SqIb9KjQr@arm.com>

On Fri, Oct 27, 2023 at 12:49:59PM +0100, Szabolcs.Nagy@arm.com wrote:
>The 10/26/2023 13:40, Deepak Gupta wrote:
>> On Thu, Oct 26, 2023 at 06:53:37PM +0100, Mark Brown wrote:
>> > I'm not sure placement control is essential but the other bit of it is
>> > the freeing of the shadow stack, especially if userspace is doing stack
>> > switches the current behaviour where we free the stack when the thread
>> > is exiting doesn't feel great exactly.  It's mainly an issue for
>> > programs that pivot stacks which isn't the common case but it is a
>> > general sharp edge.
>>
>> In general, I am assuming such placement requirements emanate because
>> regular stack holds data (local args, etc) as well and thus software may
>> make assumptions about how stack frame is prepared and may worry about
>> layout and such. In case of shadow stack, it can only hold return
>
>no. the lifetime is the issue: a stack in principle can outlive
>a thread and resumed even after the original thread exited.
>for that to work the shadow stack has to outlive the thread too.
>

I understand an application can pre-allocate a pool of stack and re-use
them whenever it's spawning new threads using clone3 system call.

However, once a new thread has been spawned how can it resume?
By resume I mean consume the callstack context from an earlier thread.
Or you meant something else by `resume` here?

Can you give an example of such an application or runtime where a newly
created thread consumes callstack context created by going away thread?

>(or the other way around: a stack can be freed before the thread
>exits, if the thread pivots away from that stack.)

This is simply a thread saying that I am moving to a different stack.
Again, interested in learning why would a thread do that. If I've to
speculate on reasons, I could think of user runtime managing it's own
pool of worker items (some people call them green threads) or current
stack became too small.

JIT runtimes (and such stuff like go routines) do such things but in
those cases, kernel has no idea about it. From kernel's perspective
there is a main thread stack (hosting thread for JIT) and then main
thread can take a decision switching stack to execute JITted code.
But in that case all it needs is a shadow stack and managing lifetime of
such shadow stack using `clone` wouldn't be helpful and perhaps
`map_shadow_stack` should be used to create on the fly shadow stack.

Another case I can think of for a thread to move to a different stack
when current stack was too small and it wants larger memory. In such
cases as well, I imagine that particular thread would be issuing `mmap`
to allocate larger memory and thus that particular thread can very well
issue `map_shadow_stack`

In both of these cases, a stack free actually means thread (application)
issuing a system call to free the going away stack memory. It can free up
going away shadow stack memory in same way using `unmap_shadow_stack`

Let me know if I misunderstood something or missing some other usecase of
a stack being freed before the thread exits.

>
>posix threads etc. don't allow this, but the linux syscall abi
>(clone) does allow it.
>
>i think it is reasonable to tie the shadow stack lifetime to the
>thread lifetime, but this clearly introduces a limitation on how
>the clone api can be used. such constraint on the userspace
>programming model is normally a bad decision, but given that most
>software (including all posix conforming code) is not affected,
>i think it is acceptable for an opt-in feature like shadow stack.
>
>IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.