sve_user_discard

sve_user_discard

[psodagud]

Hi All,

This is regarding sve_user_disable(CPACR_EL1_ZEN_EL0EN) on every system 
call.  If a userspace task is using SVE instructions and making sys 
calls in between, it would impact the performance of the thread. On 
every SVE instructions after SVC/system call, it would trap to EL1.

I think by setting CPACR_EL1_ZEN_EL0EN flag,  the processor faults when 
it runs an SVE instruction. This approach may be taken as part of FPSIMD 
registers switching optimizations.  Can below portion of the code use 
thread.fpsimd_cpu and fpsimd_last_state variables to avoid clearing 
CPACR_EL1_ZEN_EL0EN for this kind of use cases?

static inline void sve_user_discard(void)
{
	if (!system_supports_sve())
		return;

	clear_thread_flag(TIF_SVE);

	/*
	 * task_fpsimd_load() won't be called to update CPACR_EL1 in
	 * ret_to_user unless TIF_FOREIGN_FPSTATE is still set, which only
	 * happens if a context switch or kernel_neon_begin() or context
	 * modification (sigreturn, ptrace) intervenes.
	 * So, ensure that CPACR_EL1 is already correct for the fast-path case.
	 */
	sve_user_disable();
}

-Thanks, Prasad

Re: sve_user_discard

[Catalin Marinas]

On Thu, May 20, 2021 at 04:02:03PM -0700, psodagud@codeaurora.org wrote:
> This is regarding sve_user_disable(CPACR_EL1_ZEN_EL0EN) on every system
> call.  If a userspace task is using SVE instructions and making sys calls in
> between, it would impact the performance of the thread. On every SVE
> instructions after SVC/system call, it would trap to EL1.
> 
> I think by setting CPACR_EL1_ZEN_EL0EN flag,  the processor faults when it
> runs an SVE instruction. This approach may be taken as part of FPSIMD
> registers switching optimizations.  Can below portion of the code use
> thread.fpsimd_cpu and fpsimd_last_state variables to avoid clearing
> CPACR_EL1_ZEN_EL0EN for this kind of use cases?

There were attempts over the past couple of years to optimise the
syscall return use-case. I think the latest is this one:

https://lore.kernel.org/r/20201106193553.22946-2-broonie@kernel.org

I'll let Mark comment on his plans for reviving the series. Do you
happen to have some realistic workload that would be improved by this?
We can always write a micro-benchmark but I wonder how much this matters
in the real world.

-- 
Catalin

Re: sve_user_discard

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 2495 bytes --]

On Fri, May 21, 2021 at 10:12:54AM +0100, Catalin Marinas wrote:
> On Thu, May 20, 2021 at 04:02:03PM -0700, psodagud@codeaurora.org wrote:

> > This is regarding sve_user_disable(CPACR_EL1_ZEN_EL0EN) on every system
> > call.  If a userspace task is using SVE instructions and making sys calls in
> > between, it would impact the performance of the thread. On every SVE
> > instructions after SVC/system call, it would trap to EL1.

> > I think by setting CPACR_EL1_ZEN_EL0EN flag,  the processor faults when it
> > runs an SVE instruction. This approach may be taken as part of FPSIMD
> > registers switching optimizations.  Can below portion of the code use
> > thread.fpsimd_cpu and fpsimd_last_state variables to avoid clearing
> > CPACR_EL1_ZEN_EL0EN for this kind of use cases?

This mail hasn't hit the lists yet so I've only got the quoted portions,
missing some context like the "code below" referenced above so I don't
know exactly what the proposal is.

> There were attempts over the past couple of years to optimise the
> syscall return use-case. I think the latest is this one:

> https://lore.kernel.org/r/20201106193553.22946-2-broonie@kernel.org

There's actually this more recently:

    https://lore.kernel.org/linux-arm-kernel/20210512151131.27877-1-broonie@kernel.org/

which does 90% of the optimisation in a lot less code, people seem a bit
more enthusiastic about that version.

> I'll let Mark comment on his plans for reviving the series. Do you
> happen to have some realistic workload that would be improved by this?
> We can always write a micro-benchmark but I wonder how much this matters
> in the real world.

Yeah, I'm not sure how much of a meaningful overhead there is from doing
the sve_user_discard() vs testing to see if we need to do it while
maintaining correctness.  Whatever overhead there is with the current
code will only take effect if we're hitting a slow path anyway so it
feels like it might be more trouble than it's worth.  If the proposal
was to just leave SVE enabled for userspace then the issue there is that
we'd have to context switch SVE even if the process isn't using it,
there is nothing other than a syscall that lets us stop doing that.

It will be interesting to look at this stuff as SVE hardware starts to
become more widely available and used on a wider range of workloads on
systems with various vector sizes, lazy restore might be worth looking
at for example possibly in conjunction with always allowing SVE from
userspace.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]