Re: `AMD_MEM_ENCRYPT_ACTIVE_BY_DEFAULT=y` causes AMDGPU to fail on Ryzen: amdgpu: SME is not compatible with RAVEN

From: Alex Deucher <alexdeucher@gmail.com>
To: Borislav Petkov <bp@alien8.de>, Tom Lendacky <Thomas.Lendacky@amd.com>
Cc: Paul Menzel <pmenzel@molgen.mpg.de>,
	Thomas Gleixner <tglx@linutronix.de>,
	Ingo Molnar <mingo@redhat.com>, X86 ML <x86@kernel.org>,
	Dave Hansen <dave.hansen@linux.intel.com>,
	Andy Lutomirski <luto@kernel.org>,
	Peter Zijlstra <peterz@infradead.org>,
	LKML <linux-kernel@vger.kernel.org>,
	amd-gfx list <amd-gfx@lists.freedesktop.org>
Subject: Re: `AMD_MEM_ENCRYPT_ACTIVE_BY_DEFAULT=y` causes AMDGPU to fail on Ryzen: amdgpu: SME is not compatible with RAVEN
Date: Wed, 6 Oct 2021 14:36:56 -0400	[thread overview]
Message-ID: <CADnq5_O8rTnWH5md1qWauW_3sDTO+qf=vNzPZC1E70NT7Wb99A@mail.gmail.com> (raw)
In-Reply-To: <YV3pQCZ4GYbdpEm5@zn.tnic>

On Wed, Oct 6, 2021 at 2:21 PM Borislav Petkov <bp@alien8.de> wrote:
>
> On Wed, Oct 06, 2021 at 02:10:30PM -0400, Alex Deucher wrote:
> > This is not limited to Raven.
>
> That's what the innocuous "a.o." wanted to state. :)

Whoops, my eyes passed right over that.

>
> > All GPUs (and quite a few other
> > devices) have a limited DMA mask.  AMD GPUs have between 32 and 48
> > bits of DMA depending on what generation the hardware is.  So to
> > support SME, you either need swiotlb with bounce buffers or you need
> > IOMMU in remapping mode. The limitation with Raven is that if you want
> > to use it with the IOMMU enabled it requires the IOMMU to be set up in
> > passthrough mode to support IOMMUv2 functionality for compute support
> > and due to other hardware limitations on the display side. So for all
> > GPUs except raven, just having IOMMU enabled in remapping mode is
> > fine.  GPUs from other vendors would likely run into similar
> > limitations.  Raven just has further limitations.
>
> Hmm, and in passthrough mode it would use bounce buffers when SME is
> enabled. And when those 256K are not enough, it would fail there too,
> even with IOMMUv2. At least this is how it looks from here.
>
> Dunno, it feels like doing GPU compute and SME does not go hand-in-hand
> real smoothly currently but that probably doesn't matter all too much
> for both user camps. But that's just me with a hunch.

Well, this limitation only applies to Raven which is an integrated GPU
in client parts.  SME was initially productized on server parts so
there was not a lot of concern given to interactions with integrated
graphics at the time.  This has since been fixed in newer integrated
graphics.  dGPUs work fine as long as the IOMMU is in remapping mode
to handle the C bit.

>
> > Another option would be to enable SME by default on Epyc platforms,
> > but disabled by default on client APU platforms or even just raven.
>
> Thing is, we don't know at SME init time - very early during boot -
> whether we're Epyc or client. Can we find that out reliably from the hw?
>

From the x86 model and family info?  I think Raven has different
families from other Zen based CPUs.

> And even if we do, that's still not accurate enough - we wanna know
> whether the IOMMU works.

Right.

>
> So I guess we're all left to the user to decide. But I'm always open
> to suggestions for solving things in sw and not requiring any user
> interaction.

@Tom Lendacky Any ideas?

Alex

>
> > Other than these comments, looks fine to me.
>
> Thx.
>
> --
> Regards/Gruss,
>     Boris.
>
> https://people.kernel.org/tglx/notes-about-netiquette