KVM Forum BoF on I/O + secure virtualization

KVM Forum BoF on I/O + secure virtualization

[Giani, Dhaval]

[AMD Official Use Only - General]

Hi all,

We have proposed a trusted I/O BoF session at KVM forum this year. I wanted to kick off the discussion to maximize the 25 mins we have.

By trusted I/O, I mean using TDISP to have “trusted” communications with a device using something like AMD SEV-TIO [1] or Intel’s TDX connect [2].

Some topics we would like to discuss are
o What is the device model like?
o Do we enlighten the PCI subsystem?
o Do we enlighten device drivers?
o What does the guest need to know from the device?
o How does the attestation workflow work?
o Generic vs vendor specific TSMs

Some of these topics may be better suited for LPC, however we want to get the discussion going from the KVM perspective and continue wider discussions at LPC.

If there are other topics you think we should discuss, please let us know.

Thanks!

Dhaval

[1] https://www.amd.com/system/files/documents/sev-tio-whitepaper.pdf
[2] https://cdrdv2-public.intel.com/773614/intel-tdx-connect-architecture-specification.pdf

RE: KVM Forum BoF on I/O + secure virtualization

[Dan Williams]

[ add Lukas ]

Giani, Dhaval wrote:
> [AMD Official Use Only - General]
> 
> Hi all,
> 
> We have proposed a trusted I/O BoF session at KVM forum this year. I wanted
> to kick off the discussion to maximize the 25 mins we have.
> 
> By trusted I/O, I mean using TDISP to have “trusted” communications with a
> device using something like AMD SEV-TIO [1] or Intel’s TDX connect [2].
>
> Some topics we would like to discuss are
> o What is the device model like?
> o Do we enlighten the PCI subsystem?
> o Do we enlighten device drivers?

One observation in relation to these first questions is something that
has been brewing since SPDM and IDE were discussed at Plumbers 2022.

https://lpc.events/event/16/contributions/1304/

Namely, that there is value in the base specs on the way to the full
vendor TSM implementations. I.e. that if the Linux kernel can aspire to
the role of a TSM it becomes easier to incrementally add proxying to a
platform TSM later. In the meantime, platforms and endpoints that
support CMA / SPDM and PCIe/CXL IDE but not full "trusted I/O" still
gain incremental benefit.

The first proof point for that idea is teaching the PCI core to perform
CMA / SPDM session establishment and provide that result to drivers.

That is what Lukas has been working on after picking up Jonathan's
initial SPDM RFC. I expect the discussion on those forthcoming patches
starts to answer device-model questions around attestation.

> o What does the guest need to know from the device?
> o How does the attestation workflow work?
> o Generic vs vendor specific TSMs
> 
> Some of these topics may be better suited for LPC, 

Maybe, but there's so much to discuss that the more opportunities to
collaborate on the details the better. 

> however we want to get the discussion going from the KVM perspective
> and continue wider discussions at LPC.

While I worry that my points above are more suited to something like a
PCI Micro-conference than a KVM BoF, I think the nature of "trusted I/O"
requires those tribes to talk more to each other.

Re: RE: KVM Forum BoF on I/O + secure virtualization

[Alexey Kardashevskiy]

On 6/6/23 12:43, Dan Williams wrote:
> [ add Lukas ]
> 
> Giani, Dhaval wrote:
>> [AMD Official Use Only - General]
>>
>> Hi all,
>>
>> We have proposed a trusted I/O BoF session at KVM forum this year. I wanted
>> to kick off the discussion to maximize the 25 mins we have.
>>
>> By trusted I/O, I mean using TDISP to have “trusted” communications with a
>> device using something like AMD SEV-TIO [1] or Intel’s TDX connect [2].
>>
>> Some topics we would like to discuss are
>> o What is the device model like?
>> o Do we enlighten the PCI subsystem?
>> o Do we enlighten device drivers?
> 
> One observation in relation to these first questions is something that
> has been brewing since SPDM and IDE were discussed at Plumbers 2022.
> 
> https://lpc.events/event/16/contributions/1304/
> 
> Namely, that there is value in the base specs on the way to the full
> vendor TSM implementations. I.e. that if the Linux kernel can aspire to
> the role of a TSM it becomes easier to incrementally add proxying to a
> platform TSM later. In the meantime, platforms and endpoints that
> support CMA / SPDM and PCIe/CXL IDE but not full "trusted I/O" still
> gain incremental benefit.

TSM on the AMD hardware is a PSP firmware and it is going to implement 
all of SPDM/IDE and the only proxying the host kernel will do is PCI DOE.

> The first proof point for that idea is teaching the PCI core to perform
> CMA / SPDM session establishment and provide that result to drivers.
> 
> That is what Lukas has been working on after picking up Jonathan's
> initial SPDM RFC. I expect the discussion on those forthcoming patches
> starts to answer device-model questions around attestation.


Those SPDM patches should work on the AMD hw (as they do not need any 
additional host PCI support) but that's about it - IDE won't be possible 
that way as there is no way to program the IDE keys to PCI RC without 
the PSP.

If we want reuse any of that code to provide 
certificates/measurements/reports for the host kernel, then that will 
need to allow skipping the bits that the firmware implements (SPDM, IDE) 
+ calling the firmware instead. And TDISP is worse as it is based on the 
idea of not trusting the VMM (is there any use for TDISP for the 
host-only config at all?) so such SPDM-enabled linux has to not run KVM.



>> o What does the guest need to know from the device?
>> o How does the attestation workflow work?
>> o Generic vs vendor specific TSMs
>>
>> Some of these topics may be better suited for LPC,
> 
> Maybe, but there's so much to discuss that the more opportunities to
> collaborate on the details the better.
> 
>> however we want to get the discussion going from the KVM perspective
>> and continue wider discussions at LPC.
> 
> While I worry that my points above are more suited to something like a
> PCI Micro-conference than a KVM BoF, I think the nature of "trusted I/O"
> requires those tribes to talk more to each other.

-- 
Alexey

Re: RE: KVM Forum BoF on I/O + secure virtualization

[Lukas Wunner]

[cc += Jonathan, Sean]

On Thu, Jun 08, 2023 at 09:18:12AM +1000, Alexey Kardashevskiy wrote:
> On 6/6/23 12:43, Dan Williams wrote:
> > Giani, Dhaval wrote:
> > > We have proposed a trusted I/O BoF session at KVM forum this year.
> > > I wanted to kick off the discussion to maximize the 25 mins we have.
> > > 
> > > By trusted I/O, I mean using TDISP to have "trusted" communications
> > > with a device using something like AMD SEV-TIO [1] or Intel's
> > > TDX connect [2].
> > > 
> > > Some topics we would like to discuss are
> > > o What is the device model like?
> > > o Do we enlighten the PCI subsystem?
> > > o Do we enlighten device drivers?
> > 
> > One observation in relation to these first questions is something that
> > has been brewing since SPDM and IDE were discussed at Plumbers 2022.
> > 
> > https://lpc.events/event/16/contributions/1304/
> > 
> > Namely, that there is value in the base specs on the way to the full
> > vendor TSM implementations. I.e. that if the Linux kernel can aspire to
> > the role of a TSM it becomes easier to incrementally add proxying to a
> > platform TSM later. In the meantime, platforms and endpoints that
> > support CMA / SPDM and PCIe/CXL IDE but not full "trusted I/O" still
> > gain incremental benefit.
> 
> TSM on the AMD hardware is a PSP firmware and it is going to implement all
> of SPDM/IDE and the only proxying the host kernel will do is PCI DOE.

Sean has voiced a scathing critique of this model where a firmware does
all the attestation and hence needs to be trusted:

https://lore.kernel.org/all/Y+aP8rHr6H3LIf%2Fc@google.com/
https://lore.kernel.org/all/ZEfrjtgGgm1lpadq@google.com/

I think we need to entertain the idea that Linux does the attestation
and encryption setup itself.  Reliance on a "trusted" hardware module
should be optional.

That also works for KVM:  The guest can perform attestation on its own
behalf, setup encryption and drive the TDISP state machine.  If the
guest's memory is encrypted with SEV or TDX, the IDE keys generated by
the guest are invisible to the VMM and hence confidentiality is achieved.
The guest can communicate the keys securely to the device via IDE_KM and
the guest can ask the device to lock down via TDISP.

Yes, the guest may need to rely on the PSP firmware to program the
IDE key into the Root Port.  Can you provide that as a service from
the PSP firmware?

What you seem to be arguing for is a "fat" firmware which does all
the attestation.  The host kernel is relegated to being a mere DOE proxy.
And the guest is relegated to being a "dumb" receiver of the firmware's
attestation results.

What I'm arguing for is a "thin" firmware which provides a minimized
set of services (such as selecting a free IDE stream in the Root Port
and writing keys into it).

The host kernel can perform attestation and set up encryption for
devices it wants to use itself.  Once it passes through a device to
a guest, the host kernel no longer performs any SPDM exchanges with
the device as it's now owned by the guest.  The guest is responsible
for performing attestation and set up encryption for itself, possibly
with the help of firmware.

If there is no firmware to program IDE keys into the Root Port,
the guest must ask the VMM to do that.  The VMM then becomes part
of the guest's TCB, but that trade-off is unavoidable if there's
no firmware assistance.  Some customers (such as Sean) seem to
prefer that to trusting a vendor-provided firmware.

Remember, this must work for everyone, not just for people who are
happy with AMD's and Intel's shrink-wrapped offerings.

We can discuss an _OSC bit to switch between firmware-driven attestation
and OS-native attestation, similar to the existing bits for PCIe hotplug,
DPC etc.

However, past experience with firmware-handled hotplug and DPC has
generally been negative and I believe most everyone is preferring the
OS-native variant nowadays.  The OS has a better overall knowledge of
the system state than the firmware.  E.g. the kernel can detect hotplug
events caused by DPC and ignore them (see commit a97396c6eb13).

Similarly, the CMA-SPDM patches I'm working on reauthenticate devices
after a DPC-induced Hot Reset or after resume from D3cold.  I imagine
it may be difficult to achieve the same if attestation is handled by
firmware.

I'm talking about commit "PCI/CMA: Reauthenticate devices on reset and
resume" on this development branch:

https://github.com/l1k/linux/commits/doe

Thanks,

Lukas

> > The first proof point for that idea is teaching the PCI core to perform
> > CMA / SPDM session establishment and provide that result to drivers.
> > 
> > That is what Lukas has been working on after picking up Jonathan's
> > initial SPDM RFC. I expect the discussion on those forthcoming patches
> > starts to answer device-model questions around attestation.
> 
> Those SPDM patches should work on the AMD hw (as they do not need any
> additional host PCI support) but that's about it - IDE won't be possible
> that way as there is no way to program the IDE keys to PCI RC without the
> PSP.
> 
> If we want reuse any of that code to provide
> certificates/measurements/reports for the host kernel, then that will need
> to allow skipping the bits that the firmware implements (SPDM, IDE) +
> calling the firmware instead. And TDISP is worse as it is based on the idea
> of not trusting the VMM (is there any use for TDISP for the host-only config
> at all?) so such SPDM-enabled linux has to not run KVM.
> 
> > > o What does the guest need to know from the device?
> > > o How does the attestation workflow work?
> > > o Generic vs vendor specific TSMs
> > > 
> > > Some of these topics may be better suited for LPC,
> > 
> > Maybe, but there's so much to discuss that the more opportunities to
> > collaborate on the details the better.
> > 
> > > however we want to get the discussion going from the KVM perspective
> > > and continue wider discussions at LPC.
> > 
> > While I worry that my points above are more suited to something like a
> > PCI Micro-conference than a KVM BoF, I think the nature of "trusted I/O"
> > requires those tribes to talk more to each other.

RE: RE: KVM Forum BoF on I/O + secure virtualization

[Kaplan, David]

[AMD Official Use Only - General]

> -----Original Message-----
> From: Lukas Wunner <lukas@wunner.de>
> Sent: Thursday, June 8, 2023 4:12 AM
> To: Kardashevskiy, Alexey <Alexey.Kardashevskiy@amd.com>
> Cc: Dan Williams <dan.j.williams@intel.com>; Giani, Dhaval
> <Dhaval.Giani@amd.com>; Paolo Bonzini <pbonzini@redhat.com>;
> Kardashevskiy, Alexey <Alexey.Kardashevskiy@amd.com>; Kaplan, David
> <David.Kaplan@amd.com>; steffen.eiden@ibm.com; yilun.xu@intel.com;
> Suzuki K P <suzuki.kp@gmail.com>; Powell, Jeremy
> <Jeremy.Powell@amd.com>; atishp04@gmail.com; linux-
> coco@lists.linux.dev; kvm@vger.kernel.org; Jonathan Cameron
> <Jonathan.Cameron@Huawei.com>; Sean Christopherson
> <seanjc@google.com>
> Subject: Re: RE: KVM Forum BoF on I/O + secure virtualization
>
> Caution: This message originated from an External Source. Use proper
> caution when opening attachments, clicking links, or responding.
>
>
> [cc += Jonathan, Sean]
>
> On Thu, Jun 08, 2023 at 09:18:12AM +1000, Alexey Kardashevskiy wrote:
> > On 6/6/23 12:43, Dan Williams wrote:
> > > Giani, Dhaval wrote:
> > > > We have proposed a trusted I/O BoF session at KVM forum this year.
> > > > I wanted to kick off the discussion to maximize the 25 mins we have.
> > > >
> > > > By trusted I/O, I mean using TDISP to have "trusted"
> > > > communications with a device using something like AMD SEV-TIO [1]
> > > > or Intel's TDX connect [2].
> > > >
> > > > Some topics we would like to discuss are o What is the device
> > > > model like?
> > > > o Do we enlighten the PCI subsystem?
> > > > o Do we enlighten device drivers?
> > >
> > > One observation in relation to these first questions is something
> > > that has been brewing since SPDM and IDE were discussed at Plumbers
> 2022.
> > >
> > > https://lpc.events/event/16/contributions/1304/
> > >
> > > Namely, that there is value in the base specs on the way to the full
> > > vendor TSM implementations. I.e. that if the Linux kernel can aspire
> > > to the role of a TSM it becomes easier to incrementally add proxying
> > > to a platform TSM later. In the meantime, platforms and endpoints
> > > that support CMA / SPDM and PCIe/CXL IDE but not full "trusted I/O"
> > > still gain incremental benefit.
> >
> > TSM on the AMD hardware is a PSP firmware and it is going to implement
> > all of SPDM/IDE and the only proxying the host kernel will do is PCI DOE.
>
> Sean has voiced a scathing critique of this model where a firmware does all
> the attestation and hence needs to be trusted:
>
> https://lore.kernel.org/all/Y+aP8rHr6H3LIf%2Fc@google.com/
> https://lore.kernel.org/all/ZEfrjtgGgm1lpadq@google.com/
>
> I think we need to entertain the idea that Linux does the attestation and
> encryption setup itself.  Reliance on a "trusted" hardware module should be
> optional.

In the SEV-TIO model, the AMD FW implements TDISP/SPDM/IDE but it does *not* do the verification of the attestation information.  That is, it gathers the attestation info from the device but it is the guest software which makes the decision on whether that information is "good" or not.  Until the guest accepts the attestation, the device remains outside of the TCB of the guest.  How exactly software will make that decision is an open question (is it dynamic code in the driver? Is it checked via a static manifest? Something else?) but I want to clarify that AMD FW is not making any trust decisions on behalf of the guest here.

>
> That also works for KVM:  The guest can perform attestation on its own
> behalf, setup encryption and drive the TDISP state machine.  If the guest's
> memory is encrypted with SEV or TDX, the IDE keys generated by the guest
> are invisible to the VMM and hence confidentiality is achieved.
> The guest can communicate the keys securely to the device via IDE_KM and
> the guest can ask the device to lock down via TDISP.
>
> Yes, the guest may need to rely on the PSP firmware to program the IDE key
> into the Root Port.  Can you provide that as a service from the PSP firmware?

There is not one set of IDE keys per guest/device pair, that would be impractical from a HW standpoint.  Instead in the SEV-TIO architecture, there is one set of IDE keys per host/device pair, and traffic from all guests flows over that same stream.  Internal logic in the AMD SoC does access control to ensure that only the correct guest is able to generate trusted packets to specific MMIO addresses on that stream.

So what happens is the AMD FW sets up the IDE stream at the device level, but then individual guests can be bound to individual TDIs.

We have more details at https://www.amd.com/content/dam/amd/en/documents/developer/sev-tio-whitepaper.pdf.  Jeremy is also doing a talk at KVM Forum next week on this.

I'll also note that we understand SEV-TIO is not the only potential use case for these technologies like SPDM/IDE, and there may be bare-metal use cases, or other trust models.

Thanks --David Kaplan

>
> What you seem to be arguing for is a "fat" firmware which does all the
> attestation.  The host kernel is relegated to being a mere DOE proxy.
> And the guest is relegated to being a "dumb" receiver of the firmware's
> attestation results.
>
> What I'm arguing for is a "thin" firmware which provides a minimized set of
> services (such as selecting a free IDE stream in the Root Port and writing keys
> into it).
>
> The host kernel can perform attestation and set up encryption for devices it
> wants to use itself.  Once it passes through a device to a guest, the host
> kernel no longer performs any SPDM exchanges with the device as it's now
> owned by the guest.  The guest is responsible for performing attestation and
> set up encryption for itself, possibly with the help of firmware.
>
> If there is no firmware to program IDE keys into the Root Port, the guest
> must ask the VMM to do that.  The VMM then becomes part of the guest's
> TCB, but that trade-off is unavoidable if there's no firmware assistance.
> Some customers (such as Sean) seem to prefer that to trusting a vendor-
> provided firmware.
>
> Remember, this must work for everyone, not just for people who are happy
> with AMD's and Intel's shrink-wrapped offerings.
>
> We can discuss an _OSC bit to switch between firmware-driven attestation
> and OS-native attestation, similar to the existing bits for PCIe hotplug, DPC
> etc.
>
> However, past experience with firmware-handled hotplug and DPC has
> generally been negative and I believe most everyone is preferring the OS-
> native variant nowadays.  The OS has a better overall knowledge of the
> system state than the firmware.  E.g. the kernel can detect hotplug events
> caused by DPC and ignore them (see commit a97396c6eb13).
>
> Similarly, the CMA-SPDM patches I'm working on reauthenticate devices after
> a DPC-induced Hot Reset or after resume from D3cold.  I imagine it may be
> difficult to achieve the same if attestation is handled by firmware.
>
> I'm talking about commit "PCI/CMA: Reauthenticate devices on reset and
> resume" on this development branch:
>
> https://github.com/l1k/linux/commits/doe
>
> Thanks,
>
> Lukas