Re: DMA_ALIAS_DEVFN issue

Re: DMA_ALIAS_DEVFN issue

[Bjorn Helgaas]

[+cc Alex, add a subject]

On Fri, Nov 14, 2014 at 6:49 PM, Allan, Bruce W <bruce.w.allan@intel.com> wrote:
> Let's try this again as plain text...
>
> For a PCIe device with SR-IOV support enabled (e.g. the PF device ID is 0xf001 at 0000:07:00.0 and the 16 VFs have device ID 0xf002 at 0000:07:01.0 through 0000:07:02.7), if the hardware attempts a DMA read/write of memory that was mapped by the PF devfn but instead uses a requester id of one of the VF devfns (enabled but not yet assigned to a VM) it fails and generates log messages similar to:
>
> [  416.800881] dmar: DRHD: handling fault status reg 2
> [  416.800887] dmar: DMAR:[DMA Read] Request device [07:02.2] fault addr ffff0000
> DMAR:[fault reason 02] Present bit in context entry is clear
>
> Shouldn't a DMA alias quirk that sets the PCI_DEV_FLAGS_DMA_ALIAS_DEVFN flag and dma_alias_devfn to the PF devfn for all VF devfns work-around this issue, or am I misunderstanding what a DMA alias quirk is supposed to do?  If a DMA alias quirk cannot resolve this, what is the most appropriate way to handle this?
>
> Thanks,
> Bruce.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-pci" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: DMA_ALIAS_DEVFN issue

[Bjorn Helgaas]

[+cc Alex, add a subject]

On Fri, Nov 14, 2014 at 6:49 PM, Allan, Bruce W <bruce.w.allan-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> Let's try this again as plain text...
>
> For a PCIe device with SR-IOV support enabled (e.g. the PF device ID is 0xf001 at 0000:07:00.0 and the 16 VFs have device ID 0xf002 at 0000:07:01.0 through 0000:07:02.7), if the hardware attempts a DMA read/write of memory that was mapped by the PF devfn but instead uses a requester id of one of the VF devfns (enabled but not yet assigned to a VM) it fails and generates log messages similar to:
>
> [  416.800881] dmar: DRHD: handling fault status reg 2
> [  416.800887] dmar: DMAR:[DMA Read] Request device [07:02.2] fault addr ffff0000
> DMAR:[fault reason 02] Present bit in context entry is clear
>
> Shouldn't a DMA alias quirk that sets the PCI_DEV_FLAGS_DMA_ALIAS_DEVFN flag and dma_alias_devfn to the PF devfn for all VF devfns work-around this issue, or am I misunderstanding what a DMA alias quirk is supposed to do?  If a DMA alias quirk cannot resolve this, what is the most appropriate way to handle this?
>
> Thanks,
> Bruce.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-pci" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: DMA_ALIAS_DEVFN issue

[Alex Williamson]

On Fri, 2014-11-14 at 21:25 -0700, Bjorn Helgaas wrote:
> [+cc Alex, add a subject]
> 
> On Fri, Nov 14, 2014 at 6:49 PM, Allan, Bruce W <bruce.w.allan@intel.com> wrote:
> > Let's try this again as plain text...
> >
> > For a PCIe device with SR-IOV support enabled (e.g. the PF device ID
> is 0xf001 at 0000:07:00.0 and the 16 VFs have device ID 0xf002 at
> 0000:07:01.0 through 0000:07:02.7), if the hardware attempts a DMA
> read/write of memory that was mapped by the PF devfn but instead uses
> a requester id of one of the VF devfns (enabled but not yet assigned
> to a VM) it fails and generates log messages similar to:
> >
> > [  416.800881] dmar: DRHD: handling fault status reg 2
> > [  416.800887] dmar: DMAR:[DMA Read] Request device [07:02.2] fault addr ffff0000
> > DMAR:[fault reason 02] Present bit in context entry is clear

This is exactly what IOMMU isolation is meant to do.  A buffer mapped
for one device should not be accessible from another device.  It really
doesn't matter if the devices are related as VF/PF, VF/VF, or PF/PF.

> > Shouldn't a DMA alias quirk that sets the
> PCI_DEV_FLAGS_DMA_ALIAS_DEVFN flag and dma_alias_devfn to the PF devfn
> for all VF devfns work-around this issue, or am I misunderstanding
> what a DMA alias quirk is supposed to do?  If a DMA alias quirk cannot
> resolve this, what is the most appropriate way to handle this?

The buffer needs to be mapped for each VF.  If for some reason the VF
needs implicit access to a PF mapped buffer then this devices is
completely broken from an isolation perspective and using DMA aliases to
fix it means that individual VFs will never be assignable to guests.
Thanks,

Alex

RE: DMA_ALIAS_DEVFN issue

[Allan, Bruce W]

PiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiBGcm9tOiBBbGV4IFdpbGxpYW1zb24gW21h
aWx0bzphbGV4LndpbGxpYW1zb25AcmVkaGF0LmNvbV0NCj4gU2VudDogU2F0dXJkYXksIE5vdmVt
YmVyIDE1LCAyMDE0IDc6MjMgQU0NCj4gVG86IEJqb3JuIEhlbGdhYXMNCj4gQ2M6IEFsbGFuLCBC
cnVjZSBXOyBsaW51eC1wY2lAdmdlci5rZXJuZWwub3JnOyBpb21tdUBsaXN0cy5saW51eC0NCj4g
Zm91bmRhdGlvbi5vcmcNCj4gU3ViamVjdDogUmU6IERNQV9BTElBU19ERVZGTiBpc3N1ZQ0KPiAN
Cj4gT24gRnJpLCAyMDE0LTExLTE0IGF0IDIxOjI1IC0wNzAwLCBCam9ybiBIZWxnYWFzIHdyb3Rl
Og0KPiA+IFsrY2MgQWxleCwgYWRkIGEgc3ViamVjdF0NCj4gPg0KPiA+IE9uIEZyaSwgTm92IDE0
LCAyMDE0IGF0IDY6NDkgUE0sIEFsbGFuLCBCcnVjZSBXIDxicnVjZS53LmFsbGFuQGludGVsLmNv
bT4NCj4gd3JvdGU6DQo+ID4gPiBMZXQncyB0cnkgdGhpcyBhZ2FpbiBhcyBwbGFpbiB0ZXh0Li4u
DQo+ID4gPg0KPiA+ID4gRm9yIGEgUENJZSBkZXZpY2Ugd2l0aCBTUi1JT1Ygc3VwcG9ydCBlbmFi
bGVkIChlLmcuIHRoZSBQRiBkZXZpY2UgSUQNCj4gPiBpcyAweGYwMDEgYXQgMDAwMDowNzowMC4w
IGFuZCB0aGUgMTYgVkZzIGhhdmUgZGV2aWNlIElEIDB4ZjAwMiBhdA0KPiA+IDAwMDA6MDc6MDEu
MCB0aHJvdWdoIDAwMDA6MDc6MDIuNyksIGlmIHRoZSBoYXJkd2FyZSBhdHRlbXB0cyBhIERNQQ0K
PiA+IHJlYWQvd3JpdGUgb2YgbWVtb3J5IHRoYXQgd2FzIG1hcHBlZCBieSB0aGUgUEYgZGV2Zm4g
YnV0IGluc3RlYWQgdXNlcw0KPiA+IGEgcmVxdWVzdGVyIGlkIG9mIG9uZSBvZiB0aGUgVkYgZGV2
Zm5zIChlbmFibGVkIGJ1dCBub3QgeWV0IGFzc2lnbmVkDQo+ID4gdG8gYSBWTSkgaXQgZmFpbHMg
YW5kIGdlbmVyYXRlcyBsb2cgbWVzc2FnZXMgc2ltaWxhciB0bzoNCj4gPiA+DQo+ID4gPiBbICA0
MTYuODAwODgxXSBkbWFyOiBEUkhEOiBoYW5kbGluZyBmYXVsdCBzdGF0dXMgcmVnIDINCj4gPiA+
IFsgIDQxNi44MDA4ODddIGRtYXI6IERNQVI6W0RNQSBSZWFkXSBSZXF1ZXN0IGRldmljZSBbMDc6
MDIuMl0gZmF1bHQNCj4gYWRkciBmZmZmMDAwMA0KPiA+ID4gRE1BUjpbZmF1bHQgcmVhc29uIDAy
XSBQcmVzZW50IGJpdCBpbiBjb250ZXh0IGVudHJ5IGlzIGNsZWFyDQo+IA0KPiBUaGlzIGlzIGV4
YWN0bHkgd2hhdCBJT01NVSBpc29sYXRpb24gaXMgbWVhbnQgdG8gZG8uICBBIGJ1ZmZlciBtYXBw
ZWQNCj4gZm9yIG9uZSBkZXZpY2Ugc2hvdWxkIG5vdCBiZSBhY2Nlc3NpYmxlIGZyb20gYW5vdGhl
ciBkZXZpY2UuICBJdCByZWFsbHkNCj4gZG9lc24ndCBtYXR0ZXIgaWYgdGhlIGRldmljZXMgYXJl
IHJlbGF0ZWQgYXMgVkYvUEYsIFZGL1ZGLCBvciBQRi9QRi4NCj4gDQo+ID4gPiBTaG91bGRuJ3Qg
YSBETUEgYWxpYXMgcXVpcmsgdGhhdCBzZXRzIHRoZQ0KPiA+IFBDSV9ERVZfRkxBR1NfRE1BX0FM
SUFTX0RFVkZOIGZsYWcgYW5kIGRtYV9hbGlhc19kZXZmbiB0byB0aGUgUEYNCj4gZGV2Zm4NCj4g
PiBmb3IgYWxsIFZGIGRldmZucyB3b3JrLWFyb3VuZCB0aGlzIGlzc3VlLCBvciBhbSBJIG1pc3Vu
ZGVyc3RhbmRpbmcNCj4gPiB3aGF0IGEgRE1BIGFsaWFzIHF1aXJrIGlzIHN1cHBvc2VkIHRvIGRv
PyAgSWYgYSBETUEgYWxpYXMgcXVpcmsgY2Fubm90DQo+ID4gcmVzb2x2ZSB0aGlzLCB3aGF0IGlz
IHRoZSBtb3N0IGFwcHJvcHJpYXRlIHdheSB0byBoYW5kbGUgdGhpcz8NCj4gDQo+IFRoZSBidWZm
ZXIgbmVlZHMgdG8gYmUgbWFwcGVkIGZvciBlYWNoIFZGLiAgSWYgZm9yIHNvbWUgcmVhc29uIHRo
ZSBWRg0KPiBuZWVkcyBpbXBsaWNpdCBhY2Nlc3MgdG8gYSBQRiBtYXBwZWQgYnVmZmVyIHRoZW4g
dGhpcyBkZXZpY2VzIGlzDQo+IGNvbXBsZXRlbHkgYnJva2VuIGZyb20gYW4gaXNvbGF0aW9uIHBl
cnNwZWN0aXZlIGFuZCB1c2luZyBETUEgYWxpYXNlcyB0bw0KPiBmaXggaXQgbWVhbnMgdGhhdCBp
bmRpdmlkdWFsIFZGcyB3aWxsIG5ldmVyIGJlIGFzc2lnbmFibGUgdG8gZ3Vlc3RzLg0KPiBUaGFu
a3MsDQo+IA0KPiBBbGV4DQoNCkhtbSwgdGhhdC4uLmlzIHVuZm9ydHVuYXRlLiAgVGhhbmtzIEFs
ZXggZm9yIHRoZSBjbGFyaWZpY2F0aW9uLCBhbmQgdGhhbmtzIEJqb3JuIGZvcg0KYWRkaW5nIEFs
ZXggYW5kIGEgcHJvcGVyIHN1YmplY3QgdG8gdGhlIHRocmVhZC4NCg0KQnJ1Y2UuDQoNCg==

RE: DMA_ALIAS_DEVFN issue

[Allan, Bruce W]

> -----Original Message-----
> From: Alex Williamson [mailto:alex.williamson-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org]
> Sent: Saturday, November 15, 2014 7:23 AM
> To: Bjorn Helgaas
> Cc: Allan, Bruce W; linux-pci-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; iommu-cunTk1MwBs/ROKNJybVBZg@public.gmane.org
> foundation.org
> Subject: Re: DMA_ALIAS_DEVFN issue
> 
> On Fri, 2014-11-14 at 21:25 -0700, Bjorn Helgaas wrote:
> > [+cc Alex, add a subject]
> >
> > On Fri, Nov 14, 2014 at 6:49 PM, Allan, Bruce W <bruce.w.allan-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> wrote:
> > > Let's try this again as plain text...
> > >
> > > For a PCIe device with SR-IOV support enabled (e.g. the PF device ID
> > is 0xf001 at 0000:07:00.0 and the 16 VFs have device ID 0xf002 at
> > 0000:07:01.0 through 0000:07:02.7), if the hardware attempts a DMA
> > read/write of memory that was mapped by the PF devfn but instead uses
> > a requester id of one of the VF devfns (enabled but not yet assigned
> > to a VM) it fails and generates log messages similar to:
> > >
> > > [  416.800881] dmar: DRHD: handling fault status reg 2
> > > [  416.800887] dmar: DMAR:[DMA Read] Request device [07:02.2] fault
> addr ffff0000
> > > DMAR:[fault reason 02] Present bit in context entry is clear
> 
> This is exactly what IOMMU isolation is meant to do.  A buffer mapped
> for one device should not be accessible from another device.  It really
> doesn't matter if the devices are related as VF/PF, VF/VF, or PF/PF.
> 
> > > Shouldn't a DMA alias quirk that sets the
> > PCI_DEV_FLAGS_DMA_ALIAS_DEVFN flag and dma_alias_devfn to the PF
> devfn
> > for all VF devfns work-around this issue, or am I misunderstanding
> > what a DMA alias quirk is supposed to do?  If a DMA alias quirk cannot
> > resolve this, what is the most appropriate way to handle this?
> 
> The buffer needs to be mapped for each VF.  If for some reason the VF
> needs implicit access to a PF mapped buffer then this devices is
> completely broken from an isolation perspective and using DMA aliases to
> fix it means that individual VFs will never be assignable to guests.
> Thanks,
> 
> Alex

Hmm, that...is unfortunate.  Thanks Alex for the clarification, and thanks Bjorn for
adding Alex and a proper subject to the thread.

Bruce.