From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.linutronix.de (193.142.43.55:993) by crypto-ml.lab.linutronix.de with IMAP4-SSL for ; 14 Oct 2019 12:51:09 -0000 Received: from esa4.hc3370-68.iphmx.com ([216.71.155.144]) by Galois.linutronix.de with esmtps (TLS1.2:DHE_RSA_AES_256_CBC_SHA256:256) (Exim 4.80) (envelope-from ) id 1iJzoZ-0008Ev-LA for speck@linutronix.de; Mon, 14 Oct 2019 14:51:08 +0200 From: Andrew Cooper Subject: [MODERATED] EPT/IOMMU shattering on SNB Message-ID: Date: Mon, 14 Oct 2019 13:50:56 +0100 MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="qeTmLogxH4ZtfloUQ7wnnC6Lul3YblEhX"; protected-headers="v1" To: speck@linutronix.de List-ID: --qeTmLogxH4ZtfloUQ7wnnC6Lul3YblEhX Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Content-Language: en-GB Hello, Testing has revealed an issue with the use of NX superpages in combination with in-flight DMA. It is possible that this is a bug in Xen, but we can only reproduce it on SandyBridge E5 parts. The scenario is running a graphics workload (Haven benchmark) in a VM (Win7) with GPU Passthrough (Nvidia Quatro card in this case).=C2=A0 The = EPT tables are shared with the IOMMU. What we observe is that userspace starting up results in a load of shatters (as the VM is freshly booted at the time), and then we start taking IOMMU faults (read/write access denied) against IO-virtual addresses which have just been shattered. I am now certain that R and W permissions and translations, are valid at all times, so am at a loss to explain the IOMMU faults.=C2=A0 The fact th= at we can't reproduce this with the same scenario on newer parts does suggest that it might not be software related. I know the IO-TLB on SNB is a little weird.=C2=A0 The IOMMU claims suppor= t for superpages but doesn't implement them, and DMA through a 2M superpage has about 40% lower throughput than through equivalent 4k mappings.=C2=A0 We never adequately got to the bottom of this so Xen, lik= e Linux, still uses IO-superpages. Avoiding sharing the EPT and IOMMU tables does avoid the problem, but that is to be expected as the shattering activity now has no interaction with the IOMMU. Has anyone experimented with the above scenario, and if so, how has testing gone? Thanks, ~Andrew --qeTmLogxH4ZtfloUQ7wnnC6Lul3YblEhX--