Xen-Devel Archive on lore.kernel.org
 help / color / Atom feed
* Re: [Xen-devel] Having a DOM-U guest with 1:1 mapping in the second stage MMU.
       [not found] <VI1PR04MB5807A7F83F1B2763BD7EEB20F91B0@VI1PR04MB5807.eurprd04.prod.outlook.com>
@ 2020-02-12 13:45 ` Andrei Cherechesu
  2020-02-12 21:36   ` Stefano Stabellini
  0 siblings, 1 reply; 9+ messages in thread
From: Andrei Cherechesu @ 2020-02-12 13:45 UTC (permalink / raw)
  To: Stefano Stabellini; +Cc: Jorge Pereira, Julien Grall, xen-devel

[-- Attachment #1.1: Type: text/plain, Size: 5134 bytes --]

[[ Sorry. Needed to send another mail because I forgot to add xen-devel lists to CC. ]]

Hello,

I applied your direct-map patch, Stefano, on top of RELEASE-4.13.0
Xen.

I also took your advice and used the Imagebuilder tool to setup my
u-boot environment. I modified the tool to allow SDCard booting and
tweaked the parameters a little to fit our platforms, also introducing
support to add "direct-map" parameter in specific /chosen/DomU node
and "xen,passthrough" in the host dts. The tool is very helpful and
allows me to quickly change the u-boot environment without manually
entering all the fdt formatting commands.

The dom0less booting is successful, however, when I try to passthrough
any device (I tried with ethernet card and uSDHC) I get a kernel panic
in DomU when it tries to probe the driver, because of an unhandled
fault:
(XEN) DOM1: [    3.883482] sdhci: Secure Digital Host Controller Interface driver
(XEN) DOM1: [    3.891021] sdhci: Copyright(c) Pierre Ossman
(XEN) DOM1: [    3.896389] sdhci-pltfm: SDHCI platform and OF driver helper
(XEN) DOM1: [    3.903298] Unhandled fault at 0xffffff800800d048
(XEN) DOM1: [    3.909021] Mem abort info:
(XEN) DOM1: [    3.912863]   ESR = 0x96000000
(XEN) DOM1: [    3.917019]   Exception class = DABT (current EL), IL = 32 bits
(XEN) DOM1: [    3.924115]   SET = 0, FnV = 0
(XEN) DOM1: [    3.928206]   EA = 0, S1PTW = 0
(XEN) DOM1: [    3.932457] Data abort info:
(XEN) DOM1: [    3.936514]   ISV = 0, ISS = 0x00000000
(XEN) DOM1: [    3.941398]   CM = 0, WnR = 0
(XEN) DOM1: [    3.945481] swapper pgtable: 4k pages, 39-bit VAs, pgdp = (____ptrval____)
(XEN) DOM1: [    3.953532] [ffffff800800d048] pgd=00000000bfffe803, pud=00000000bfffe803, pmd=00000000bfffd803, pte=00e80000402f0f07
(XEN) DOM1: [    3.965278] Internal error: ttbr address size fault: 96000000 [#1] PREEMPT SMP
(XEN) DOM1: [    3.973546] Modules linked in:
(XEN) DOM1: [    3.977709] Process swapper/0 (pid: 1, stack limit = 0x(____ptrval____))
(XEN) DOM1: [    3.985525] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.19.59-rt24+g00334f2 #1
(XEN) DOM1: [    3.993855] pstate: 60000005 (nZCv daif -PAN -UAO)
(XEN) DOM1: [    3.999755] pc : 0xffffff80083ac864
(XEN) DOM1: [    4.004354] lr : 0xffffff80083ac810
(XEN) DOM1: [    4.008955] sp : ffffff800800bba0
(XEN) DOM1: [    4.013382] x29: ffffff800800bba0 x28: 0000000000000000
(XEN) DOM1: [    4.019805] x27: ffffff800864f068 x26: ffffff80086ba000
(XEN) DOM1: [    4.026228] x25: ffffffc031564980 x24: ffffff800856e0c0
(XEN) DOM1: [    4.032651] x23: ffffffc03e8eec00 x22: ffffffc03e8eec10
(XEN) DOM1: [    4.039074] x21: ffffffc03e8bf500 x20: ffffffc03e8bf800
(XEN) DOM1: [    4.045497] x19: 0000000000000000 x18: ffffffffffffffff
(XEN) DOM1: [    4.051921] x17: 0000000000000000 x16: 0000000000000000
(XEN) DOM1: [    4.058344] x15: ffffff8008678548 x14: ffffffffffffffff
(XEN) DOM1: [    4.064767] x13: 0000000000000018 x12: 0101010101010101
(XEN) DOM1: [    4.071190] x11: 0000000000000020 x10: 0101010101010101
(XEN) DOM1: [    4.077613] x9 : 0000000000000000 x8 : ffffffc031564c00
(XEN) DOM1: [    4.084036] x7 : 0000000000000000 x6 : 000000000000003f
(XEN) DOM1: [    4.090459] x5 : 0000000000000002 x4 : ffffffc03e83b4c0
(XEN) DOM1: [    4.096883] x3 : 0000000000000000 x2 : 0000000000000000
(XEN) DOM1: [    4.103306] x1 : ffffffc03e8bf000 x0 : ffffff800800d048
(XEN) DOM1: [    4.109729] Call trace:
(XEN) DOM1: [    4.113290]  0xffffff80083ac864
(XEN) DOM1: [    4.117541]  0xffffff800832e3b8
(XEN) DOM1: [    4.121795]  0xffffff800832c49c
(XEN) DOM1: [    4.126047]  0xffffff800832c6bc
(XEN) DOM1: [    4.130301]  0xffffff800832c808
(XEN) DOM1: [    4.134554]  0xffffff800832a208
(XEN) DOM1: [    4.138807]  0xffffff800832bd38
(XEN) DOM1: [    4.143060]  0xffffff800832b5d8
(XEN) DOM1: [    4.147314]  0xffffff800832d1f0
(XEN) DOM1: [    4.151567]  0xffffff800832e318
(XEN) DOM1: [    4.155820]  0xffffff800861d5f8
(XEN) DOM1: [    4.160073]  0xffffff800808397c
(XEN) DOM1: [    4.164326]  0xffffff8008600db4
(XEN) DOM1: [    4.168580]  0xffffff80085078c0
(XEN) DOM1: [    4.172833]  0xffffff8008084c30
(XEN) DOM1: [    4.177091] Code: b9000ea0 d5033e9f f9400ea0 91012000 (b900001f)
(XEN) DOM1: [    4.184298] ---[ end trace 7dc5f6b878cccbfa ]---
(XEN) DOM1: [    4.191546] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b

I uploaded on pastebin.com the u-boot env settings [0], my device
passthrough partial dts [1], and the whole log of boot messages
from xen, Dom0 and DomU [2]. I also modified the guest address
layout and mapped the PL011 UART and GICv3 addresses to match
the physical ones, as well as setting the GUEST_GNTTAB_BASE and
GUEST_MAGIC_BASE to addresses before our board's RAM start address.
I updated the GUEST_RAM0_BASE and GUEST_RAM0_SIZE to match the
physical ones.

Maybe you could check if I did anything wrong, because I couldn't
figure it out.

[0] https://pastebin.com/As6PgVFf
[1] https://pastebin.com/j0NS4x5Z
[2] https://pastebin.com/TaZR8pii

Thank you once again for your support,
Andrei

[-- Attachment #1.2: Type: text/html, Size: 11635 bytes --]

<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
	{font-family:"Cambria Math";
	panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
	{font-family:Calibri;
	panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0in;
	margin-bottom:.0001pt;
	font-size:11.0pt;
	font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
	{mso-style-priority:99;
	color:#0563C1;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{mso-style-priority:99;
	color:#954F72;
	text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
	{mso-style-name:msonormal;
	mso-margin-top-alt:auto;
	margin-right:0in;
	mso-margin-bottom-alt:auto;
	margin-left:0in;
	font-size:11.0pt;
	font-family:"Calibri",sans-serif;}
span.EmailStyle18
	{mso-style-type:personal;
	font-family:"Calibri",sans-serif;
	color:windowtext;}
span.EmailStyle20
	{mso-style-type:personal-reply;
	font-family:"Calibri",sans-serif;
	color:windowtext;}
.MsoChpDefault
	{mso-style-type:export-only;
	font-size:10.0pt;}
@page WordSection1
	{size:8.5in 11.0in;
	margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
	{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">[[ Sorry. Needed to send another mail because I forgot to add xen-devel lists to CC. ]]<o:p></o:p></p>
<p class="MsoNormal"><b><o:p>&nbsp;</o:p></b></p>
<p class="MsoNormal">Hello,<o:p></o:p></p>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal">I applied your direct-map patch, Stefano, on top of RELEASE-4.13.0<o:p></o:p></p>
<p class="MsoNormal">Xen.<o:p></o:p></p>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal">I also took your advice and used the Imagebuilder tool to setup my<o:p></o:p></p>
<p class="MsoNormal">u-boot environment. I modified the tool to allow SDCard booting and<o:p></o:p></p>
<p class="MsoNormal">tweaked the parameters a little to fit our platforms, also introducing<o:p></o:p></p>
<p class="MsoNormal">support to add &#8220;direct-map&#8221; parameter in specific /chosen/DomU node<o:p></o:p></p>
<p class="MsoNormal">and &#8220;xen,passthrough&#8221; in the host dts. The tool is very helpful and<o:p></o:p></p>
<p class="MsoNormal">allows me to quickly change the u-boot environment without manually<o:p></o:p></p>
<p class="MsoNormal">entering all the fdt formatting commands.<o:p></o:p></p>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal">The dom0less booting is successful, however, when I try to passthrough<o:p></o:p></p>
<p class="MsoNormal">any device (I tried with ethernet card and uSDHC) I get a kernel panic<o:p></o:p></p>
<p class="MsoNormal">in DomU when it tries to probe the driver, because of an unhandled<o:p></o:p></p>
<p class="MsoNormal">fault:<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 3.883482] sdhci: Secure Digital Host Controller Interface driver<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 3.891021] sdhci: Copyright(c) Pierre Ossman<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 3.896389] sdhci-pltfm: SDHCI platform and OF driver helper<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 3.903298] Unhandled fault at 0xffffff800800d048<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 3.909021] Mem abort info:<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 3.912863]&nbsp;&nbsp; ESR = 0x96000000<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 3.917019]&nbsp;&nbsp; Exception class = DABT (current EL), IL = 32 bits<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 3.924115]&nbsp;&nbsp; SET = 0, FnV = 0<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 3.928206]&nbsp;&nbsp; EA = 0, S1PTW = 0<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 3.932457] Data abort info:<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 3.936514]&nbsp;&nbsp; ISV = 0, ISS = 0x00000000<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 3.941398]&nbsp;&nbsp; CM = 0, WnR = 0<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 3.945481] swapper pgtable: 4k pages, 39-bit VAs, pgdp = (____ptrval____)<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 3.953532] [ffffff800800d048] pgd=00000000bfffe803, pud=00000000bfffe803, pmd=00000000bfffd803, pte=00e80000402f0f07<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 3.965278] Internal error: ttbr address size fault: 96000000 [#1] PREEMPT SMP<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 3.973546] Modules linked in:<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 3.977709] Process swapper/0 (pid: 1, stack limit = 0x(____ptrval____))<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 3.985525] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.19.59-rt24&#43;g00334f2 #1<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 3.993855] pstate: 60000005 (nZCv daif -PAN -UAO)<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 3.999755] pc : 0xffffff80083ac864<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 4.004354] lr : 0xffffff80083ac810<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 4.008955] sp : ffffff800800bba0<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 4.013382] x29: ffffff800800bba0 x28: 0000000000000000<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 4.019805] x27: ffffff800864f068 x26: ffffff80086ba000<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 4.026228] x25: ffffffc031564980 x24: ffffff800856e0c0<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 4.032651] x23: ffffffc03e8eec00 x22: ffffffc03e8eec10<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 4.039074] x21: ffffffc03e8bf500 x20: ffffffc03e8bf800<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [ &nbsp;&nbsp;&nbsp;4.045497] x19: 0000000000000000 x18: ffffffffffffffff<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 4.051921] x17: 0000000000000000 x16: 0000000000000000<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 4.058344] x15: ffffff8008678548 x14: ffffffffffffffff<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 4.064767] x13: 0000000000000018 x12: 0101010101010101<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 4.071190] x11: 0000000000000020 x10: 0101010101010101<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 4.077613] x9 : 0000000000000000 x8 : ffffffc031564c00<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 4.084036] x7 : 0000000000000000 x6 : 000000000000003f<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 4.090459] x5 : 0000000000000002 x4 : ffffffc03e83b4c0<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 4.096883] x3 : 0000000000000000 x2 : 0000000000000000<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 4.103306] x1 : ffffffc03e8bf000 x0 : ffffff800800d048<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 4.109729] Call trace:<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 4.113290]&nbsp; 0xffffff80083ac864<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 4.117541]&nbsp; 0xffffff800832e3b8<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 4.121795]&nbsp; 0xffffff800832c49c<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 4.126047]&nbsp; 0xffffff800832c6bc<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 4.130301]&nbsp; 0xffffff800832c808<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 4.134554]&nbsp; 0xffffff800832a208<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 4.138807]&nbsp; 0xffffff800832bd38<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 4.143060]&nbsp; 0xffffff800832b5d8<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 4.147314]&nbsp; 0xffffff800832d1f0<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 4.151567]&nbsp; 0xffffff800832e318<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 4.155820]&nbsp; 0xffffff800861d5f8<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 4.160073]&nbsp; 0xffffff800808397c<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 4.164326]&nbsp; 0xffffff8008600db4<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 4.168580]&nbsp; 0xffffff80085078c0<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 4.172833]&nbsp; 0xffffff8008084c30<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 4.177091] Code: b9000ea0 d5033e9f f9400ea0 91012000 (b900001f)<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 4.184298] ---[ end trace 7dc5f6b878cccbfa ]---<o:p></o:p></p>
<p class="MsoNormal">(XEN) DOM1: [&nbsp;&nbsp;&nbsp; 4.191546] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b<o:p></o:p></p>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal">I uploaded on pastebin.com the u-boot env settings [0], my device<o:p></o:p></p>
<p class="MsoNormal">passthrough partial dts [1], and the whole log of boot messages<o:p></o:p></p>
<p class="MsoNormal">from xen, Dom0 and DomU [2]. I also modified the guest address<o:p></o:p></p>
<p class="MsoNormal">layout and mapped the PL011 UART and GICv3 addresses to match<o:p></o:p></p>
<p class="MsoNormal">the physical ones, as well as setting the GUEST_GNTTAB_BASE and<o:p></o:p></p>
<p class="MsoNormal">GUEST_MAGIC_BASE to addresses before our board's RAM start address.<o:p></o:p></p>
<p class="MsoNormal">I updated the GUEST_RAM0_BASE and GUEST_RAM0_SIZE to match the<o:p></o:p></p>
<p class="MsoNormal">physical ones.<o:p></o:p></p>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal">Maybe you could check if I did anything wrong, because I couldn't<o:p></o:p></p>
<p class="MsoNormal">figure it out.<o:p></o:p></p>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal">[0] <a href="https://pastebin.com/As6PgVFf">https://pastebin.com/As6PgVFf</a><o:p></o:p></p>
<p class="MsoNormal">[1] <a href="https://pastebin.com/j0NS4x5Z">https://pastebin.com/j0NS4x5Z</a><o:p></o:p></p>
<p class="MsoNormal">[2] <a href="https://pastebin.com/TaZR8pii">https://pastebin.com/TaZR8pii</a><o:p></o:p></p>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal">Thank you once again for your support,<o:p></o:p></p>
<p class="MsoNormal">Andrei<o:p></o:p></p>
</div>
</body>
</html>

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [Xen-devel] Having a DOM-U guest with 1:1 mapping in the second stage MMU.
  2020-02-12 13:45 ` [Xen-devel] Having a DOM-U guest with 1:1 mapping in the second stage MMU Andrei Cherechesu
@ 2020-02-12 21:36   ` Stefano Stabellini
  2020-02-12 22:03     ` Julien Grall
  0 siblings, 1 reply; 9+ messages in thread
From: Stefano Stabellini @ 2020-02-12 21:36 UTC (permalink / raw)
  To: Andrei Cherechesu
  Cc: Jorge Pereira, Stefano Stabellini, Julien Grall, xen-devel

[-- Attachment #1: Type: text/plain, Size: 8373 bytes --]

On Wed, 12 Feb 2020, Andrei Cherechesu wrote:
> Hello,
>  
> 
> I applied your direct-map patch, Stefano, on top of RELEASE-4.13.0
> Xen.

FYI I am working on a direct-map patch series but it is still
work-in-progress:

  http://xenbits.xenproject.org/git-http/people/sstabellini/xen-unstable.git direct-map-1
  
There are a few fixes on top of the original direct-map patch I sent.

 
> I also took your advice and used the Imagebuilder tool to setup my
> u-boot environment. I modified the tool to allow SDCard booting and
> tweaked the parameters a little to fit our platforms, also introducing
> support to add “direct-map” parameter in specific /chosen/DomU node
> and “xen,passthrough” in the host dts. The tool is very helpful and
> allows me to quickly change the u-boot environment without manually
> entering all the fdt formatting commands.

That's great to hear :-)

For your information, if you have any changes that are worth
upstreaming, I'd be happy to take patches for imagebuilder. The mailing
list for that is viryaos-discuss@lists.sourceforge.net.

 
> The dom0less booting is successful, 

Good! I know it is not easy to setup a dom0less system. I am trying to
build tools and features to make it easier going forward.


> however, when I try to passthrough any device (I tried with ethernet
> card and uSDHC) I get a kernel panic in DomU when it tries to probe
> the driver, because of an unhandled
> 
> fault:
> 
> (XEN) DOM1: [    3.883482] sdhci: Secure Digital Host Controller Interface driver
> (XEN) DOM1: [    3.891021] sdhci: Copyright(c) Pierre Ossman
> (XEN) DOM1: [    3.896389] sdhci-pltfm: SDHCI platform and OF driver helper
> (XEN) DOM1: [    3.903298] Unhandled fault at 0xffffff800800d048
> (XEN) DOM1: [    3.909021] Mem abort info:
> (XEN) DOM1: [    3.912863]   ESR = 0x96000000
> (XEN) DOM1: [    3.917019]   Exception class = DABT (current EL), IL = 32 bits
> (XEN) DOM1: [    3.924115]   SET = 0, FnV = 0
> (XEN) DOM1: [    3.928206]   EA = 0, S1PTW = 0
> (XEN) DOM1: [    3.932457] Data abort info:
> (XEN) DOM1: [    3.936514]   ISV = 0, ISS = 0x00000000
> (XEN) DOM1: [    3.941398]   CM = 0, WnR = 0
> (XEN) DOM1: [    3.945481] swapper pgtable: 4k pages, 39-bit VAs, pgdp = (____ptrval____)
> (XEN) DOM1: [    3.953532] [ffffff800800d048] pgd=00000000bfffe803, pud=00000000bfffe803, pmd=00000000bfffd803, pte=00e80000402f0f07
> (XEN) DOM1: [    3.965278] Internal error: ttbr address size fault: 96000000 [#1] PREEMPT SMP
> (XEN) DOM1: [    3.973546] Modules linked in:
> (XEN) DOM1: [    3.977709] Process swapper/0 (pid: 1, stack limit = 0x(____ptrval____))
> (XEN) DOM1: [    3.985525] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.19.59-rt24+g00334f2 #1
> (XEN) DOM1: [    3.993855] pstate: 60000005 (nZCv daif -PAN -UAO)
> (XEN) DOM1: [    3.999755] pc : 0xffffff80083ac864
> (XEN) DOM1: [    4.004354] lr : 0xffffff80083ac810
> (XEN) DOM1: [    4.008955] sp : ffffff800800bba0
> (XEN) DOM1: [    4.013382] x29: ffffff800800bba0 x28: 0000000000000000
> (XEN) DOM1: [    4.019805] x27: ffffff800864f068 x26: ffffff80086ba000
> (XEN) DOM1: [    4.026228] x25: ffffffc031564980 x24: ffffff800856e0c0
> (XEN) DOM1: [    4.032651] x23: ffffffc03e8eec00 x22: ffffffc03e8eec10
> (XEN) DOM1: [    4.039074] x21: ffffffc03e8bf500 x20: ffffffc03e8bf800
> (XEN) DOM1: [    4.045497] x19: 0000000000000000 x18: ffffffffffffffff
> (XEN) DOM1: [    4.051921] x17: 0000000000000000 x16: 0000000000000000
> (XEN) DOM1: [    4.058344] x15: ffffff8008678548 x14: ffffffffffffffff
> (XEN) DOM1: [    4.064767] x13: 0000000000000018 x12: 0101010101010101
> (XEN) DOM1: [    4.071190] x11: 0000000000000020 x10: 0101010101010101
> (XEN) DOM1: [    4.077613] x9 : 0000000000000000 x8 : ffffffc031564c00
> (XEN) DOM1: [    4.084036] x7 : 0000000000000000 x6 : 000000000000003f
> (XEN) DOM1: [    4.090459] x5 : 0000000000000002 x4 : ffffffc03e83b4c0
> (XEN) DOM1: [    4.096883] x3 : 0000000000000000 x2 : 0000000000000000
> (XEN) DOM1: [    4.103306] x1 : ffffffc03e8bf000 x0 : ffffff800800d048
> (XEN) DOM1: [    4.109729] Call trace:
> (XEN) DOM1: [    4.113290]  0xffffff80083ac864
> (XEN) DOM1: [    4.117541]  0xffffff800832e3b8
> (XEN) DOM1: [    4.121795]  0xffffff800832c49c
> (XEN) DOM1: [    4.126047]  0xffffff800832c6bc
> (XEN) DOM1: [    4.130301]  0xffffff800832c808
> (XEN) DOM1: [    4.134554]  0xffffff800832a208
> (XEN) DOM1: [    4.138807]  0xffffff800832bd38
> (XEN) DOM1: [    4.143060]  0xffffff800832b5d8
> (XEN) DOM1: [    4.147314]  0xffffff800832d1f0
> (XEN) DOM1: [    4.151567]  0xffffff800832e318
> (XEN) DOM1: [    4.155820]  0xffffff800861d5f8
> (XEN) DOM1: [    4.160073]  0xffffff800808397c
> (XEN) DOM1: [    4.164326]  0xffffff8008600db4
> (XEN) DOM1: [    4.168580]  0xffffff80085078c0
> (XEN) DOM1: [    4.172833]  0xffffff8008084c30
> (XEN) DOM1: [    4.177091] Code: b9000ea0 d5033e9f f9400ea0 91012000 (b900001f)
> (XEN) DOM1: [    4.184298] ---[ end trace 7dc5f6b878cccbfa ]---
> (XEN) DOM1: [    4.191546] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
>  
> 
> I uploaded on pastebin.com the u-boot env settings [0], my device
> passthrough partial dts [1], and the whole log of boot messages
> from xen, Dom0 and DomU [2].

I don't know for sure what caused the kernel panic you are seeing,
however, I spotted a couple of issues:

1) A missing feature in the original direct-map patch, specifically it
   couldn't handle interrupts. Please give a try at the updated branch.
2) missing information in the partial device tree.

You have:

        usdhc@402F0000 {
            xen,force-assign-without-iommu;
            #address-cells = <1>;
            #size-cells = <0>;
            compatible = "fsl,s32gen1-usdhc";
            status = "okay";
            reg = <0x0 0x402f0000 0x1000>;
            interrupt-parent = <&gic>;
            interrupts = <0 36 4>;
            clocks = <&misc_clk &misc_clk1 &misc_clk2>;
            clock-names = "ipg", "ahb", "per";
            bus-width = <8>;
            xen,reg = <0x0 0x4002f000 0x1000 0x0 0x4002f000>;
        };

You also need to specify xen,path so that the interrupts are properly
remapped (with my latest direct-map patch series.) Something like:

            xen,path = "/amba/usdhc@402F0000";
            xen,reg = <0x0 0x402f0000 0x10000 0x0 0x402f0000>;
            xen,force-assign-without-iommu;

Assuming that /amba/usdhc@402F0000 is the right path on the host device
tree. Also you shouldn't need the following under usdhc@402F0000:

            #address-cells = <1>;
            #size-cells = <0>;

So overall I'd use:

        usdhc@402F0000 {
            compatible = "fsl,s32gen1-usdhc";
            status = "okay";
            reg = <0x0 0x402f0000 0x1000>;
            interrupt-parent = <&gic>;
            interrupts = <0 36 4>;
            clocks = <&misc_clk &misc_clk1 &misc_clk2>;
            clock-names = "ipg", "ahb", "per";
            bus-width = <8>;
            xen,path = "/amba/usdhc@402F0000";
            xen,reg = <0x0 0x4002f000 0x1000 0x0 0x4002f000>;
            xen,force-assign-without-iommu;
        };


> I also modified the guest address
> layout and mapped the PL011 UART and GICv3 addresses to match
> the physical ones, as well as setting the GUEST_GNTTAB_BASE and
> GUEST_MAGIC_BASE to addresses before our board's RAM start address.
> I updated the GUEST_RAM0_BASE and GUEST_RAM0_SIZE to match the
> physical ones.

Well done! FYI one of the new things in my updated patch series is the
ability to set emulated devices addresses based on the corresponding
physical addresses automatically. Not everything is done yet, but it is
a start.


> Maybe you could check if I did anything wrong, because I couldn't
> figure it out.

Let me know how it goes with the updated partial device tree and
direct-map branch. The changes I suggested should fix the interrupts
setup. However, the kernel panic you are seeing might be caused by
something else -- there might be also another bug.

  
 
> [0] https://pastebin.com/As6PgVFf
> 
> [1] https://pastebin.com/j0NS4x5Z
> 
> [2] https://pastebin.com/TaZR8pii
> 
>  
> 
> Thank you once again for your support,
> 
> Andrei

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [Xen-devel] Having a DOM-U guest with 1:1 mapping in the second stage MMU.
  2020-02-12 21:36   ` Stefano Stabellini
@ 2020-02-12 22:03     ` Julien Grall
  0 siblings, 0 replies; 9+ messages in thread
From: Julien Grall @ 2020-02-12 22:03 UTC (permalink / raw)
  To: Stefano Stabellini, Andrei Cherechesu; +Cc: Jorge Pereira, xen-devel



On 12/02/2020 22:36, Stefano Stabellini wrote:
> On Wed, 12 Feb 2020, Andrei Cherechesu wrote:
>> (XEN) DOM1: [    3.883482] sdhci: Secure Digital Host Controller Interface driver
>> (XEN) DOM1: [    3.891021] sdhci: Copyright(c) Pierre Ossman
>> (XEN) DOM1: [    3.896389] sdhci-pltfm: SDHCI platform and OF driver helper
>> (XEN) DOM1: [    3.903298] Unhandled fault at 0xffffff800800d048
>> (XEN) DOM1: [    3.909021] Mem abort info:
>> (XEN) DOM1: [    3.912863]   ESR = 0x96000000
>> (XEN) DOM1: [    3.917019]   Exception class = DABT (current EL), IL = 32 bits
>> (XEN) DOM1: [    3.924115]   SET = 0, FnV = 0
>> (XEN) DOM1: [    3.928206]   EA = 0, S1PTW = 0
>> (XEN) DOM1: [    3.932457] Data abort info:
>> (XEN) DOM1: [    3.936514]   ISV = 0, ISS = 0x00000000
>> (XEN) DOM1: [    3.941398]   CM = 0, WnR = 0
>> (XEN) DOM1: [    3.945481] swapper pgtable: 4k pages, 39-bit VAs, pgdp = (____ptrval____)
>> (XEN) DOM1: [    3.953532] [ffffff800800d048] pgd=00000000bfffe803, pud=00000000bfffe803, pmd=00000000bfffd803, pte=00e80000402f0f07
>> (XEN) DOM1: [    3.965278] Internal error: ttbr address size fault: 96000000 [#1] PREEMPT SMP
>> (XEN) DOM1: [    3.973546] Modules linked in:
>> (XEN) DOM1: [    3.977709] Process swapper/0 (pid: 1, stack limit = 0x(____ptrval____))
>> (XEN) DOM1: [    3.985525] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.19.59-rt24+g00334f2 #1
>> (XEN) DOM1: [    3.993855] pstate: 60000005 (nZCv daif -PAN -UAO)
>> (XEN) DOM1: [    3.999755] pc : 0xffffff80083ac864
>> (XEN) DOM1: [    4.004354] lr : 0xffffff80083ac810
>> (XEN) DOM1: [    4.008955] sp : ffffff800800bba0
>> (XEN) DOM1: [    4.013382] x29: ffffff800800bba0 x28: 0000000000000000
>> (XEN) DOM1: [    4.019805] x27: ffffff800864f068 x26: ffffff80086ba000
>> (XEN) DOM1: [    4.026228] x25: ffffffc031564980 x24: ffffff800856e0c0
>> (XEN) DOM1: [    4.032651] x23: ffffffc03e8eec00 x22: ffffffc03e8eec10
>> (XEN) DOM1: [    4.039074] x21: ffffffc03e8bf500 x20: ffffffc03e8bf800
>> (XEN) DOM1: [    4.045497] x19: 0000000000000000 x18: ffffffffffffffff
>> (XEN) DOM1: [    4.051921] x17: 0000000000000000 x16: 0000000000000000
>> (XEN) DOM1: [    4.058344] x15: ffffff8008678548 x14: ffffffffffffffff
>> (XEN) DOM1: [    4.064767] x13: 0000000000000018 x12: 0101010101010101
>> (XEN) DOM1: [    4.071190] x11: 0000000000000020 x10: 0101010101010101
>> (XEN) DOM1: [    4.077613] x9 : 0000000000000000 x8 : ffffffc031564c00
>> (XEN) DOM1: [    4.084036] x7 : 0000000000000000 x6 : 000000000000003f
>> (XEN) DOM1: [    4.090459] x5 : 0000000000000002 x4 : ffffffc03e83b4c0
>> (XEN) DOM1: [    4.096883] x3 : 0000000000000000 x2 : 0000000000000000
>> (XEN) DOM1: [    4.103306] x1 : ffffffc03e8bf000 x0 : ffffff800800d048
>> (XEN) DOM1: [    4.109729] Call trace:
>> (XEN) DOM1: [    4.113290]  0xffffff80083ac864
>> (XEN) DOM1: [    4.117541]  0xffffff800832e3b8
>> (XEN) DOM1: [    4.121795]  0xffffff800832c49c
>> (XEN) DOM1: [    4.126047]  0xffffff800832c6bc
>> (XEN) DOM1: [    4.130301]  0xffffff800832c808
>> (XEN) DOM1: [    4.134554]  0xffffff800832a208
>> (XEN) DOM1: [    4.138807]  0xffffff800832bd38
>> (XEN) DOM1: [    4.143060]  0xffffff800832b5d8
>> (XEN) DOM1: [    4.147314]  0xffffff800832d1f0
>> (XEN) DOM1: [    4.151567]  0xffffff800832e318
>> (XEN) DOM1: [    4.155820]  0xffffff800861d5f8
>> (XEN) DOM1: [    4.160073]  0xffffff800808397c
>> (XEN) DOM1: [    4.164326]  0xffffff8008600db4
>> (XEN) DOM1: [    4.168580]  0xffffff80085078c0
>> (XEN) DOM1: [    4.172833]  0xffffff8008084c30
>> (XEN) DOM1: [    4.177091] Code: b9000ea0 d5033e9f f9400ea0 91012000 (b900001f)
>> (XEN) DOM1: [    4.184298] ---[ end trace 7dc5f6b878cccbfa ]---
>> (XEN) DOM1: [    4.191546] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
>>   
>>
>> I uploaded on pastebin.com the u-boot env settings [0], my device
>> passthrough partial dts [1], and the whole log of boot messages
>> from xen, Dom0 and DomU [2].
> 
> I don't know for sure what caused the kernel panic you are seeing,

This is mostly likely because Linux is trying to access a region that is 
not mapped in stage-2. You can rebuild Xen with debug enabled and you 
should see a message "traps.c:..." telling the exact physical address 
accessed.

In general I would recommend to build Xen with debug enabled during 
development as the hypervisor will give you more information of what's 
going on.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [Xen-devel] Having a DOM-U guest with 1:1 mapping in the second stage MMU.
  2020-02-13 17:27 Andrei Cherechesu
  2020-02-13 21:23 ` Julien Grall
@ 2020-02-13 21:28 ` Stefano Stabellini
  1 sibling, 0 replies; 9+ messages in thread
From: Stefano Stabellini @ 2020-02-13 21:28 UTC (permalink / raw)
  To: Andrei Cherechesu; +Cc: xen-devel, Stefano Stabellini, Julien Grall

On Thu, 13 Feb 2020, Andrei Cherechesu wrote:
> Hello,
> 
> I used the Xen from Stefano's tree and made the updates to the partial
> dtb that he specified.
> 
> > This is mostly likely because Linux is trying to access a region
> > that is not mapped in stage-2. You can rebuild Xen with debug enabled
> > and you should see a message "traps.c:..." telling the exact physical
> > address accessed.
> > 
> > In general I would recommend to build Xen with debug enabled during development as the hypervisor will give you more information of what's going on.
> > 
> > Cheers,
> > 
> > --
> > Julien Grall
> 
> I enabled debug config and gave it another try. But I'm still
> getting the same unhandled fault error, that seems to match what
> Julien described above.
> 
> It is indeed a stage-2 abort caused by the guest.
> 
> I attached the DomU1 crash log at [0].
> 
> [0] https://pastebin.com/BSHVFQiK
> 
> How should I proceed in this case?

Looking at the logs, you can see:

(XEN) traps.c:1973:d1v0 HSR=0x939f0046 pc=0xffffff80083ac864 gva=0xffffff800800d048 gpa=0x000000402f0048

So the guest was accessing address 0x402f0048, however, the MMIO address
range of the device that you are remapping (looking at
https://pastebin.com/j0NS4x5Z) is 0x4002f000-0x40030000.

I spotted the mistake now: looking at the partial DTB again, the
address of the device is:

  reg = <0x0 0x402f0000 0x1000>;

but the address that you are remapping is:

  xen,reg = <0x0 0x4002f000 0x1000 0x0 0x4002f000>;

They are not the same! :-)

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [Xen-devel] Having a DOM-U guest with 1:1 mapping in the second stage MMU.
  2020-02-13 17:27 Andrei Cherechesu
@ 2020-02-13 21:23 ` Julien Grall
  2020-02-13 21:28 ` Stefano Stabellini
  1 sibling, 0 replies; 9+ messages in thread
From: Julien Grall @ 2020-02-13 21:23 UTC (permalink / raw)
  To: Andrei Cherechesu; +Cc: xen-devel, Stefano Stabellini

Hi,

On 13/02/2020 18:27, Andrei Cherechesu wrote:
> -----Original Message-----
> From: Julien Grall <julien@xen.org>
> Sent: Thursday, February 13, 2020 00:03
> To: Stefano Stabellini <sstabellini@kernel.org>; Andrei Cherechesu <andrei.cherechesu@nxp.com>
> Cc: Jorge Pereira <jorge.pereira@nxp.com>; xen-devel@lists.xenproject.org
> Subject: Re: [Xen-devel] Having a DOM-U guest with 1:1 mapping in the second stage MMU.
> 
> Hello,
> 
> I used the Xen from Stefano's tree and made the updates to the partial
> dtb that he specified.
> 
>> This is mostly likely because Linux is trying to access a region
>> that is not mapped in stage-2. You can rebuild Xen with debug enabled
>> and you should see a message "traps.c:..." telling the exact physical
>> address accessed.
>>
>> In general I would recommend to build Xen with debug enabled during development as the hypervisor will give you more information of what's going on.
>>
>> Cheers,
>>
>> --
>> Julien Grall
> 
> I enabled debug config and gave it another try. But I'm still
> getting the same unhandled fault error, that seems to match what
> Julien described above.
> 
> It is indeed a stage-2 abort caused by the guest.
> 
> I attached the DomU1 crash log at [0].
> 
> [0] https://pastebin.com/BSHVFQiK

 From the log:

(XEN) traps.c:1973:d1v0 HSR=0x939f0046 pc=0xffffff80083ac864 
gva=0xffffff800800d048 gpa=0x000000402f0048

So the guest is trying to access the guest physical address 
0x000000402f0048. Looking at the partial device tree you provided, you 
are requesting Xen to map the range 0x4002f000 - 0x40030000.

The address does not belong to the range. Could you check whether you 
did passthrough the correct range? (It seems like a zero is missing).

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [Xen-devel] Having a DOM-U guest with 1:1 mapping in the second stage MMU.
@ 2020-02-13 17:27 Andrei Cherechesu
  2020-02-13 21:23 ` Julien Grall
  2020-02-13 21:28 ` Stefano Stabellini
  0 siblings, 2 replies; 9+ messages in thread
From: Andrei Cherechesu @ 2020-02-13 17:27 UTC (permalink / raw)
  To: Julien Grall; +Cc: xen-devel, Stefano Stabellini

-----Original Message-----
From: Julien Grall <julien@xen.org> 
Sent: Thursday, February 13, 2020 00:03
To: Stefano Stabellini <sstabellini@kernel.org>; Andrei Cherechesu <andrei.cherechesu@nxp.com>
Cc: Jorge Pereira <jorge.pereira@nxp.com>; xen-devel@lists.xenproject.org
Subject: Re: [Xen-devel] Having a DOM-U guest with 1:1 mapping in the second stage MMU.

Hello,

I used the Xen from Stefano's tree and made the updates to the partial
dtb that he specified.

> This is mostly likely because Linux is trying to access a region
> that is not mapped in stage-2. You can rebuild Xen with debug enabled
> and you should see a message "traps.c:..." telling the exact physical
> address accessed.
> 
> In general I would recommend to build Xen with debug enabled during development as the hypervisor will give you more information of what's going on.
> 
> Cheers,
> 
> --
> Julien Grall

I enabled debug config and gave it another try. But I'm still
getting the same unhandled fault error, that seems to match what
Julien described above.

It is indeed a stage-2 abort caused by the guest.

I attached the DomU1 crash log at [0].

[0] https://pastebin.com/BSHVFQiK

How should I proceed in this case?

Thank you again for your support,
Andrei
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [Xen-devel] Having a DOM-U guest with 1:1 mapping in the second stage MMU.
  2020-01-15 11:07 ` Julien Grall
@ 2020-01-16 23:55   ` Stefano Stabellini
  0 siblings, 0 replies; 9+ messages in thread
From: Stefano Stabellini @ 2020-01-16 23:55 UTC (permalink / raw)
  To: Julien Grall
  Cc: Jorge Pereira, Stefano Stabellini, Andrei Cherechesu, xen-devel

[-- Attachment #1: Type: text/plain, Size: 3016 bytes --]

On Wed, 15 Jan 2020, Julien Grall wrote:
> On 14/01/2020 21:39, Jorge Pereira wrote:
> > Hi Guys,
> 
> Hello,
> 
> > 
> > I’m currently using XEN in order to run side-by-side a DOM-0 with a DOM-U
> > guest. My use-case scenario requires in the DOM-U direct access to some
> > dma-capable devices such ethernet and some GPUs.
> > 
> > Since our target platform (i.MX8MM) does not support IOMMU, we can’t assign
> > dma-capable devices to the DOM-U guest because XEN does not create 1:1
> > mapping for that guest in the 2^nd stage MMU. So, guest-virtual addresses
> > are different than the physical ones.
> 
> Bear in mind this setup is going to be insecure unless you have another way to
> prevent your passthrough-ed device to access memory it should not (e.g an
> MPU).
> 
> > Is it possible to have 1:1 mapping for DOM-U guests?

I have a patch that enables 1:1 mapping of memory for dom0-less DomUs,
see attached. It introduces a new option called "direct-map" in the domU
specific device tree section. When direct-map is found, the memory gets
allocated 1:1 for the DomU. direct-map can be used in conjuction with
the existing xen,force-assign-without-iommu parameter to enable device
assignment to domUs without a SMMU.

Note that direct-map is supposed to go under /chosen in the main device
tree, while xen,force-assign-without-iommu is supposed to go under
/passthrough in the partial device tree for device assignment.


The patch is only lightly tested and might not work on your platform.
Also, in addition to the security concerns Julien pointed out, it is
very possible to run into trouble with other static addresses.
Currently, we have a number of resources with fixed addresses in the
DomU address space. Give a look at xen/include/public/arch-arm.h: the
domU GICD is fixed at 0x03001000 for example. If the 1:1 memory
allocation selects a memory region that conflicts with any of those
fixed addresses, you are going to have a problem. I am not even sure
that Xen will be able to detect the error and fail explicitly: it could
just silently fail to boot.



> It is not possible at the moment. There are been various effort to try to do
> it, but I have always push back as this is actively defeating the purposing of
> an hypervisor.
> 
> This would be a different story if we had support for MPU in Xen.

A MPU is a requirement to make 1:1 secure, however, it would be
difficult to add support for it in Xen: on Xilinx platforms, the MPU is
not only for VM/VM protection but also for Cortex-Rs/Cortex-As
protection so it is typically programmed beforehand by somebody with a
system wide view (Xen has only a Cortex-A view of the system.) Xen might
not be the right place to configure the MPU at least on Xilinx boards.

This makes me realize that maybe we need to be able to allow the user to
specify not just that she wants direct mapping (like I did in the
attached patch), but also that she wants a specific range of memory for
each DomU, so that she can go and configure the MPU herself.

[-- Attachment #2: Type: text/plain, Size: 4137 bytes --]

diff --git a/docs/misc/arm/passthrough-noiommu.txt b/docs/misc/arm/passthrough-noiommu.txt
new file mode 100644
index 0000000000..d9b7a9b109
--- /dev/null
+++ b/docs/misc/arm/passthrough-noiommu.txt
@@ -0,0 +1,29 @@
+Disable IOMMU
+=============
+
+Add status = "disabled"; under the smmu node:
+
+		smmu@fd800000 {
+			compatible = "arm,mmu-500";
+			status = "disabled";
+
+
+Request Device Assignment without IOMMU support
+===============================================
+
+Add xen,force-assign-without-iommu; to the device tree snippet
+
+		ethernet: ethernet@ff0e0000 {
+			compatible = "cdns,zynqmp-gem";
+			xen,path = "/amba/ethernet@ff0e0000";
+			xen,reg = <0x0 0xff0e0000 0x1000 0x0 0xff0e0000>;
+			xen,force-assign-without-iommu;
+
+
+Request 1:1 memory mapping for the domain
+=========================================
+
+Add direct-map under the appropriate /chosen/domU node.
+If the user is using imagebuilder, you can add to boot.source:
+
+fdt set /chosen/domU0 direct-map <0x1>
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index dd9c3b73ba..51ec7d76db 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -269,9 +269,9 @@ static void __init allocate_memory_11(struct domain *d,
      */
     BUG_ON(!is_domain_direct_mapped(d));
 
-    printk("Allocating 1:1 mappings totalling %ldMB for dom0:\n",
+    printk("Allocating 1:1 mappings totalling %ldMB for dom%u:\n",
            /* Don't want format this as PRIpaddr (16 digit hex) */
-           (unsigned long)(kinfo->unassigned_mem >> 20));
+           (unsigned long)(kinfo->unassigned_mem >> 20), d->domain_id);
 
     kinfo->mem.nr_banks = 0;
 
@@ -2434,7 +2434,11 @@ static int __init construct_domU(struct domain *d,
     /* type must be set before allocate memory */
     d->arch.type = kinfo.type;
 #endif
-    allocate_memory(d, &kinfo);
+
+    if ( !is_domain_direct_mapped(d) )
+        allocate_memory(d, &kinfo);
+    else
+        allocate_memory_11(d, &kinfo);
 
     rc = prepare_dtb_domU(d, &kinfo);
     if ( rc < 0 )
@@ -2454,6 +2458,8 @@ void __init create_domUs(void)
 {
     struct dt_device_node *node;
     const struct dt_device_node *chosen = dt_find_node_by_path("/chosen");
+    int rc;
+    u32 direct_map = 0;
 
     BUG_ON(chosen == NULL);
     dt_for_each_child_node(chosen, node)
@@ -2474,7 +2480,9 @@ void __init create_domUs(void)
             panic("Missing property 'cpus' for domain %s\n",
                   dt_node_name(node));
 
-        if ( dt_find_compatible_node(node, NULL, "multiboot,device-tree") )
+        rc = dt_property_read_u32(node, "direct-map", &direct_map);
+        if ( dt_find_compatible_node(node, NULL, "multiboot,device-tree") &&
+             (!rc || !direct_map) )
             d_cfg.flags |= XEN_DOMCTL_CDF_iommu;
 
         if ( !dt_property_read_u32(node, "nr_spis", &d_cfg.arch.nr_spis) )
@@ -2495,6 +2503,7 @@ void __init create_domUs(void)
             panic("Error creating domain %s\n", dt_node_name(node));
 
         d->is_console = true;
+        d->arch.direct_map = (rc && direct_map != 0);
 
         if ( construct_domU(d, node) != 0 )
             panic("Could not set up domain %s\n", dt_node_name(node));
@@ -2524,6 +2533,7 @@ int __init construct_dom0(struct domain *d)
 
     iommu_hwdom_init(d);
 
+    d->arch.direct_map = true;
     d->max_pages = ~0U;
 
     kinfo.unassigned_mem = dom0_mem;
diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
index f3f3fb7d7f..8b5e1542e5 100644
--- a/xen/include/asm-arm/domain.h
+++ b/xen/include/asm-arm/domain.h
@@ -32,7 +32,7 @@ enum domain_type {
 #endif
 
 /* The hardware domain has always its memory direct mapped. */
-#define is_domain_direct_mapped(d) ((d) == hardware_domain)
+#define is_domain_direct_mapped(d) ((d)->arch.direct_map != false)
 
 struct vtimer {
     struct vcpu *v;
@@ -101,6 +101,8 @@ struct arch_domain
 #ifdef CONFIG_TEE
     void *tee;
 #endif
+
+    bool direct_map;
 }  __cacheline_aligned;
 
 struct arch_vcpu

[-- Attachment #3: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [Xen-devel] Having a DOM-U guest with 1:1 mapping in the second stage MMU.
  2020-01-14 21:39 Jorge Pereira
@ 2020-01-15 11:07 ` Julien Grall
  2020-01-16 23:55   ` Stefano Stabellini
  0 siblings, 1 reply; 9+ messages in thread
From: Julien Grall @ 2020-01-15 11:07 UTC (permalink / raw)
  To: Jorge Pereira, xen-devel, Andrei Cherechesu, Stefano Stabellini



On 14/01/2020 21:39, Jorge Pereira wrote:
> Hi Guys,

Hello,

> 
> I’m currently using XEN in order to run side-by-side a DOM-0 with a 
> DOM-U guest. My use-case scenario requires in the DOM-U direct access to 
> some dma-capable devices such ethernet and some GPUs.
> 
> Since our target platform (i.MX8MM) does not support IOMMU, we can’t 
> assign dma-capable devices to the DOM-U guest because XEN does not 
> create 1:1 mapping for that guest in the 2^nd stage MMU. So, 
> guest-virtual addresses are different than the physical ones.

Bear in mind this setup is going to be insecure unless you have another 
way to prevent your passthrough-ed device to access memory it should not 
(e.g an MPU).

> Is it possible to have 1:1 mapping for DOM-U guests?

It is not possible at the moment. There are been various effort to try 
to do it, but I have always push back as this is actively defeating the 
purposing of an hypervisor.

This would be a different story if we had support for MPU in Xen.

> If not, I’m 
> interested to know what would be the estimated effort to support this 
> feature?

I think you have someone else in NXP looking at 1:1 mapping for Xen (in 
CC). I provided to Andrei some tips how to get 1:1 mapping for DomU 
using dom0less in December (see [1]). So you may want to sync-up with 
him here.

If you are looking at 1:1 DomU using xl, then it is going to require 
more work as the hypercall allocating memory is based on guest frame 
number. There was a thread on the ML a few years ago, I can try to dig 
it down if you are interested.

Cheers,

[1] 
https://lists.xenproject.org/archives/html/xen-devel/2019-12/msg01364.html

> 
> Thanks in advance,
> 
> Cheers,
> 
> Jorge
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xenproject.org
> https://lists.xenproject.org/mailman/listinfo/xen-devel
> 

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

^ permalink raw reply	[flat|nested] 9+ messages in thread

* [Xen-devel] Having a DOM-U guest with 1:1 mapping in the second stage MMU.
@ 2020-01-14 21:39 Jorge Pereira
  2020-01-15 11:07 ` Julien Grall
  0 siblings, 1 reply; 9+ messages in thread
From: Jorge Pereira @ 2020-01-14 21:39 UTC (permalink / raw)
  To: xen-devel

[-- Attachment #1.1: Type: text/plain, Size: 656 bytes --]

Hi Guys,

I'm currently using XEN in order to run side-by-side a DOM-0 with a DOM-U guest. My use-case scenario requires in the DOM-U direct access to some dma-capable devices such ethernet and some GPUs.

Since our target platform (i.MX8MM) does not support IOMMU, we can't assign dma-capable devices to the DOM-U guest because XEN does not create 1:1 mapping for that guest in the 2nd stage MMU. So, guest-virtual addresses are different than the physical ones.

Is it possible to have 1:1 mapping for DOM-U guests? If not, I'm interested to know what would be the estimated effort to support this feature?

Thanks in advance,

Cheers,
Jorge

[-- Attachment #1.2: Type: text/html, Size: 2909 bytes --]

<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
	{font-family:"Cambria Math";
	panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
	{font-family:DengXian;
	panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
	{font-family:Calibri;
	panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
	{font-family:"\@DengXian";
	panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0in;
	margin-bottom:.0001pt;
	font-size:11.0pt;
	font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
	{mso-style-priority:99;
	color:#0563C1;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{mso-style-priority:99;
	color:#954F72;
	text-decoration:underline;}
span.EmailStyle17
	{mso-style-type:personal-compose;
	font-family:"Calibri",sans-serif;
	color:windowtext;}
.MsoChpDefault
	{mso-style-type:export-only;
	font-family:"Calibri",sans-serif;}
@page WordSection1
	{size:8.5in 11.0in;
	margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
	{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">Hi Guys,<o:p></o:p></p>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal">I&#8217;m currently using XEN in order to run side-by-side a DOM-0 with a DOM-U guest. My use-case scenario requires in the DOM-U direct access to some dma-capable devices such ethernet and some GPUs.
<o:p></o:p></p>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal">Since our target platform (i.MX8MM) does not support IOMMU, we can&#8217;t assign dma-capable devices to the DOM-U guest because XEN does not create 1:1 mapping for that guest in the 2<sup>nd</sup> stage MMU. So, guest-virtual addresses are different
 than the physical ones.<o:p></o:p></p>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal">Is it possible to have 1:1 mapping for DOM-U guests? If not, I&#8217;m interested to know what would be the estimated effort to support this feature?<o:p></o:p></p>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal">Thanks in advance,<o:p></o:p></p>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal">Cheers,<o:p></o:p></p>
<p class="MsoNormal">Jorge<o:p></o:p></p>
</div>
</body>
</html>

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, back to index

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
     [not found] <VI1PR04MB5807A7F83F1B2763BD7EEB20F91B0@VI1PR04MB5807.eurprd04.prod.outlook.com>
2020-02-12 13:45 ` [Xen-devel] Having a DOM-U guest with 1:1 mapping in the second stage MMU Andrei Cherechesu
2020-02-12 21:36   ` Stefano Stabellini
2020-02-12 22:03     ` Julien Grall
2020-02-13 17:27 Andrei Cherechesu
2020-02-13 21:23 ` Julien Grall
2020-02-13 21:28 ` Stefano Stabellini
  -- strict thread matches above, loose matches on Subject: below --
2020-01-14 21:39 Jorge Pereira
2020-01-15 11:07 ` Julien Grall
2020-01-16 23:55   ` Stefano Stabellini

Xen-Devel Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/xen-devel/0 xen-devel/git/0.git
	git clone --mirror https://lore.kernel.org/xen-devel/1 xen-devel/git/1.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 xen-devel xen-devel/ https://lore.kernel.org/xen-devel \
		xen-devel@lists.xenproject.org xen-devel@lists.xen.org
	public-inbox-index xen-devel

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.xenproject.lists.xen-devel


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git