* Re: Question about xen and Rasp 4B
[not found] <CAFnJQOouOUox_kBB66sfSuriUiUSvmhSjPxucJjgrB9DdLOwkg@mail.gmail.com>
@ 2021-01-23 0:27 ` Stefano Stabellini
2021-01-23 1:00 ` Roman Shaposhnik
` (2 more replies)
0 siblings, 3 replies; 33+ messages in thread
From: Stefano Stabellini @ 2021-01-23 0:27 UTC (permalink / raw)
To: Jukka Kaartinen; +Cc: sstabellini, xen-devel, roman, julien, Bertrand.Marquis
[-- Attachment #1: Type: text/plain, Size: 2643 bytes --]
+ xen-devel, Roman,
On Fri, 22 Jan 2021, Jukka Kaartinen wrote:
> Hi Stefano,
> I'm Jukka Kaartinen a SW developer working on enabling hypervisors on mobile platforms. One of our HW that we use on development is
> Raspberry Pi 4B. I wonder if you could help me a bit :).
>
> I'm trying to enable the GPU with Xen + Raspberry Pi for dom0. https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323#p1797605
>
> I got so far that GPU drivers are loaded (v3d & vc4) without errors. But now Xen returns error when X is starting:
> (XEN) traps.c:1986:d0v1 HSR=0x93880045 pc=0x00007f97b14e70 gva=0x7f7f817000 gpa=0x0000401315d000
> I tried to debug what causes this and looks like find_mmio_handler cannot find handler.
> (See more here: https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323&start=25#p1801691 )
>
> Any ideas why the handler is not found?
Hi Jukka,
I am glad to hear that you are interested in Xen on RaspberryPi :-) I
haven't tried the GPU yet, I have been using the serial only.
Roman, did you ever get the GPU working?
The error is a data abort error: Linux is trying to access an address
which is not mapped to dom0. The address seems to be 0x401315d000. It is
a pretty high address; I looked in device tree but couldn't spot it.
From the HSR (the syndrom register) it looks like it is a translation
fault at EL1 on stage1. As if the Linux address mapping was wrong.
Anyone has any ideas how this could happen? Maybe a reserved-memory
misconfiguration?
> p.s.
> While testing I found issue with Xen master branch and your patch: xen/rpi4: implement watchdog-based reset
>
> Looks like black listing the bcm2835-pm
> @@ -37,12 +41,69 @@ static const struct dt_device_match rpi4_blacklist_dev[] __initconst =
> * The aux peripheral also shares a page with the aux UART.
> */
> DT_MATCH_COMPATIBLE("brcm,bcm2835-aux"),
> + /* Special device used for rebooting */
> + DT_MATCH_COMPATIBLE("brcm,bcm2835-pm"),
>
> will prevent v3d driver to locate phandle. I think it will use the same resource:
> pm: watchdog@7e100000 {
> compatible = "brcm,bcm2835-pm", "brcm,bcm2835-pm-wdt";
> #power-domain-cells = <1>;
> #reset-cells = <1>;
> reg = <0x7e100000 0x114>,
> <0x7e00a000 0x24>,
> <0x7ec11000 0x20>;
> clocks = <&clocks BCM2835_CLOCK_V3D>,
> <&clocks BCM2835_CLOCK_PERI_IMAGE>,
> <&clocks BCM2835_CLOCK_H264>,
> <&clocks BCM2835_CLOCK_ISP>;
> clock-names = "v3d", "peri_image", "h264", "isp";
> system-power-controller;
>
> };
Yeah, I imagine it could be possible. Can you post the error message you
are seeing from the v3d driver?
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Question about xen and Rasp 4B
2021-01-23 0:27 ` Question about xen and Rasp 4B Stefano Stabellini
@ 2021-01-23 1:00 ` Roman Shaposhnik
2021-01-23 14:33 ` Jukka Kaartinen
2021-01-23 14:29 ` Jukka Kaartinen
2021-01-23 16:48 ` Julien Grall
2 siblings, 1 reply; 33+ messages in thread
From: Roman Shaposhnik @ 2021-01-23 1:00 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Jukka Kaartinen, Xen-devel, Julien Grall, Bertrand Marquis
On Fri, Jan 22, 2021 at 4:27 PM Stefano Stabellini
<sstabellini@kernel.org> wrote:
>
> + xen-devel, Roman,
>
>
> On Fri, 22 Jan 2021, Jukka Kaartinen wrote:
> > Hi Stefano,
> > I'm Jukka Kaartinen a SW developer working on enabling hypervisors on mobile platforms. One of our HW that we use on development is
> > Raspberry Pi 4B. I wonder if you could help me a bit :).
> >
> > I'm trying to enable the GPU with Xen + Raspberry Pi for dom0. https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323#p1797605
> >
> > I got so far that GPU drivers are loaded (v3d & vc4) without errors. But now Xen returns error when X is starting:
> > (XEN) traps.c:1986:d0v1 HSR=0x93880045 pc=0x00007f97b14e70 gva=0x7f7f817000 gpa=0x0000401315d000
> > I tried to debug what causes this and looks like find_mmio_handler cannot find handler.
> > (See more here: https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323&start=25#p1801691 )
> >
> > Any ideas why the handler is not found?
>
>
> Hi Jukka,
>
> I am glad to hear that you are interested in Xen on RaspberryPi :-) I
> haven't tried the GPU yet, I have been using the serial only.
> Roman, did you ever get the GPU working?
I've never tested it with X -- sorry. But I have to ask what kernel is being
used? It made a lot of difference for me to actually STOP using RPi
supplied kernels and switch directly to 5.10.7.
Thanks,
Roman.
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Question about xen and Rasp 4B
2021-01-23 0:27 ` Question about xen and Rasp 4B Stefano Stabellini
2021-01-23 1:00 ` Roman Shaposhnik
@ 2021-01-23 14:29 ` Jukka Kaartinen
2021-01-26 0:54 ` Stefano Stabellini
2021-01-23 16:48 ` Julien Grall
2 siblings, 1 reply; 33+ messages in thread
From: Jukka Kaartinen @ 2021-01-23 14:29 UTC (permalink / raw)
To: Stefano Stabellini; +Cc: xen-devel, roman, julien, Bertrand.Marquis
[-- Attachment #1: Type: text/plain, Size: 3588 bytes --]
Thanks for the response!
On Sat, Jan 23, 2021 at 2:27 AM Stefano Stabellini <sstabellini@kernel.org>
wrote:
> + xen-devel, Roman,
>
>
> On Fri, 22 Jan 2021, Jukka Kaartinen wrote:
> > Hi Stefano,
> > I'm Jukka Kaartinen a SW developer working on enabling hypervisors on
> mobile platforms. One of our HW that we use on development is
> > Raspberry Pi 4B. I wonder if you could help me a bit :).
> >
> > I'm trying to enable the GPU with Xen + Raspberry Pi for dom0.
> https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323#p1797605
> >
> > I got so far that GPU drivers are loaded (v3d & vc4) without errors. But
> now Xen returns error when X is starting:
> > (XEN) traps.c:1986:d0v1 HSR=0x93880045 pc=0x00007f97b14e70
> gva=0x7f7f817000 gpa=0x0000401315d000
> > I tried to debug what causes this and looks
> like find_mmio_handler cannot find handler.
> > (See more here:
> https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323&start=25#p1801691
> )
> >
> > Any ideas why the handler is not found?
>
>
> Hi Jukka,
>
> I am glad to hear that you are interested in Xen on RaspberryPi :-) I
> haven't tried the GPU yet, I have been using the serial only.
> Roman, did you ever get the GPU working?
>
>
> The error is a data abort error: Linux is trying to access an address
> which is not mapped to dom0. The address seems to be 0x401315d000. It is
> a pretty high address; I looked in device tree but couldn't spot it.
>
> From the HSR (the syndrom register) it looks like it is a translation
> fault at EL1 on stage1. As if the Linux address mapping was wrong.
> Anyone has any ideas how this could happen? Maybe a reserved-memory
> misconfiguration?
>
I had issues with loading the driver in the first place. Apparently swiotlb
is used, maybe it can cause this. I also tried to enable CMA.
config.txt:
dtoverlay=vc4-fkms-v3d,cma=320M@0x0-0x40000000
gpu_mem=128
That memory is very high. Success full address is for example
this: 0xff841f00
(XEN) GICv2 initialization:
(XEN) gic_dist_addr=00000000ff841000
(XEN) gic_cpu_addr=00000000ff842000
(XEN) gic_hyp_addr=00000000ff844000
(XEN) gic_vcpu_addr=00000000ff846000
(XEN) gic_maintenance_irq=25
> > p.s.
> > While testing I found issue with Xen master branch and your
> patch: xen/rpi4: implement watchdog-based reset
> >
> > Looks like black listing the bcm2835-pm
> > @@ -37,12 +41,69 @@ static const struct dt_device_match
> rpi4_blacklist_dev[] __initconst =
> > * The aux peripheral also shares a page with the aux UART.
> > */
> > DT_MATCH_COMPATIBLE("brcm,bcm2835-aux"),
> > + /* Special device used for rebooting */
> > + DT_MATCH_COMPATIBLE("brcm,bcm2835-pm"),
> >
> > will prevent v3d driver to locate phandle. I think it will use the same
> resource:
> > pm: watchdog@7e100000 {
> > compatible = "brcm,bcm2835-pm", "brcm,bcm2835-pm-wdt";
> > #power-domain-cells = <1>;
> > #reset-cells = <1>;
> > reg = <0x7e100000 0x114>,
> > <0x7e00a000 0x24>,
> > <0x7ec11000 0x20>;
> > clocks = <&clocks BCM2835_CLOCK_V3D>,
> > <&clocks BCM2835_CLOCK_PERI_IMAGE>,
> > <&clocks BCM2835_CLOCK_H264>,
> > <&clocks BCM2835_CLOCK_ISP>;
> > clock-names = "v3d", "peri_image", "h264", "isp";
> > system-power-controller;
> >
> > };
>
> Yeah, I imagine it could be possible. Can you post the error message you
> are seeing from the v3d driver?
This is the error:
[ 0.069682] OF: /v3dbus/v3d@7ec04000: could not find phandle
[ 0.074828] OF: /v3dbus/v3d@7ec04000: could not find phandle
v3d driver is not loaded.
--
Br,
Jukka Kaartinen
[-- Attachment #2: Type: text/html, Size: 5107 bytes --]
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Question about xen and Rasp 4B
2021-01-23 1:00 ` Roman Shaposhnik
@ 2021-01-23 14:33 ` Jukka Kaartinen
0 siblings, 0 replies; 33+ messages in thread
From: Jukka Kaartinen @ 2021-01-23 14:33 UTC (permalink / raw)
To: Roman Shaposhnik
Cc: Stefano Stabellini, Xen-devel, Julien Grall, Bertrand Marquis
[-- Attachment #1: Type: text/plain, Size: 1691 bytes --]
On Sat, Jan 23, 2021 at 3:00 AM Roman Shaposhnik <roman@zededa.com> wrote:
> On Fri, Jan 22, 2021 at 4:27 PM Stefano Stabellini
> <sstabellini@kernel.org> wrote:
> >
> > + xen-devel, Roman,
> >
> >
> > On Fri, 22 Jan 2021, Jukka Kaartinen wrote:
> > > Hi Stefano,
> > > I'm Jukka Kaartinen a SW developer working on enabling hypervisors on
> mobile platforms. One of our HW that we use on development is
> > > Raspberry Pi 4B. I wonder if you could help me a bit :).
> > >
> > > I'm trying to enable the GPU with Xen + Raspberry Pi for dom0.
> https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323#p1797605
> > >
> > > I got so far that GPU drivers are loaded (v3d & vc4) without errors.
> But now Xen returns error when X is starting:
> > > (XEN) traps.c:1986:d0v1 HSR=0x93880045 pc=0x00007f97b14e70
> gva=0x7f7f817000 gpa=0x0000401315d000
> > > I tried to debug what causes this and looks like find_mmio_handler
> cannot find handler.
> > > (See more here:
> https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323&start=25#p1801691
> )
> > >
> > > Any ideas why the handler is not found?
> >
> >
> > Hi Jukka,
> >
> > I am glad to hear that you are interested in Xen on RaspberryPi :-) I
> > haven't tried the GPU yet, I have been using the serial only.
> > Roman, did you ever get the GPU working?
>
> I've never tested it with X -- sorry. But I have to ask what kernel is
> being
> used? It made a lot of difference for me to actually STOP using RPi
> supplied kernels and switch directly to 5.10.7.
>
I'm using rpi-5.9.y -branch from https://github.com/raspberrypi/linux.
I tried to use rpi-5.10.y but with that raspberry is not booting up.
--
Br,
Jukka Kaartinen
[-- Attachment #2: Type: text/html, Size: 2712 bytes --]
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Question about xen and Rasp 4B
2021-01-23 0:27 ` Question about xen and Rasp 4B Stefano Stabellini
2021-01-23 1:00 ` Roman Shaposhnik
2021-01-23 14:29 ` Jukka Kaartinen
@ 2021-01-23 16:48 ` Julien Grall
2021-01-24 16:23 ` Jukka Kaartinen
2 siblings, 1 reply; 33+ messages in thread
From: Julien Grall @ 2021-01-23 16:48 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Jukka Kaartinen, xen-devel, Roman Shaposhnik, Bertrand Marquis
On Sat, 23 Jan 2021 at 00:27, Stefano Stabellini <sstabellini@kernel.org> wrote:
>
> + xen-devel, Roman,
>
>
> On Fri, 22 Jan 2021, Jukka Kaartinen wrote:
> > Hi Stefano,
> > I'm Jukka Kaartinen a SW developer working on enabling hypervisors on mobile platforms. One of our HW that we use on development is
> > Raspberry Pi 4B. I wonder if you could help me a bit :).
> >
> > I'm trying to enable the GPU with Xen + Raspberry Pi for dom0. https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323#p1797605
> >
> > I got so far that GPU drivers are loaded (v3d & vc4) without errors. But now Xen returns error when X is starting:
> > (XEN) traps.c:1986:d0v1 HSR=0x93880045 pc=0x00007f97b14e70 gva=0x7f7f817000 gpa=0x0000401315d000
> > I tried to debug what causes this and looks like find_mmio_handler cannot find handler.
> > (See more here: https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323&start=25#p1801691 )
> >
> > Any ideas why the handler is not found?
>
>
> Hi Jukka,
>
> I am glad to hear that you are interested in Xen on RaspberryPi :-) I
> haven't tried the GPU yet, I have been using the serial only.
> Roman, did you ever get the GPU working?
>
>
> The error is a data abort error: Linux is trying to access an address
> which is not mapped to dom0. The address seems to be 0x401315d000. It is
> a pretty high address; I looked in device tree but couldn't spot it.
>
> From the HSR (the syndrom register) it looks like it is a translation
> fault at EL1 on stage1.
Hmmm.... We don't trap stage 1 faults to Xen, instead they are
received by the OS directly.
In fact the ESR_El1, indicates that it is a translation fault at level
1 when walking the stage-2 page-table. So it means the guest physical
address is not mapped in the P2M. But...
> As if the Linux address mapping was wrong.
... I think the GPA is bogus. So I agree Linux may have configured the
stage-1 page-tables incorrectly.
>
> Anyone has any ideas how this could happen? Maybe a reserved-memory
> misconfiguration?
It looks like to me both the PC and the GVA points to user memory.
Normally, Xen will inject a data abort into Linux and will usually
dump some information (e.g. the task running).
@Jukka, would it be possible to provide the console out for both Xen and Linux?
Would you also be able to confirm if the same setup is working when
Linux is running directly on the HW? This would give us an idea
whether the issue is specific to Xen.
Cheers,
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Question about xen and Rasp 4B
2021-01-23 16:48 ` Julien Grall
@ 2021-01-24 16:23 ` Jukka Kaartinen
0 siblings, 0 replies; 33+ messages in thread
From: Jukka Kaartinen @ 2021-01-24 16:23 UTC (permalink / raw)
To: Julien Grall
Cc: Stefano Stabellini, xen-devel, Roman Shaposhnik, Bertrand Marquis
[-- Attachment #1: Type: text/plain, Size: 3821 bytes --]
On Sat, Jan 23, 2021 at 6:48 PM Julien Grall <julien.grall.oss@gmail.com>
wrote:
> On Sat, 23 Jan 2021 at 00:27, Stefano Stabellini <sstabellini@kernel.org>
> wrote:
> >
> > + xen-devel, Roman,
> >
> >
> > On Fri, 22 Jan 2021, Jukka Kaartinen wrote:
> > > Hi Stefano,
> > > I'm Jukka Kaartinen a SW developer working on enabling hypervisors on
> mobile platforms. One of our HW that we use on development is
> > > Raspberry Pi 4B. I wonder if you could help me a bit :).
> > >
> > > I'm trying to enable the GPU with Xen + Raspberry Pi for dom0.
> https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323#p1797605
> > >
> > > I got so far that GPU drivers are loaded (v3d & vc4) without errors.
> But now Xen returns error when X is starting:
> > > (XEN) traps.c:1986:d0v1 HSR=0x93880045 pc=0x00007f97b14e70
> gva=0x7f7f817000 gpa=0x0000401315d000
> > > I tried to debug what causes this and looks like find_mmio_handler
> cannot find handler.
> > > (See more here:
> https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323&start=25#p1801691
> )
> > >
> > > Any ideas why the handler is not found?
> >
> >
> > Hi Jukka,
> >
> > I am glad to hear that you are interested in Xen on RaspberryPi :-) I
> > haven't tried the GPU yet, I have been using the serial only.
> > Roman, did you ever get the GPU working?
> >
> >
> > The error is a data abort error: Linux is trying to access an address
> > which is not mapped to dom0. The address seems to be 0x401315d000. It is
> > a pretty high address; I looked in device tree but couldn't spot it.
> >
> > From the HSR (the syndrom register) it looks like it is a translation
> > fault at EL1 on stage1.
>
> Hmmm.... We don't trap stage 1 faults to Xen, instead they are
> received by the OS directly.
>
> In fact the ESR_El1, indicates that it is a translation fault at level
> 1 when walking the stage-2 page-table. So it means the guest physical
> address is not mapped in the P2M. But...
>
>
> > As if the Linux address mapping was wrong.
>
> ... I think the GPA is bogus. So I agree Linux may have configured the
> stage-1 page-tables incorrectly.
>
> >
> > Anyone has any ideas how this could happen? Maybe a reserved-memory
> > misconfiguration?
>
> It looks like to me both the PC and the GVA points to user memory.
> Normally, Xen will inject a data abort into Linux and will usually
> dump some information (e.g. the task running).
>
I observed that when xen outputs this:
(XEN) traps.c:1983:d0v1 HSR=0x93830045 pc=0x00007fa9df9684 gva=0x7faa26b000
gpa=0x0000401207f000
vc4_bo_create is called with this size:
[ 44.449287] vc4_bo_create size: 4096
> @Jukka, would it be possible to provide the console out for both Xen and
> Linux?
>
Here are the logs:
https://drive.google.com/drive/folders/1fhcwpsf6SfPYrk0QR6zexGia0v1XysUr?usp=sharing
I hope you can access them.
Would you also be able to confirm if the same setup is working when
> Linux is running directly on the HW? This would give us an idea
> whether the issue is specific to Xen.
>
Yes, the same setup works when running directly on the HW. I even boot the
linux using u-boot and the same device tree.
There is one change that bothers me and that is the swiotlb size. I had to
increase its size:
20 * Maximum allowable number of contiguous slabs to map,
21 * must be a power of 2. What is the appropriate value ?
22 * The complexity of {map,unmap}_single is linearly dependent on this
value.
23 */
24 //#define IO_TLB_SEGSIZE▸▸▸▸▸▸▸▸128
25 #define IO_TLB_SEGSIZE▸▸4096
without this change I get this error:
[ 79.444987] v3d fec00000.v3d: swiotlb buffer is full (sz: 8294400
bytes), total 32768 (slots), used 1 (slots)
--
Br,
Jukka Kaartinen
[-- Attachment #2: Type: text/html, Size: 5442 bytes --]
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Question about xen and Rasp 4B
2021-01-23 14:29 ` Jukka Kaartinen
@ 2021-01-26 0:54 ` Stefano Stabellini
2021-01-26 10:59 ` Jukka Kaartinen
0 siblings, 1 reply; 33+ messages in thread
From: Stefano Stabellini @ 2021-01-26 0:54 UTC (permalink / raw)
To: Jukka Kaartinen
Cc: Stefano Stabellini, xen-devel, roman, julien, Bertrand.Marquis
[-- Attachment #1: Type: text/plain, Size: 4569 bytes --]
On Sat, 23 Jan 2021, Jukka Kaartinen wrote:
> Thanks for the response!
>
> On Sat, Jan 23, 2021 at 2:27 AM Stefano Stabellini <sstabellini@kernel.org> wrote:
> + xen-devel, Roman,
>
>
> On Fri, 22 Jan 2021, Jukka Kaartinen wrote:
> > Hi Stefano,
> > I'm Jukka Kaartinen a SW developer working on enabling hypervisors on mobile platforms. One of our HW that we use on
> development is
> > Raspberry Pi 4B. I wonder if you could help me a bit :).
> >
> > I'm trying to enable the GPU with Xen + Raspberry Pi for
> dom0. https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323#p1797605
> >
> > I got so far that GPU drivers are loaded (v3d & vc4) without errors. But now Xen returns error when X is starting:
> > (XEN) traps.c:1986:d0v1 HSR=0x93880045 pc=0x00007f97b14e70 gva=0x7f7f817000 gpa=0x0000401315d000
> > I tried to debug what causes this and looks like find_mmio_handler cannot find handler.
> > (See more here: https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323&start=25#p1801691 )
> >
> > Any ideas why the handler is not found?
>
>
> Hi Jukka,
>
> I am glad to hear that you are interested in Xen on RaspberryPi :-) I
> haven't tried the GPU yet, I have been using the serial only.
> Roman, did you ever get the GPU working?
>
>
> The error is a data abort error: Linux is trying to access an address
> which is not mapped to dom0. The address seems to be 0x401315d000. It is
> a pretty high address; I looked in device tree but couldn't spot it.
>
> >From the HSR (the syndrom register) it looks like it is a translation
> fault at EL1 on stage1. As if the Linux address mapping was wrong.
> Anyone has any ideas how this could happen? Maybe a reserved-memory
> misconfiguration?
>
> I had issues with loading the driver in the first place. Apparently swiotlb is used, maybe it can cause this. I also tried to enable CMA.
> config.txt:
> dtoverlay=vc4-fkms-v3d,cma=320M@0x0-0x40000000
> gpu_mem=128
Also looking at your other reply and the implementation of
vc4_bo_create, it looks like this is a CMA problem.
It would be good to run a test with the swiotlb-xen disabled:
diff --git a/arch/arm/xen/mm.c b/arch/arm/xen/mm.c
index 467fa225c3d0..2bdd12785d14 100644
--- a/arch/arm/xen/mm.c
+++ b/arch/arm/xen/mm.c
@@ -138,8 +138,7 @@ void xen_destroy_contiguous_region(phys_addr_t pstart, unsigned int order)
static int __init xen_mm_init(void)
{
struct gnttab_cache_flush cflush;
- if (!xen_initial_domain())
- return 0;
+ return 0;
xen_swiotlb_init(1, false);
cflush.op = 0;
It is going to be fine just to boot Dom0 and DomUs without PV drivers.
Also, can you post the device tree that you are using here? Just in case
there is an issue with Xen parsing any possible /reserved-memory nodes
with CMA info that need to be passed on to Dom0.
> > p.s.
> > While testing I found issue with Xen master branch and your patch: xen/rpi4: implement watchdog-based reset
> >
> > Looks like black listing the bcm2835-pm
> > @@ -37,12 +41,69 @@ static const struct dt_device_match rpi4_blacklist_dev[] __initconst =
> > * The aux peripheral also shares a page with the aux UART.
> > */
> > DT_MATCH_COMPATIBLE("brcm,bcm2835-aux"),
> > + /* Special device used for rebooting */
> > + DT_MATCH_COMPATIBLE("brcm,bcm2835-pm"),
> >
> > will prevent v3d driver to locate phandle. I think it will use the same resource:
> > pm: watchdog@7e100000 {
> > compatible = "brcm,bcm2835-pm", "brcm,bcm2835-pm-wdt";
> > #power-domain-cells = <1>;
> > #reset-cells = <1>;
> > reg = <0x7e100000 0x114>,
> > <0x7e00a000 0x24>,
> > <0x7ec11000 0x20>;
> > clocks = <&clocks BCM2835_CLOCK_V3D>,
> > <&clocks BCM2835_CLOCK_PERI_IMAGE>,
> > <&clocks BCM2835_CLOCK_H264>,
> > <&clocks BCM2835_CLOCK_ISP>;
> > clock-names = "v3d", "peri_image", "h264", "isp";
> > system-power-controller;
> >
> > };
>
> Yeah, I imagine it could be possible. Can you post the error message you
> are seeing from the v3d driver?
>
> This is the error:
> [ 0.069682] OF: /v3dbus/v3d@7ec04000: could not find phandle
> [ 0.074828] OF: /v3dbus/v3d@7ec04000: could not find phandle
> v3d driver is not loaded.
>
> --
> Br,
> Jukka Kaartinen
>
>
^ permalink raw reply related [flat|nested] 33+ messages in thread
* Re: Question about xen and Rasp 4B
2021-01-26 0:54 ` Stefano Stabellini
@ 2021-01-26 10:59 ` Jukka Kaartinen
2021-01-26 13:58 ` Jukka Kaartinen
2021-01-26 20:22 ` Stefano Stabellini
0 siblings, 2 replies; 33+ messages in thread
From: Jukka Kaartinen @ 2021-01-26 10:59 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Xen-devel, Roman Shaposhnik, Julien Grall, Bertrand Marquis
[-- Attachment #1: Type: text/plain, Size: 6266 bytes --]
On Tue, Jan 26, 2021 at 2:54 AM Stefano Stabellini <sstabellini@kernel.org>
wrote:
> On Sat, 23 Jan 2021, Jukka Kaartinen wrote:
> > Thanks for the response!
> >
> > On Sat, Jan 23, 2021 at 2:27 AM Stefano Stabellini <
> sstabellini@kernel.org> wrote:
> > + xen-devel, Roman,
> >
> >
> > On Fri, 22 Jan 2021, Jukka Kaartinen wrote:
> > > Hi Stefano,
> > > I'm Jukka Kaartinen a SW developer working on enabling
> hypervisors on mobile platforms. One of our HW that we use on
> > development is
> > > Raspberry Pi 4B. I wonder if you could help me a bit :).
> > >
> > > I'm trying to enable the GPU with Xen + Raspberry Pi for
> > dom0.
> https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323#p1797605
> > >
> > > I got so far that GPU drivers are loaded (v3d & vc4) without
> errors. But now Xen returns error when X is starting:
> > > (XEN) traps.c:1986:d0v1 HSR=0x93880045 pc=0x00007f97b14e70
> gva=0x7f7f817000 gpa=0x0000401315d000
> > > I tried to debug what causes this and looks
> like find_mmio_handler cannot find handler.
> > > (See more here:
> https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323&start=25#p1801691
> )
> > >
> > > Any ideas why the handler is not found?
> >
> >
> > Hi Jukka,
> >
> > I am glad to hear that you are interested in Xen on RaspberryPi
> :-) I
> > haven't tried the GPU yet, I have been using the serial only.
> > Roman, did you ever get the GPU working?
> >
> >
> > The error is a data abort error: Linux is trying to access an
> address
> > which is not mapped to dom0. The address seems to be 0x401315d000.
> It is
> > a pretty high address; I looked in device tree but couldn't spot
> it.
> >
> > >From the HSR (the syndrom register) it looks like it is a
> translation
> > fault at EL1 on stage1. As if the Linux address mapping was wrong.
> > Anyone has any ideas how this could happen? Maybe a reserved-memory
> > misconfiguration?
> >
> > I had issues with loading the driver in the first place. Apparently
> swiotlb is used, maybe it can cause this. I also tried to enable CMA.
> > config.txt:
> > dtoverlay=vc4-fkms-v3d,cma=320M@0x0-0x40000000
> > gpu_mem=128
>
> Also looking at your other reply and the implementation of
> vc4_bo_create, it looks like this is a CMA problem.
>
> It would be good to run a test with the swiotlb-xen disabled:
>
> diff --git a/arch/arm/xen/mm.c b/arch/arm/xen/mm.c
> index 467fa225c3d0..2bdd12785d14 100644
> --- a/arch/arm/xen/mm.c
> +++ b/arch/arm/xen/mm.c
> @@ -138,8 +138,7 @@ void xen_destroy_contiguous_region(phys_addr_t pstart,
> unsigned int order)
> static int __init xen_mm_init(void)
> {
> struct gnttab_cache_flush cflush;
> - if (!xen_initial_domain())
> - return 0;
> + return 0;
> xen_swiotlb_init(1, false);
>
> cflush.op = 0;
>
With this change the kernel is not booting up. (btw. I'm using USB SSD for
my OS.)
[ 0.071081] bcm2835-dma fe007000.dma: Unable to set DMA mask
[ 0.076277] bcm2835-dma fe007b00.dma: Unable to set DMA mask
(XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=25: not implemented
(XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=15: not implemented
[ 0.592695] pci 0000:00:00.0: Failed to add - passthrough or MSI/MSI-X
might fail!
(XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=15: not implemented
[ 0.606819] pci 0000:01:00.0: Failed to add - passthrough or MSI/MSI-X
might fail!
[ 1.212820] usb 1-1: device descriptor read/64, error 18
[ 1.452815] usb 1-1: device descriptor read/64, error 18
[ 1.820813] usb 1-1: device descriptor read/64, error 18
[ 2.060815] usb 1-1: device descriptor read/64, error 18
[ 2.845548] usb 1-1: device descriptor read/8, error -61
[ 2.977603] usb 1-1: device descriptor read/8, error -61
[ 3.237530] usb 1-1: device descriptor read/8, error -61
[ 3.369585] usb 1-1: device descriptor read/8, error -61
[ 3.480765] usb usb1-port1: unable to enumerate USB device
Traces stop here. I could try with a memory card. Maybe it makes a
difference.
>
> It is going to be fine just to boot Dom0 and DomUs without PV drivers.
> Also, can you post the device tree that you are using here? Just in case
> there is an issue with Xen parsing any possible /reserved-memory nodes
> with CMA info that need to be passed on to Dom0.
Here is the device dumped from command line:
dtc -I fs /proc/device-tree
https://drive.google.com/file/d/17u18dJHxRfbGZMtRXIwtLVZZfMj9KwN-/view?usp=sharing
>
>
> > > p.s.
> > > While testing I found issue with Xen master branch and your
> patch: xen/rpi4: implement watchdog-based reset
> > >
> > > Looks like black listing the bcm2835-pm
> > > @@ -37,12 +41,69 @@ static const struct dt_device_match
> rpi4_blacklist_dev[] __initconst =
> > > * The aux peripheral also shares a page with the aux UART.
> > > */
> > > DT_MATCH_COMPATIBLE("brcm,bcm2835-aux"),
> > > + /* Special device used for rebooting */
> > > + DT_MATCH_COMPATIBLE("brcm,bcm2835-pm"),
> > >
> > > will prevent v3d driver to locate phandle. I think it will use
> the same resource:
> > > pm: watchdog@7e100000 {
> > > compatible = "brcm,bcm2835-pm", "brcm,bcm2835-pm-wdt";
> > > #power-domain-cells = <1>;
> > > #reset-cells = <1>;
> > > reg = <0x7e100000 0x114>,
> > > <0x7e00a000 0x24>,
> > > <0x7ec11000 0x20>;
> > > clocks = <&clocks BCM2835_CLOCK_V3D>,
> > > <&clocks BCM2835_CLOCK_PERI_IMAGE>,
> > > <&clocks BCM2835_CLOCK_H264>,
> > > <&clocks BCM2835_CLOCK_ISP>;
> > > clock-names = "v3d", "peri_image", "h264", "isp";
> > > system-power-controller;
> > >
> > > };
> >
> > Yeah, I imagine it could be possible. Can you post the error
> message you
> > are seeing from the v3d driver?
> >
> > This is the error:
> > [ 0.069682] OF: /v3dbus/v3d@7ec04000: could not find phandle
> > [ 0.074828] OF: /v3dbus/v3d@7ec04000: could not find phandle
> > v3d driver is not loaded.
> >
> > --
> > Br,
> > Jukka Kaartinen
> >
> >
--
Br,
Jukka Kaartinen
[-- Attachment #2: Type: text/html, Size: 8711 bytes --]
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Question about xen and Rasp 4B
2021-01-26 10:59 ` Jukka Kaartinen
@ 2021-01-26 13:58 ` Jukka Kaartinen
2021-01-26 20:22 ` Stefano Stabellini
1 sibling, 0 replies; 33+ messages in thread
From: Jukka Kaartinen @ 2021-01-26 13:58 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Xen-devel, Roman Shaposhnik, Julien Grall, Bertrand Marquis
[-- Attachment #1: Type: text/plain, Size: 6734 bytes --]
On Tue, Jan 26, 2021 at 12:59 PM Jukka Kaartinen <jukka.kaartinen@unikie.com>
wrote:
>
>
> On Tue, Jan 26, 2021 at 2:54 AM Stefano Stabellini <sstabellini@kernel.org>
> wrote:
>
>> On Sat, 23 Jan 2021, Jukka Kaartinen wrote:
>> > Thanks for the response!
>> >
>> > On Sat, Jan 23, 2021 at 2:27 AM Stefano Stabellini <
>> sstabellini@kernel.org> wrote:
>> > + xen-devel, Roman,
>> >
>> >
>> > On Fri, 22 Jan 2021, Jukka Kaartinen wrote:
>> > > Hi Stefano,
>> > > I'm Jukka Kaartinen a SW developer working on enabling
>> hypervisors on mobile platforms. One of our HW that we use on
>> > development is
>> > > Raspberry Pi 4B. I wonder if you could help me a bit :).
>> > >
>> > > I'm trying to enable the GPU with Xen + Raspberry Pi for
>> > dom0.
>> https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323#p1797605
>> > >
>> > > I got so far that GPU drivers are loaded (v3d & vc4) without
>> errors. But now Xen returns error when X is starting:
>> > > (XEN) traps.c:1986:d0v1 HSR=0x93880045 pc=0x00007f97b14e70
>> gva=0x7f7f817000 gpa=0x0000401315d000
>> > > I tried to debug what causes this and looks
>> like find_mmio_handler cannot find handler.
>> > > (See more here:
>> https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323&start=25#p1801691
>> )
>> > >
>> > > Any ideas why the handler is not found?
>> >
>> >
>> > Hi Jukka,
>> >
>> > I am glad to hear that you are interested in Xen on RaspberryPi
>> :-) I
>> > haven't tried the GPU yet, I have been using the serial only.
>> > Roman, did you ever get the GPU working?
>> >
>> >
>> > The error is a data abort error: Linux is trying to access an
>> address
>> > which is not mapped to dom0. The address seems to be
>> 0x401315d000. It is
>> > a pretty high address; I looked in device tree but couldn't spot
>> it.
>> >
>> > >From the HSR (the syndrom register) it looks like it is a
>> translation
>> > fault at EL1 on stage1. As if the Linux address mapping was wrong.
>> > Anyone has any ideas how this could happen? Maybe a
>> reserved-memory
>> > misconfiguration?
>> >
>> > I had issues with loading the driver in the first place. Apparently
>> swiotlb is used, maybe it can cause this. I also tried to enable CMA.
>> > config.txt:
>> > dtoverlay=vc4-fkms-v3d,cma=320M@0x0-0x40000000
>> > gpu_mem=128
>>
>> Also looking at your other reply and the implementation of
>> vc4_bo_create, it looks like this is a CMA problem.
>>
>> It would be good to run a test with the swiotlb-xen disabled:
>>
>> diff --git a/arch/arm/xen/mm.c b/arch/arm/xen/mm.c
>> index 467fa225c3d0..2bdd12785d14 100644
>> --- a/arch/arm/xen/mm.c
>> +++ b/arch/arm/xen/mm.c
>> @@ -138,8 +138,7 @@ void xen_destroy_contiguous_region(phys_addr_t
>> pstart, unsigned int order)
>> static int __init xen_mm_init(void)
>> {
>> struct gnttab_cache_flush cflush;
>> - if (!xen_initial_domain())
>> - return 0;
>> + return 0;
>> xen_swiotlb_init(1, false);
>>
>> cflush.op = 0;
>>
> With this change the kernel is not booting up. (btw. I'm using USB SSD for
> my OS.)
> [ 0.071081] bcm2835-dma fe007000.dma: Unable to set DMA mask
> [ 0.076277] bcm2835-dma fe007b00.dma: Unable to set DMA mask
> (XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=25: not implemented
> (XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=15: not implemented
> [ 0.592695] pci 0000:00:00.0: Failed to add - passthrough or MSI/MSI-X
> might fail!
> (XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=15: not implemented
> [ 0.606819] pci 0000:01:00.0: Failed to add - passthrough or MSI/MSI-X
> might fail!
> [ 1.212820] usb 1-1: device descriptor read/64, error 18
> [ 1.452815] usb 1-1: device descriptor read/64, error 18
> [ 1.820813] usb 1-1: device descriptor read/64, error 18
> [ 2.060815] usb 1-1: device descriptor read/64, error 18
> [ 2.845548] usb 1-1: device descriptor read/8, error -61
> [ 2.977603] usb 1-1: device descriptor read/8, error -61
> [ 3.237530] usb 1-1: device descriptor read/8, error -61
> [ 3.369585] usb 1-1: device descriptor read/8, error -61
> [ 3.480765] usb usb1-port1: unable to enumerate USB device
>
> Traces stop here. I could try with a memory card. Maybe it makes a
> difference.
>
>
>>
>> It is going to be fine just to boot Dom0 and DomUs without PV drivers.
>> Also, can you post the device tree that you are using here? Just in case
>> there is an issue with Xen parsing any possible /reserved-memory nodes
>> with CMA info that need to be passed on to Dom0.
>
> Here is the device dumped from command line:
> dtc -I fs /proc/device-tree
>
> https://drive.google.com/file/d/17u18dJHxRfbGZMtRXIwtLVZZfMj9KwN-/view?usp=sharing
>
>
Here is log from XEN when it goes through the device tree:
https://drive.google.com/file/d/1jvT3ZNJeXHfxPOffiaRSRg8FPwQHS1mb/view?usp=sharing
>
>>
>> > > p.s.
>> > > While testing I found issue with Xen master branch and your
>> patch: xen/rpi4: implement watchdog-based reset
>> > >
>> > > Looks like black listing the bcm2835-pm
>> > > @@ -37,12 +41,69 @@ static const struct dt_device_match
>> rpi4_blacklist_dev[] __initconst =
>> > > * The aux peripheral also shares a page with the aux UART.
>> > > */
>> > > DT_MATCH_COMPATIBLE("brcm,bcm2835-aux"),
>> > > + /* Special device used for rebooting */
>> > > + DT_MATCH_COMPATIBLE("brcm,bcm2835-pm"),
>> > >
>> > > will prevent v3d driver to locate phandle. I think it will use
>> the same resource:
>> > > pm: watchdog@7e100000 {
>> > > compatible = "brcm,bcm2835-pm", "brcm,bcm2835-pm-wdt";
>> > > #power-domain-cells = <1>;
>> > > #reset-cells = <1>;
>> > > reg = <0x7e100000 0x114>,
>> > > <0x7e00a000 0x24>,
>> > > <0x7ec11000 0x20>;
>> > > clocks = <&clocks BCM2835_CLOCK_V3D>,
>> > > <&clocks BCM2835_CLOCK_PERI_IMAGE>,
>> > > <&clocks BCM2835_CLOCK_H264>,
>> > > <&clocks BCM2835_CLOCK_ISP>;
>> > > clock-names = "v3d", "peri_image", "h264", "isp";
>> > > system-power-controller;
>> > >
>> > > };
>> >
>> > Yeah, I imagine it could be possible. Can you post the error
>> message you
>> > are seeing from the v3d driver?
>> >
>> > This is the error:
>> > [ 0.069682] OF: /v3dbus/v3d@7ec04000: could not find phandle
>> > [ 0.074828] OF: /v3dbus/v3d@7ec04000: could not find phandle
>> > v3d driver is not loaded.
>> >
>> > --
>> > Br,
>> > Jukka Kaartinen
>> >
>> >
>
>
>
> --
> Br,
> Jukka Kaartinen
>
--
Br,
Jukka Kaartinen
[-- Attachment #2: Type: text/html, Size: 9747 bytes --]
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Question about xen and Rasp 4B
2021-01-26 10:59 ` Jukka Kaartinen
2021-01-26 13:58 ` Jukka Kaartinen
@ 2021-01-26 20:22 ` Stefano Stabellini
2021-01-27 11:47 ` Jukka Kaartinen
1 sibling, 1 reply; 33+ messages in thread
From: Stefano Stabellini @ 2021-01-26 20:22 UTC (permalink / raw)
To: Jukka Kaartinen
Cc: Stefano Stabellini, Xen-devel, Roman Shaposhnik, Julien Grall,
Bertrand Marquis
[-- Attachment #1: Type: text/plain, Size: 8285 bytes --]
On Tue, 26 Jan 2021, Jukka Kaartinen wrote:
> On Tue, Jan 26, 2021 at 2:54 AM Stefano Stabellini <sstabellini@kernel.org> wrote:
> On Sat, 23 Jan 2021, Jukka Kaartinen wrote:
> > Thanks for the response!
> >
> > On Sat, Jan 23, 2021 at 2:27 AM Stefano Stabellini <sstabellini@kernel.org> wrote:
> > + xen-devel, Roman,
> >
> >
> > On Fri, 22 Jan 2021, Jukka Kaartinen wrote:
> > > Hi Stefano,
> > > I'm Jukka Kaartinen a SW developer working on enabling hypervisors on mobile platforms. One of our HW that we use on
> > development is
> > > Raspberry Pi 4B. I wonder if you could help me a bit :).
> > >
> > > I'm trying to enable the GPU with Xen + Raspberry Pi for
> > dom0. https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323#p1797605
> > >
> > > I got so far that GPU drivers are loaded (v3d & vc4) without errors. But now Xen returns error when X is starting:
> > > (XEN) traps.c:1986:d0v1 HSR=0x93880045 pc=0x00007f97b14e70 gva=0x7f7f817000 gpa=0x0000401315d000
> > > I tried to debug what causes this and looks like find_mmio_handler cannot find handler.
> > > (See more here: https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323&start=25#p1801691 )
> > >
> > > Any ideas why the handler is not found?
> >
> >
> > Hi Jukka,
> >
> > I am glad to hear that you are interested in Xen on RaspberryPi :-) I
> > haven't tried the GPU yet, I have been using the serial only.
> > Roman, did you ever get the GPU working?
> >
> >
> > The error is a data abort error: Linux is trying to access an address
> > which is not mapped to dom0. The address seems to be 0x401315d000. It is
> > a pretty high address; I looked in device tree but couldn't spot it.
> >
> > >From the HSR (the syndrom register) it looks like it is a translation
> > fault at EL1 on stage1. As if the Linux address mapping was wrong.
> > Anyone has any ideas how this could happen? Maybe a reserved-memory
> > misconfiguration?
> >
> > I had issues with loading the driver in the first place. Apparently swiotlb is used, maybe it can cause this. I also tried to
> enable CMA.
> > config.txt:
> > dtoverlay=vc4-fkms-v3d,cma=320M@0x0-0x40000000
> > gpu_mem=128
>
> Also looking at your other reply and the implementation of
> vc4_bo_create, it looks like this is a CMA problem.
>
> It would be good to run a test with the swiotlb-xen disabled:
>
> diff --git a/arch/arm/xen/mm.c b/arch/arm/xen/mm.c
> index 467fa225c3d0..2bdd12785d14 100644
> --- a/arch/arm/xen/mm.c
> +++ b/arch/arm/xen/mm.c
> @@ -138,8 +138,7 @@ void xen_destroy_contiguous_region(phys_addr_t pstart, unsigned int order)
> static int __init xen_mm_init(void)
> {
> struct gnttab_cache_flush cflush;
> - if (!xen_initial_domain())
> - return 0;
> + return 0;
> xen_swiotlb_init(1, false);
>
> cflush.op = 0;
>
> With this change the kernel is not booting up. (btw. I'm using USB SSD for my OS.)
> [ 0.071081] bcm2835-dma fe007000.dma: Unable to set DMA mask
> [ 0.076277] bcm2835-dma fe007b00.dma: Unable to set DMA mask
> (XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=25: not implemented
> (XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=15: not implemented
> [ 0.592695] pci 0000:00:00.0: Failed to add - passthrough or MSI/MSI-X might fail!
> (XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=15: not implemented
> [ 0.606819] pci 0000:01:00.0: Failed to add - passthrough or MSI/MSI-X might fail!
> [ 1.212820] usb 1-1: device descriptor read/64, error 18
> [ 1.452815] usb 1-1: device descriptor read/64, error 18
> [ 1.820813] usb 1-1: device descriptor read/64, error 18
> [ 2.060815] usb 1-1: device descriptor read/64, error 18
> [ 2.845548] usb 1-1: device descriptor read/8, error -61
> [ 2.977603] usb 1-1: device descriptor read/8, error -61
> [ 3.237530] usb 1-1: device descriptor read/8, error -61
> [ 3.369585] usb 1-1: device descriptor read/8, error -61
> [ 3.480765] usb usb1-port1: unable to enumerate USB device
>
> Traces stop here. I could try with a memory card. Maybe it makes a difference.
This is very surprising. Disabling swiotlb-xen should make things better
not worse. The only reason I can think of why it could make things worse
is if Linux runs out of low memory. Julien's patch
437b0aa06a014ce174e24c0d3530b3e9ab19b18b for Xen should have addressed
that issue though. Julien, any ideas?
> It is going to be fine just to boot Dom0 and DomUs without PV drivers.
> Also, can you post the device tree that you are using here? Just in case
> there is an issue with Xen parsing any possible /reserved-memory nodes
> with CMA info that need to be passed on to Dom0.
>
> Here is the device dumped from command line:
> dtc -I fs /proc/device-tree
> https://drive.google.com/file/d/17u18dJHxRfbGZMtRXIwtLVZZfMj9KwN-/view?usp=sharing
There is a reserved-memory node:
reserved-memory {
#address-cells = <0x02>;
#size-cells = <0x01>;
ranges;
phandle = <0x3f>;
linux,cma {
linux,cma-default;
alloc-ranges = <0x00 0x00 0x30000000>;
compatible = "shared-dma-pool";
size = <0x10000000>;
phandle = <0x40>;
reusable;
};
};
But in theory Xen should be able to export it to Dom0. It would be worth
verifying that by running the same dtc -I fs /proc/device-tree in dom0 (in
the dom0 configuration that can finish booting of course) you get the
same reserved-memory node.
There is also another suspicious node here:
axi {
vc_mem {
reg = <0x3eb00000 0x3ff00000 0xc0000000>;
};
};
Which doesn't seem device tree exactly compliant and maybe GPU related.
But again, Xen would probably export it as is to Dom0.
> > > p.s.
> > > While testing I found issue with Xen master branch and your patch: xen/rpi4: implement watchdog-based reset
> > >
> > > Looks like black listing the bcm2835-pm
> > > @@ -37,12 +41,69 @@ static const struct dt_device_match rpi4_blacklist_dev[] __initconst =
> > > * The aux peripheral also shares a page with the aux UART.
> > > */
> > > DT_MATCH_COMPATIBLE("brcm,bcm2835-aux"),
> > > + /* Special device used for rebooting */
> > > + DT_MATCH_COMPATIBLE("brcm,bcm2835-pm"),
> > >
> > > will prevent v3d driver to locate phandle. I think it will use the same resource:
> > > pm: watchdog@7e100000 {
> > > compatible = "brcm,bcm2835-pm", "brcm,bcm2835-pm-wdt";
> > > #power-domain-cells = <1>;
> > > #reset-cells = <1>;
> > > reg = <0x7e100000 0x114>,
> > > <0x7e00a000 0x24>,
> > > <0x7ec11000 0x20>;
> > > clocks = <&clocks BCM2835_CLOCK_V3D>,
> > > <&clocks BCM2835_CLOCK_PERI_IMAGE>,
> > > <&clocks BCM2835_CLOCK_H264>,
> > > <&clocks BCM2835_CLOCK_ISP>;
> > > clock-names = "v3d", "peri_image", "h264", "isp";
> > > system-power-controller;
> > >
> > > };
> >
> > Yeah, I imagine it could be possible. Can you post the error message you
> > are seeing from the v3d driver?
> >
> > This is the error:
> > [ 0.069682] OF: /v3dbus/v3d@7ec04000: could not find phandle
> > [ 0.074828] OF: /v3dbus/v3d@7ec04000: could not find phandle
> > v3d driver is not loaded.
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Question about xen and Rasp 4B
2021-01-26 20:22 ` Stefano Stabellini
@ 2021-01-27 11:47 ` Jukka Kaartinen
2021-01-28 1:10 ` Stefano Stabellini
2021-01-29 18:50 ` Julien Grall
0 siblings, 2 replies; 33+ messages in thread
From: Jukka Kaartinen @ 2021-01-27 11:47 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Xen-devel, Roman Shaposhnik, Julien Grall, Bertrand Marquis
[-- Attachment #1: Type: text/plain, Size: 10098 bytes --]
On Tue, Jan 26, 2021 at 10:22 PM Stefano Stabellini <sstabellini@kernel.org>
wrote:
> On Tue, 26 Jan 2021, Jukka Kaartinen wrote:
> > On Tue, Jan 26, 2021 at 2:54 AM Stefano Stabellini <
> sstabellini@kernel.org> wrote:
> > On Sat, 23 Jan 2021, Jukka Kaartinen wrote:
> > > Thanks for the response!
> > >
> > > On Sat, Jan 23, 2021 at 2:27 AM Stefano Stabellini <
> sstabellini@kernel.org> wrote:
> > > + xen-devel, Roman,
> > >
> > >
> > > On Fri, 22 Jan 2021, Jukka Kaartinen wrote:
> > > > Hi Stefano,
> > > > I'm Jukka Kaartinen a SW developer working on enabling
> hypervisors on mobile platforms. One of our HW that we use on
> > > development is
> > > > Raspberry Pi 4B. I wonder if you could help me a bit :).
> > > >
> > > > I'm trying to enable the GPU with Xen + Raspberry Pi for
> > > dom0.
> https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323#p1797605
> > > >
> > > > I got so far that GPU drivers are loaded (v3d & vc4)
> without errors. But now Xen returns error when X is starting:
> > > > (XEN) traps.c:1986:d0v1 HSR=0x93880045
> pc=0x00007f97b14e70 gva=0x7f7f817000 gpa=0x0000401315d000
> > > > I tried to debug what causes this and looks
> like find_mmio_handler cannot find handler.
> > > > (See more here:
> https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323&start=25#p1801691
> )
> > > >
> > > > Any ideas why the handler is not found?
> > >
> > >
> > > Hi Jukka,
> > >
> > > I am glad to hear that you are interested in Xen on
> RaspberryPi :-) I
> > > haven't tried the GPU yet, I have been using the serial
> only.
> > > Roman, did you ever get the GPU working?
> > >
> > >
> > > The error is a data abort error: Linux is trying to access
> an address
> > > which is not mapped to dom0. The address seems to be
> 0x401315d000. It is
> > > a pretty high address; I looked in device tree but
> couldn't spot it.
> > >
> > > >From the HSR (the syndrom register) it looks like it is a
> translation
> > > fault at EL1 on stage1. As if the Linux address mapping
> was wrong.
> > > Anyone has any ideas how this could happen? Maybe a
> reserved-memory
> > > misconfiguration?
> > >
> > > I had issues with loading the driver in the first place.
> Apparently swiotlb is used, maybe it can cause this. I also tried to
> > enable CMA.
> > > config.txt:
> > > dtoverlay=vc4-fkms-v3d,cma=320M@0x0-0x40000000
> > > gpu_mem=128
> >
> > Also looking at your other reply and the implementation of
> > vc4_bo_create, it looks like this is a CMA problem.
> >
> > It would be good to run a test with the swiotlb-xen disabled:
> >
> > diff --git a/arch/arm/xen/mm.c b/arch/arm/xen/mm.c
> > index 467fa225c3d0..2bdd12785d14 100644
> > --- a/arch/arm/xen/mm.c
> > +++ b/arch/arm/xen/mm.c
> > @@ -138,8 +138,7 @@ void xen_destroy_contiguous_region(phys_addr_t
> pstart, unsigned int order)
> > static int __init xen_mm_init(void)
> > {
> > struct gnttab_cache_flush cflush;
> > - if (!xen_initial_domain())
> > - return 0;
> > + return 0;
> > xen_swiotlb_init(1, false);
> >
> > cflush.op = 0;
> >
> > With this change the kernel is not booting up. (btw. I'm using USB SSD
> for my OS.)
> > [ 0.071081] bcm2835-dma fe007000.dma: Unable to set DMA mask
> > [ 0.076277] bcm2835-dma fe007b00.dma: Unable to set DMA mask
> > (XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=25: not implemented
> > (XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=15: not implemented
> > [ 0.592695] pci 0000:00:00.0: Failed to add - passthrough or
> MSI/MSI-X might fail!
> > (XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=15: not implemented
> > [ 0.606819] pci 0000:01:00.0: Failed to add - passthrough or
> MSI/MSI-X might fail!
> > [ 1.212820] usb 1-1: device descriptor read/64, error 18
> > [ 1.452815] usb 1-1: device descriptor read/64, error 18
> > [ 1.820813] usb 1-1: device descriptor read/64, error 18
> > [ 2.060815] usb 1-1: device descriptor read/64, error 18
> > [ 2.845548] usb 1-1: device descriptor read/8, error -61
> > [ 2.977603] usb 1-1: device descriptor read/8, error -61
> > [ 3.237530] usb 1-1: device descriptor read/8, error -61
> > [ 3.369585] usb 1-1: device descriptor read/8, error -61
> > [ 3.480765] usb usb1-port1: unable to enumerate USB device
> >
> > Traces stop here. I could try with a memory card. Maybe it makes a
> difference.
>
> This is very surprising. Disabling swiotlb-xen should make things better
> not worse. The only reason I can think of why it could make things worse
> is if Linux runs out of low memory. Julien's patch
> 437b0aa06a014ce174e24c0d3530b3e9ab19b18b for Xen should have addressed
> that issue though. Julien, any ideas?
>
I really don't know if this is a problem but in the
allocate_memory_11 arch_get_dma_bitsize is called. That should return the
platform->dma_bitsize but at the early stage of boot platform is not
initialized so default 32 is returned. I tried changing the hard code from
32 -> 30 but it didn't make any difference.
static void __init allocate_memory_11(struct domain *d,
struct kernel_info *kinfo)
{
const unsigned int min_low_order =
get_order_from_bytes(min_t(paddr_t, dom0_mem, MB(128)));
const unsigned int min_order = get_order_from_bytes(MB(4));
struct page_info *pg;
unsigned int order = get_allocation_size(kinfo->unassigned_mem);
int i;
bool lowmem = true;
unsigned int lowmem_bitsize = min(32U, *arch_get_dma_bitsize*());
also here the place where static dma_bitsize is set is not called for dom0
void __init end_boot_allocator(void)
{
....
if ( !dma_bitsize && (num_online_nodes() > 1) )
dma_bitsize = arch_get_dma_bitsize();
and will lead alloc_domheap_pages not to use dma_bitsize since it is not
set.
struct page_info *alloc_domheap_pages(
struct domain *d, unsigned int order, unsigned int memflags)
{
uses static: dma_bitsize and currently is not set for raspberry pi.
>
> > It is going to be fine just to boot Dom0 and DomUs without PV
> drivers.
> > Also, can you post the device tree that you are using here? Just
> in case
> > there is an issue with Xen parsing any possible /reserved-memory
> nodes
> > with CMA info that need to be passed on to Dom0.
> >
> > Here is the device dumped from command line:
> > dtc -I fs /proc/device-tree
> >
> https://drive.google.com/file/d/17u18dJHxRfbGZMtRXIwtLVZZfMj9KwN-/view?usp=sharing
>
> There is a reserved-memory node:
>
> reserved-memory {
> #address-cells = <0x02>;
> #size-cells = <0x01>;
> ranges;
> phandle = <0x3f>;
>
> linux,cma {
> linux,cma-default;
> alloc-ranges = <0x00 0x00 0x30000000>;
> compatible = "shared-dma-pool";
> size = <0x10000000>;
> phandle = <0x40>;
> reusable;
> };
> };
>
> But in theory Xen should be able to export it to Dom0. It would be worth
> verifying that by running the same dtc -I fs /proc/device-tree in dom0 (in
> the dom0 configuration that can finish booting of course) you get the
> same reserved-memory node.
>
Actually the device-tree is exported from the dom0.
> There is also another suspicious node here:
>
> axi {
> vc_mem {
> reg = <0x3eb00000 0x3ff00000 0xc0000000>;
> };
> };
>
> Which doesn't seem device tree exactly compliant and maybe GPU related.
> But again, Xen would probably export it as is to Dom0.
>
>
>
> > > > p.s.
> > > > While testing I found issue with Xen master branch and
> your patch: xen/rpi4: implement watchdog-based reset
> > > >
> > > > Looks like black listing the bcm2835-pm
> > > > @@ -37,12 +41,69 @@ static const struct dt_device_match
> rpi4_blacklist_dev[] __initconst =
> > > > * The aux peripheral also shares a page with the
> aux UART.
> > > > */
> > > > DT_MATCH_COMPATIBLE("brcm,bcm2835-aux"),
> > > > + /* Special device used for rebooting */
> > > > + DT_MATCH_COMPATIBLE("brcm,bcm2835-pm"),
> > > >
> > > > will prevent v3d driver to locate phandle. I think it
> will use the same resource:
> > > > pm: watchdog@7e100000 {
> > > > compatible = "brcm,bcm2835-pm",
> "brcm,bcm2835-pm-wdt";
> > > > #power-domain-cells = <1>;
> > > > #reset-cells = <1>;
> > > > reg = <0x7e100000 0x114>,
> > > > <0x7e00a000 0x24>,
> > > > <0x7ec11000 0x20>;
> > > > clocks = <&clocks BCM2835_CLOCK_V3D>,
> > > > <&clocks BCM2835_CLOCK_PERI_IMAGE>,
> > > > <&clocks BCM2835_CLOCK_H264>,
> > > > <&clocks BCM2835_CLOCK_ISP>;
> > > > clock-names = "v3d", "peri_image", "h264", "isp";
> > > > system-power-controller;
> > > >
> > > > };
> > >
> > > Yeah, I imagine it could be possible. Can you post the
> error message you
> > > are seeing from the v3d driver?
> > >
> > > This is the error:
> > > [ 0.069682] OF: /v3dbus/v3d@7ec04000: could not find phandle
> > > [ 0.074828] OF: /v3dbus/v3d@7ec04000: could not find phandle
> > > v3d driver is not loaded.
--
Br,
Jukka Kaartinen
[-- Attachment #2: Type: text/html, Size: 14693 bytes --]
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Question about xen and Rasp 4B
2021-01-27 11:47 ` Jukka Kaartinen
@ 2021-01-28 1:10 ` Stefano Stabellini
2021-01-28 2:19 ` Roman Shaposhnik
2021-01-30 1:42 ` Stefano Stabellini
2021-01-29 18:50 ` Julien Grall
1 sibling, 2 replies; 33+ messages in thread
From: Stefano Stabellini @ 2021-01-28 1:10 UTC (permalink / raw)
To: Jukka Kaartinen
Cc: Stefano Stabellini, Xen-devel, Roman Shaposhnik, Julien Grall,
Bertrand Marquis
[-- Attachment #1: Type: text/plain, Size: 12443 bytes --]
On Wed, 27 Jan 2021, Jukka Kaartinen wrote:
> On Tue, Jan 26, 2021 at 10:22 PM Stefano Stabellini <sstabellini@kernel.org> wrote:
> On Tue, 26 Jan 2021, Jukka Kaartinen wrote:
> > On Tue, Jan 26, 2021 at 2:54 AM Stefano Stabellini <sstabellini@kernel.org> wrote:
> > On Sat, 23 Jan 2021, Jukka Kaartinen wrote:
> > > Thanks for the response!
> > >
> > > On Sat, Jan 23, 2021 at 2:27 AM Stefano Stabellini <sstabellini@kernel.org> wrote:
> > > + xen-devel, Roman,
> > >
> > >
> > > On Fri, 22 Jan 2021, Jukka Kaartinen wrote:
> > > > Hi Stefano,
> > > > I'm Jukka Kaartinen a SW developer working on enabling hypervisors on mobile platforms. One of our HW that we
> use on
> > > development is
> > > > Raspberry Pi 4B. I wonder if you could help me a bit :).
> > > >
> > > > I'm trying to enable the GPU with Xen + Raspberry Pi for
> > > dom0. https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323#p1797605
> > > >
> > > > I got so far that GPU drivers are loaded (v3d & vc4) without errors. But now Xen returns error when X is
> starting:
> > > > (XEN) traps.c:1986:d0v1 HSR=0x93880045 pc=0x00007f97b14e70 gva=0x7f7f817000 gpa=0x0000401315d000
> > > > I tried to debug what causes this and looks like find_mmio_handler cannot find handler.
> > > > (See more here: https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323&start=25#p1801691 )
> > > >
> > > > Any ideas why the handler is not found?
> > >
> > >
> > > Hi Jukka,
> > >
> > > I am glad to hear that you are interested in Xen on RaspberryPi :-) I
> > > haven't tried the GPU yet, I have been using the serial only.
> > > Roman, did you ever get the GPU working?
> > >
> > >
> > > The error is a data abort error: Linux is trying to access an address
> > > which is not mapped to dom0. The address seems to be 0x401315d000. It is
> > > a pretty high address; I looked in device tree but couldn't spot it.
> > >
> > > >From the HSR (the syndrom register) it looks like it is a translation
> > > fault at EL1 on stage1. As if the Linux address mapping was wrong.
> > > Anyone has any ideas how this could happen? Maybe a reserved-memory
> > > misconfiguration?
> > >
> > > I had issues with loading the driver in the first place. Apparently swiotlb is used, maybe it can cause this. I also
> tried to
> > enable CMA.
> > > config.txt:
> > > dtoverlay=vc4-fkms-v3d,cma=320M@0x0-0x40000000
> > > gpu_mem=128
> >
> > Also looking at your other reply and the implementation of
> > vc4_bo_create, it looks like this is a CMA problem.
> >
> > It would be good to run a test with the swiotlb-xen disabled:
> >
> > diff --git a/arch/arm/xen/mm.c b/arch/arm/xen/mm.c
> > index 467fa225c3d0..2bdd12785d14 100644
> > --- a/arch/arm/xen/mm.c
> > +++ b/arch/arm/xen/mm.c
> > @@ -138,8 +138,7 @@ void xen_destroy_contiguous_region(phys_addr_t pstart, unsigned int order)
> > static int __init xen_mm_init(void)
> > {
> > struct gnttab_cache_flush cflush;
> > - if (!xen_initial_domain())
> > - return 0;
> > + return 0;
> > xen_swiotlb_init(1, false);
> >
> > cflush.op = 0;
> >
> > With this change the kernel is not booting up. (btw. I'm using USB SSD for my OS.)
> > [ 0.071081] bcm2835-dma fe007000.dma: Unable to set DMA mask
> > [ 0.076277] bcm2835-dma fe007b00.dma: Unable to set DMA mask
> > (XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=25: not implemented
> > (XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=15: not implemented
> > [ 0.592695] pci 0000:00:00.0: Failed to add - passthrough or MSI/MSI-X might fail!
> > (XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=15: not implemented
> > [ 0.606819] pci 0000:01:00.0: Failed to add - passthrough or MSI/MSI-X might fail!
> > [ 1.212820] usb 1-1: device descriptor read/64, error 18
> > [ 1.452815] usb 1-1: device descriptor read/64, error 18
> > [ 1.820813] usb 1-1: device descriptor read/64, error 18
> > [ 2.060815] usb 1-1: device descriptor read/64, error 18
> > [ 2.845548] usb 1-1: device descriptor read/8, error -61
> > [ 2.977603] usb 1-1: device descriptor read/8, error -61
> > [ 3.237530] usb 1-1: device descriptor read/8, error -61
> > [ 3.369585] usb 1-1: device descriptor read/8, error -61
> > [ 3.480765] usb usb1-port1: unable to enumerate USB device
> >
> > Traces stop here. I could try with a memory card. Maybe it makes a difference.
>
> This is very surprising. Disabling swiotlb-xen should make things better
> not worse. The only reason I can think of why it could make things worse
> is if Linux runs out of low memory. Julien's patch
> 437b0aa06a014ce174e24c0d3530b3e9ab19b18b for Xen should have addressed
> that issue though. Julien, any ideas?
>
> I really don't know if this is a problem but in the allocate_memory_11 arch_get_dma_bitsize is called. That should return the
> platform->dma_bitsize but at the early stage of boot platform is not initialized so default 32 is returned. I tried changing the hard code
> from 32 -> 30 but it didn't make any difference.
> static void __init allocate_memory_11(struct domain *d,
> struct kernel_info *kinfo)
> {
> const unsigned int min_low_order =
> get_order_from_bytes(min_t(paddr_t, dom0_mem, MB(128)));
> const unsigned int min_order = get_order_from_bytes(MB(4));
> struct page_info *pg;
> unsigned int order = get_allocation_size(kinfo->unassigned_mem);
> int i;
>
> bool lowmem = true;
> unsigned int lowmem_bitsize = min(32U, arch_get_dma_bitsize());
>
> also here the place where static dma_bitsize is set is not called for dom0
>
> void __init end_boot_allocator(void)
> {
> ....
> if ( !dma_bitsize && (num_online_nodes() > 1) )
> dma_bitsize = arch_get_dma_bitsize();
>
>
> and will lead alloc_domheap_pages not to use dma_bitsize since it is not set.
> struct page_info *alloc_domheap_pages(
> struct domain *d, unsigned int order, unsigned int memflags)
> {
>
> uses static: dma_bitsize and currently is not set for raspberry pi.
You can easily check from the device tree in dom0 what memory range was
allocated for it by Xen. The memory node is simply called "memory". We
might be able to understand if the range is too high from there.
FYI I have just ordered a micro HDMI cable so I might be able to provide
more useful feedback in the following days.
> > It is going to be fine just to boot Dom0 and DomUs without PV drivers.
> > Also, can you post the device tree that you are using here? Just in case
> > there is an issue with Xen parsing any possible /reserved-memory nodes
> > with CMA info that need to be passed on to Dom0.
> >
> > Here is the device dumped from command line:
> > dtc -I fs /proc/device-tree
> > https://drive.google.com/file/d/17u18dJHxRfbGZMtRXIwtLVZZfMj9KwN-/view?usp=sharing
>
> There is a reserved-memory node:
>
> reserved-memory {
> #address-cells = <0x02>;
> #size-cells = <0x01>;
> ranges;
> phandle = <0x3f>;
>
> linux,cma {
> linux,cma-default;
> alloc-ranges = <0x00 0x00 0x30000000>;
> compatible = "shared-dma-pool";
> size = <0x10000000>;
> phandle = <0x40>;
> reusable;
> };
> };
>
> But in theory Xen should be able to export it to Dom0. It would be worth
> verifying that by running the same dtc -I fs /proc/device-tree in dom0 (in
> the dom0 configuration that can finish booting of course) you get the
> same reserved-memory node.
>
> Actually the device-tree is exported from the dom0.
That's good
> There is also another suspicious node here:
>
> axi {
> vc_mem {
> reg = <0x3eb00000 0x3ff00000 0xc0000000>;
> };
> };
>
> Which doesn't seem device tree exactly compliant and maybe GPU related.
> But again, Xen would probably export it as is to Dom0.
>
>
>
> > > > p.s.
> > > > While testing I found issue with Xen master branch and your patch: xen/rpi4: implement watchdog-based reset
> > > >
> > > > Looks like black listing the bcm2835-pm
> > > > @@ -37,12 +41,69 @@ static const struct dt_device_match rpi4_blacklist_dev[] __initconst =
> > > > * The aux peripheral also shares a page with the aux UART.
> > > > */
> > > > DT_MATCH_COMPATIBLE("brcm,bcm2835-aux"),
> > > > + /* Special device used for rebooting */
> > > > + DT_MATCH_COMPATIBLE("brcm,bcm2835-pm"),
> > > >
> > > > will prevent v3d driver to locate phandle. I think it will use the same resource:
> > > > pm: watchdog@7e100000 {
> > > > compatible = "brcm,bcm2835-pm", "brcm,bcm2835-pm-wdt";
> > > > #power-domain-cells = <1>;
> > > > #reset-cells = <1>;
> > > > reg = <0x7e100000 0x114>,
> > > > <0x7e00a000 0x24>,
> > > > <0x7ec11000 0x20>;
> > > > clocks = <&clocks BCM2835_CLOCK_V3D>,
> > > > <&clocks BCM2835_CLOCK_PERI_IMAGE>,
> > > > <&clocks BCM2835_CLOCK_H264>,
> > > > <&clocks BCM2835_CLOCK_ISP>;
> > > > clock-names = "v3d", "peri_image", "h264", "isp";
> > > > system-power-controller;
> > > >
> > > > };
> > >
> > > Yeah, I imagine it could be possible. Can you post the error message you
> > > are seeing from the v3d driver?
> > >
> > > This is the error:
> > > [ 0.069682] OF: /v3dbus/v3d@7ec04000: could not find phandle
> > > [ 0.074828] OF: /v3dbus/v3d@7ec04000: could not find phandle
> > > v3d driver is not loaded.
>
>
>
> --
> Br,
> Jukka Kaartinen
>
>
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Question about xen and Rasp 4B
2021-01-28 1:10 ` Stefano Stabellini
@ 2021-01-28 2:19 ` Roman Shaposhnik
2021-01-30 1:42 ` Stefano Stabellini
1 sibling, 0 replies; 33+ messages in thread
From: Roman Shaposhnik @ 2021-01-28 2:19 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Jukka Kaartinen, Xen-devel, Julien Grall, Bertrand Marquis
On Wed, Jan 27, 2021 at 5:10 PM Stefano Stabellini
<sstabellini@kernel.org> wrote:
>
> On Wed, 27 Jan 2021, Jukka Kaartinen wrote:
> > On Tue, Jan 26, 2021 at 10:22 PM Stefano Stabellini <sstabellini@kernel.org> wrote:
> > On Tue, 26 Jan 2021, Jukka Kaartinen wrote:
> > > On Tue, Jan 26, 2021 at 2:54 AM Stefano Stabellini <sstabellini@kernel.org> wrote:
> > > On Sat, 23 Jan 2021, Jukka Kaartinen wrote:
> > > > Thanks for the response!
> > > >
> > > > On Sat, Jan 23, 2021 at 2:27 AM Stefano Stabellini <sstabellini@kernel.org> wrote:
> > > > + xen-devel, Roman,
> > > >
> > > >
> > > > On Fri, 22 Jan 2021, Jukka Kaartinen wrote:
> > > > > Hi Stefano,
> > > > > I'm Jukka Kaartinen a SW developer working on enabling hypervisors on mobile platforms. One of our HW that we
> > use on
> > > > development is
> > > > > Raspberry Pi 4B. I wonder if you could help me a bit :).
> > > > >
> > > > > I'm trying to enable the GPU with Xen + Raspberry Pi for
> > > > dom0. https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323#p1797605
> > > > >
> > > > > I got so far that GPU drivers are loaded (v3d & vc4) without errors. But now Xen returns error when X is
> > starting:
> > > > > (XEN) traps.c:1986:d0v1 HSR=0x93880045 pc=0x00007f97b14e70 gva=0x7f7f817000 gpa=0x0000401315d000
> > > > > I tried to debug what causes this and looks like find_mmio_handler cannot find handler.
> > > > > (See more here: https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323&start=25#p1801691 )
> > > > >
> > > > > Any ideas why the handler is not found?
> > > >
> > > >
> > > > Hi Jukka,
> > > >
> > > > I am glad to hear that you are interested in Xen on RaspberryPi :-) I
> > > > haven't tried the GPU yet, I have been using the serial only.
> > > > Roman, did you ever get the GPU working?
> > > >
> > > >
> > > > The error is a data abort error: Linux is trying to access an address
> > > > which is not mapped to dom0. The address seems to be 0x401315d000. It is
> > > > a pretty high address; I looked in device tree but couldn't spot it.
> > > >
> > > > >From the HSR (the syndrom register) it looks like it is a translation
> > > > fault at EL1 on stage1. As if the Linux address mapping was wrong.
> > > > Anyone has any ideas how this could happen? Maybe a reserved-memory
> > > > misconfiguration?
> > > >
> > > > I had issues with loading the driver in the first place. Apparently swiotlb is used, maybe it can cause this. I also
> > tried to
> > > enable CMA.
> > > > config.txt:
> > > > dtoverlay=vc4-fkms-v3d,cma=320M@0x0-0x40000000
> > > > gpu_mem=128
> > >
> > > Also looking at your other reply and the implementation of
> > > vc4_bo_create, it looks like this is a CMA problem.
> > >
> > > It would be good to run a test with the swiotlb-xen disabled:
> > >
> > > diff --git a/arch/arm/xen/mm.c b/arch/arm/xen/mm.c
> > > index 467fa225c3d0..2bdd12785d14 100644
> > > --- a/arch/arm/xen/mm.c
> > > +++ b/arch/arm/xen/mm.c
> > > @@ -138,8 +138,7 @@ void xen_destroy_contiguous_region(phys_addr_t pstart, unsigned int order)
> > > static int __init xen_mm_init(void)
> > > {
> > > struct gnttab_cache_flush cflush;
> > > - if (!xen_initial_domain())
> > > - return 0;
> > > + return 0;
> > > xen_swiotlb_init(1, false);
> > >
> > > cflush.op = 0;
> > >
> > > With this change the kernel is not booting up. (btw. I'm using USB SSD for my OS.)
> > > [ 0.071081] bcm2835-dma fe007000.dma: Unable to set DMA mask
> > > [ 0.076277] bcm2835-dma fe007b00.dma: Unable to set DMA mask
> > > (XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=25: not implemented
> > > (XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=15: not implemented
> > > [ 0.592695] pci 0000:00:00.0: Failed to add - passthrough or MSI/MSI-X might fail!
> > > (XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=15: not implemented
> > > [ 0.606819] pci 0000:01:00.0: Failed to add - passthrough or MSI/MSI-X might fail!
> > > [ 1.212820] usb 1-1: device descriptor read/64, error 18
> > > [ 1.452815] usb 1-1: device descriptor read/64, error 18
> > > [ 1.820813] usb 1-1: device descriptor read/64, error 18
> > > [ 2.060815] usb 1-1: device descriptor read/64, error 18
> > > [ 2.845548] usb 1-1: device descriptor read/8, error -61
> > > [ 2.977603] usb 1-1: device descriptor read/8, error -61
> > > [ 3.237530] usb 1-1: device descriptor read/8, error -61
> > > [ 3.369585] usb 1-1: device descriptor read/8, error -61
> > > [ 3.480765] usb usb1-port1: unable to enumerate USB device
> > >
> > > Traces stop here. I could try with a memory card. Maybe it makes a difference.
> >
> > This is very surprising. Disabling swiotlb-xen should make things better
> > not worse. The only reason I can think of why it could make things worse
> > is if Linux runs out of low memory. Julien's patch
> > 437b0aa06a014ce174e24c0d3530b3e9ab19b18b for Xen should have addressed
> > that issue though. Julien, any ideas?
> >
> > I really don't know if this is a problem but in the allocate_memory_11 arch_get_dma_bitsize is called. That should return the
> > platform->dma_bitsize but at the early stage of boot platform is not initialized so default 32 is returned. I tried changing the hard code
> > from 32 -> 30 but it didn't make any difference.
> > static void __init allocate_memory_11(struct domain *d,
> > struct kernel_info *kinfo)
> > {
> > const unsigned int min_low_order =
> > get_order_from_bytes(min_t(paddr_t, dom0_mem, MB(128)));
> > const unsigned int min_order = get_order_from_bytes(MB(4));
> > struct page_info *pg;
> > unsigned int order = get_allocation_size(kinfo->unassigned_mem);
> > int i;
> >
> > bool lowmem = true;
> > unsigned int lowmem_bitsize = min(32U, arch_get_dma_bitsize());
> >
> > also here the place where static dma_bitsize is set is not called for dom0
> >
> > void __init end_boot_allocator(void)
> > {
> > ....
> > if ( !dma_bitsize && (num_online_nodes() > 1) )
> > dma_bitsize = arch_get_dma_bitsize();
> >
> >
> > and will lead alloc_domheap_pages not to use dma_bitsize since it is not set.
> > struct page_info *alloc_domheap_pages(
> > struct domain *d, unsigned int order, unsigned int memflags)
> > {
> >
> > uses static: dma_bitsize and currently is not set for raspberry pi.
>
> You can easily check from the device tree in dom0 what memory range was
> allocated for it by Xen. The memory node is simply called "memory". We
> might be able to understand if the range is too high from there.
>
> FYI I have just ordered a micro HDMI cable so I might be able to provide
> more useful feedback in the following days.
Speaking of which -- I'm also curious where this leads (esp. given our
use of newer,
non-Rpi specific kernels) so I can run all sorts of experiments with EVE too.
Thanks,
Roman.
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Question about xen and Rasp 4B
2021-01-27 11:47 ` Jukka Kaartinen
2021-01-28 1:10 ` Stefano Stabellini
@ 2021-01-29 18:50 ` Julien Grall
2021-01-30 1:37 ` Stefano Stabellini
2021-01-30 13:53 ` Jukka Kaartinen
1 sibling, 2 replies; 33+ messages in thread
From: Julien Grall @ 2021-01-29 18:50 UTC (permalink / raw)
To: Jukka Kaartinen, Stefano Stabellini
Cc: Xen-devel, Roman Shaposhnik, Bertrand Marquis
Hi,
@Jukka, would it be possible to at least configure your client to quote
with '>'? This would make easier to understand who wrote what
(tabulation is not great for that).
If you are using gmail, then configuring it to send it as plain text
should do the job.
On 27/01/2021 11:47, Jukka Kaartinen wrote:
>
>
> On Tue, Jan 26, 2021 at 10:22 PM Stefano Stabellini
> <sstabellini@kernel.org <mailto:sstabellini@kernel.org>> wrote:
>
> On Tue, 26 Jan 2021, Jukka Kaartinen wrote:
> > On Tue, Jan 26, 2021 at 2:54 AM Stefano Stabellini
> <sstabellini@kernel.org <mailto:sstabellini@kernel.org>> wrote:
> > On Sat, 23 Jan 2021, Jukka Kaartinen wrote:
> > > Thanks for the response!
> > >
> > > On Sat, Jan 23, 2021 at 2:27 AM Stefano Stabellini
> <sstabellini@kernel.org <mailto:sstabellini@kernel.org>> wrote:
> > > + xen-devel, Roman,
> > >
> > >
> > > On Fri, 22 Jan 2021, Jukka Kaartinen wrote:
> > > > Hi Stefano,
> > > > I'm Jukka Kaartinen a SW developer working on
> enabling hypervisors on mobile platforms. One of our HW that we use on
> > > development is
> > > > Raspberry Pi 4B. I wonder if you could help me a
> bit :).
> > > >
> > > > I'm trying to enable the GPU with Xen + Raspberry
> Pi for
> > > dom0.
> https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323#p1797605
> <https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323#p1797605>
> > > >
> > > > I got so far that GPU drivers are loaded (v3d &
> vc4) without errors. But now Xen returns error when X is starting:
> > > > (XEN) traps.c:1986:d0v1 HSR=0x93880045
> pc=0x00007f97b14e70 gva=0x7f7f817000 gpa=0x0000401315d000
> > > > I tried to debug what causes this and looks
> like find_mmio_handler cannot find handler.
> > > > (See more here:
> https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323&start=25#p1801691
> <https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323&start=25#p1801691>
> )
> > > >
> > > > Any ideas why the handler is not found?
> > >
> > >
> > > Hi Jukka,
> > >
> > > I am glad to hear that you are interested in Xen on
> RaspberryPi :-) I
> > > haven't tried the GPU yet, I have been using the
> serial only.
> > > Roman, did you ever get the GPU working?
> > >
> > >
> > > The error is a data abort error: Linux is trying to
> access an address
> > > which is not mapped to dom0. The address seems to
> be 0x401315d000. It is
> > > a pretty high address; I looked in device tree but
> couldn't spot it.
> > >
> > > >From the HSR (the syndrom register) it looks like
> it is a translation
> > > fault at EL1 on stage1. As if the Linux address
> mapping was wrong.
> > > Anyone has any ideas how this could happen? Maybe a
> reserved-memory
> > > misconfiguration?
> > >
> > > I had issues with loading the driver in the first place.
> Apparently swiotlb is used, maybe it can cause this. I also tried to
> > enable CMA.
> > > config.txt:
> > > dtoverlay=vc4-fkms-v3d,cma=320M@0x0-0x40000000
> > > gpu_mem=128
> >
> > Also looking at your other reply and the implementation of
> > vc4_bo_create, it looks like this is a CMA problem.
> >
> > It would be good to run a test with the swiotlb-xen disabled:
> >
> > diff --git a/arch/arm/xen/mm.c b/arch/arm/xen/mm.c
> > index 467fa225c3d0..2bdd12785d14 100644
> > --- a/arch/arm/xen/mm.c
> > +++ b/arch/arm/xen/mm.c
> > @@ -138,8 +138,7 @@ void
> xen_destroy_contiguous_region(phys_addr_t pstart, unsigned int order)
> > static int __init xen_mm_init(void)
> > {
> > struct gnttab_cache_flush cflush;
> > - if (!xen_initial_domain())
> > - return 0;
> > + return 0;
> > xen_swiotlb_init(1, false);
> >
> > cflush.op = 0;
> >
> > With this change the kernel is not booting up. (btw. I'm using
> USB SSD for my OS.)
> > [ 0.071081] bcm2835-dma fe007000.dma: Unable to set DMA mask
> > [ 0.076277] bcm2835-dma fe007b00.dma: Unable to set DMA mask
> > (XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=25: not implemented
> > (XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=15: not implemented
> > [ 0.592695] pci 0000:00:00.0: Failed to add - passthrough or
> MSI/MSI-X might fail!
> > (XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=15: not implemented
> > [ 0.606819] pci 0000:01:00.0: Failed to add - passthrough or
> MSI/MSI-X might fail!
> > [ 1.212820] usb 1-1: device descriptor read/64, error 18
> > [ 1.452815] usb 1-1: device descriptor read/64, error 18
> > [ 1.820813] usb 1-1: device descriptor read/64, error 18
> > [ 2.060815] usb 1-1: device descriptor read/64, error 18
> > [ 2.845548] usb 1-1: device descriptor read/8, error -61
> > [ 2.977603] usb 1-1: device descriptor read/8, error -61
> > [ 3.237530] usb 1-1: device descriptor read/8, error -61
> > [ 3.369585] usb 1-1: device descriptor read/8, error -61
> > [ 3.480765] usb usb1-port1: unable to enumerate USB device
> >
> > Traces stop here. I could try with a memory card. Maybe it makes
> a difference.
>
> This is very surprising. Disabling swiotlb-xen should make things better
> not worse. The only reason I can think of why it could make things worse
> is if Linux runs out of low memory. Julien's patch
> 437b0aa06a014ce174e24c0d3530b3e9ab19b18b for Xen should have addressed
> that issue though. Julien, any ideas?
I think, Stefano's small patch is not enough to disable the swiotlb as
we will still override the DMA ops. You also likely want:
diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
index 8a8949174b1c..aa43e249ecdd 100644
--- a/arch/arm/mm/dma-mapping.c
+++ b/arch/arm/mm/dma-mapping.c
@@ -2279,7 +2279,7 @@ void arch_setup_dma_ops(struct device *dev, u64
dma_base, u64 size,
set_dma_ops(dev, dma_ops);
#ifdef CONFIG_XEN
- if (xen_initial_domain())
+ if (0 || xen_initial_domain())
dev->dma_ops = &xen_swiotlb_dma_ops;
#endif
dev->archdata.dma_ops_setup = true;
Otherwise, you would still use the swiotlb DMA ops that would not be
functional as we disabled the swiotlb.
This would explain the following error because it will check whether the
mask is valid using the callback dma_supported():
[ 0.071081] bcm2835-dma fe007000.dma: Unable to set DMA mask
>
> I really don't know if this is a problem but in the
> allocate_memory_11 arch_get_dma_bitsize is called. That should return
> the platform->dma_bitsize but at the early stage of boot platform is not
> initialized so default 32 is returned. I tried changing the hard code
> from 32 -> 30 but it didn't make any difference.
>
> static void __init allocate_memory_11(struct domain *d,
> struct kernel_info *kinfo)
> {
> const unsigned int min_low_order =
> get_order_from_bytes(min_t(paddr_t, dom0_mem, MB(128)));
> const unsigned int min_order = get_order_from_bytes(MB(4));
> struct page_info *pg;
> unsigned int order = get_allocation_size(kinfo->unassigned_mem);
> int i;
>
> bool lowmem = true;
> unsigned int lowmem_bitsize = min(32U, *arch_get_dma_bitsize*());
Domain memory allocation will always happen after platform_init() is
called. So the value returned here will be correct.
>
> also here the place where static dma_bitsize is set is not called for dom0
>
>
> void __init end_boot_allocator(void)
> {
> ....
> if ( !dma_bitsize && (num_online_nodes() > 1) )
> dma_bitsize = arch_get_dma_bitsize();
>
> and will lead alloc_domheap_pages not to use dma_bitsize since it is not
> set.
> struct page_info *alloc_domheap_pages(
> struct domain *d, unsigned int order, unsigned int memflags)
> {
>
> uses static: dma_bitsize and currently is not set for raspberry pi.
Right, this is normal because we don't support NUMA on Arm so
num_online_nodes() would always return 1.
But I don't yet see the problem with the existing code. We will still
try to allocate at least one bank below 30 bits on RPI.
From the log you provided:
(XEN) Allocating 1:1 mappings totalling 5120MB for dom0:
(XEN) BANK[0] 0x00000008000000-0x00000028000000 (512MB)
(XEN) BANK[1] 0x00000080000000-0x000000c0000000 (1024MB)
(XEN) BANK[2] 0x00000100000000-0x000001e0000000 (3584MB)
You have 512MB of memory allocated below the 30 bits mark.
Are you expecting more memory to be allocated below the 30-bits?
If you
Cheers,
--
Julien Grall
^ permalink raw reply related [flat|nested] 33+ messages in thread
* Re: Question about xen and Rasp 4B
2021-01-29 18:50 ` Julien Grall
@ 2021-01-30 1:37 ` Stefano Stabellini
2021-01-30 13:53 ` Jukka Kaartinen
1 sibling, 0 replies; 33+ messages in thread
From: Stefano Stabellini @ 2021-01-30 1:37 UTC (permalink / raw)
To: Julien Grall
Cc: Jukka Kaartinen, Stefano Stabellini, Xen-devel, Roman Shaposhnik,
Bertrand Marquis
[-- Attachment #1: Type: text/plain, Size: 8692 bytes --]
On Fri, 29 Jan 2021, Julien Grall wrote:
> Hi,
>
> @Jukka, would it be possible to at least configure your client to quote with
> '>'? This would make easier to understand who wrote what (tabulation is not
> great for that).
>
> If you are using gmail, then configuring it to send it as plain text should do
> the job.
>
> On 27/01/2021 11:47, Jukka Kaartinen wrote:
> >
> >
> > On Tue, Jan 26, 2021 at 10:22 PM Stefano Stabellini <sstabellini@kernel.org
> > <mailto:sstabellini@kernel.org>> wrote:
> >
> > On Tue, 26 Jan 2021, Jukka Kaartinen wrote:
> > > On Tue, Jan 26, 2021 at 2:54 AM Stefano Stabellini
> > <sstabellini@kernel.org <mailto:sstabellini@kernel.org>> wrote:
> > > On Sat, 23 Jan 2021, Jukka Kaartinen wrote:
> > > > Thanks for the response!
> > > >
> > > > On Sat, Jan 23, 2021 at 2:27 AM Stefano Stabellini
> > <sstabellini@kernel.org <mailto:sstabellini@kernel.org>> wrote:
> > > > + xen-devel, Roman,
> > > >
> > > >
> > > > On Fri, 22 Jan 2021, Jukka Kaartinen wrote:
> > > > > Hi Stefano,
> > > > > I'm Jukka Kaartinen a SW developer working on
> > enabling hypervisors on mobile platforms. One of our HW that we use on
> > > > development is
> > > > > Raspberry Pi 4B. I wonder if you could help me a
> > bit :).
> > > > >
> > > > > I'm trying to enable the GPU with Xen + Raspberry
> > Pi for
> > > > dom0.
> > https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323#p1797605
> > <https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323#p1797605>
> > > > >
> > > > > I got so far that GPU drivers are loaded (v3d &
> > vc4) without errors. But now Xen returns error when X is starting:
> > > > > (XEN) traps.c:1986:d0v1 HSR=0x93880045
> > pc=0x00007f97b14e70 gva=0x7f7f817000 gpa=0x0000401315d000
> > > > > I tried to debug what causes this and looks
> > like find_mmio_handler cannot find handler.
> > > > > (See more here:
> > https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323&start=25#p1801691
> > <https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323&start=25#p1801691>
> > )
> > > > >
> > > > > Any ideas why the handler is not found?
> > > >
> > > >
> > > > Hi Jukka,
> > > >
> > > > I am glad to hear that you are interested in Xen on
> > RaspberryPi :-) I
> > > > haven't tried the GPU yet, I have been using the
> > serial only.
> > > > Roman, did you ever get the GPU working?
> > > >
> > > >
> > > > The error is a data abort error: Linux is trying to
> > access an address
> > > > which is not mapped to dom0. The address seems to
> > be 0x401315d000. It is
> > > > a pretty high address; I looked in device tree but
> > couldn't spot it.
> > > >
> > > > >From the HSR (the syndrom register) it looks like
> > it is a translation
> > > > fault at EL1 on stage1. As if the Linux address
> > mapping was wrong.
> > > > Anyone has any ideas how this could happen? Maybe a
> > reserved-memory
> > > > misconfiguration?
> > > >
> > > > I had issues with loading the driver in the first place.
> > Apparently swiotlb is used, maybe it can cause this. I also tried to
> > > enable CMA.
> > > > config.txt:
> > > > dtoverlay=vc4-fkms-v3d,cma=320M@0x0-0x40000000
> > > > gpu_mem=128
> > >
> > > Also looking at your other reply and the implementation of
> > > vc4_bo_create, it looks like this is a CMA problem.
> > >
> > > It would be good to run a test with the swiotlb-xen disabled:
> > >
> > > diff --git a/arch/arm/xen/mm.c b/arch/arm/xen/mm.c
> > > index 467fa225c3d0..2bdd12785d14 100644
> > > --- a/arch/arm/xen/mm.c
> > > +++ b/arch/arm/xen/mm.c
> > > @@ -138,8 +138,7 @@ void
> > xen_destroy_contiguous_region(phys_addr_t pstart, unsigned int order)
> > > static int __init xen_mm_init(void)
> > > {
> > > struct gnttab_cache_flush cflush;
> > > - if (!xen_initial_domain())
> > > - return 0;
> > > + return 0;
> > > xen_swiotlb_init(1, false);
> > >
> > > cflush.op = 0;
> > >
> > > With this change the kernel is not booting up. (btw. I'm using
> > USB SSD for my OS.)
> > > [ 0.071081] bcm2835-dma fe007000.dma: Unable to set DMA mask
> > > [ 0.076277] bcm2835-dma fe007b00.dma: Unable to set DMA mask
> > > (XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=25: not implemented
> > > (XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=15: not implemented
> > > [ 0.592695] pci 0000:00:00.0: Failed to add - passthrough or
> > MSI/MSI-X might fail!
> > > (XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=15: not implemented
> > > [ 0.606819] pci 0000:01:00.0: Failed to add - passthrough or
> > MSI/MSI-X might fail!
> > > [ 1.212820] usb 1-1: device descriptor read/64, error 18
> > > [ 1.452815] usb 1-1: device descriptor read/64, error 18
> > > [ 1.820813] usb 1-1: device descriptor read/64, error 18
> > > [ 2.060815] usb 1-1: device descriptor read/64, error 18
> > > [ 2.845548] usb 1-1: device descriptor read/8, error -61
> > > [ 2.977603] usb 1-1: device descriptor read/8, error -61
> > > [ 3.237530] usb 1-1: device descriptor read/8, error -61
> > > [ 3.369585] usb 1-1: device descriptor read/8, error -61
> > > [ 3.480765] usb usb1-port1: unable to enumerate USB device
> > >
> > > Traces stop here. I could try with a memory card. Maybe it makes
> > a difference.
> >
> > This is very surprising. Disabling swiotlb-xen should make things better
> > not worse. The only reason I can think of why it could make things worse
> > is if Linux runs out of low memory. Julien's patch
> > 437b0aa06a014ce174e24c0d3530b3e9ab19b18b for Xen should have addressed
> > that issue though. Julien, any ideas?
>
> I think, Stefano's small patch is not enough to disable the swiotlb as we will
> still override the DMA ops. You also likely want:
>
> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
> index 8a8949174b1c..aa43e249ecdd 100644
> --- a/arch/arm/mm/dma-mapping.c
> +++ b/arch/arm/mm/dma-mapping.c
> @@ -2279,7 +2279,7 @@ void arch_setup_dma_ops(struct device *dev, u64
> dma_base, u64 size,
> set_dma_ops(dev, dma_ops);
>
> #ifdef CONFIG_XEN
> - if (xen_initial_domain())
> + if (0 || xen_initial_domain())
> dev->dma_ops = &xen_swiotlb_dma_ops;
> #endif
> dev->archdata.dma_ops_setup = true;
>
> Otherwise, you would still use the swiotlb DMA ops that would not be
> functional as we disabled the swiotlb.
>
> This would explain the following error because it will check whether the mask
> is valid using the callback dma_supported():
>
> [ 0.071081] bcm2835-dma fe007000.dma: Unable to set DMA mask
Well spotted! On older kernels the change to xen_mm_init was sufficient,
but on more recent kernels it is not enough anymore. Sorry about that!
BTW if (0 || xen_initial_domain()) would still return true so you might
want something like:
diff --git a/arch/arm64/mm/dma-mapping.c b/arch/arm64/mm/dma-mapping.c
index 93e87b287556..f73066bed3ec 100644
--- a/arch/arm64/mm/dma-mapping.c
+++ b/arch/arm64/mm/dma-mapping.c
@@ -53,7 +53,7 @@ void arch_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
iommu_setup_dma_ops(dev, dma_base, size);
#ifdef CONFIG_XEN
- if (xen_initial_domain())
+ if (0)
dev->dma_ops = &xen_swiotlb_dma_ops;
#endif
}
^ permalink raw reply related [flat|nested] 33+ messages in thread
* Re: Question about xen and Rasp 4B
2021-01-28 1:10 ` Stefano Stabellini
2021-01-28 2:19 ` Roman Shaposhnik
@ 2021-01-30 1:42 ` Stefano Stabellini
2021-01-30 13:44 ` Jukka Kaartinen
1 sibling, 1 reply; 33+ messages in thread
From: Stefano Stabellini @ 2021-01-30 1:42 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Jukka Kaartinen, Xen-devel, Roman Shaposhnik, Julien Grall,
Bertrand Marquis
On Wed, 27 Jan 2021, Stefano Stabellini wrote:
> FYI I have just ordered a micro HDMI cable so I might be able to provide
> more useful feedback in the following days.
What did you use to setup the graphic environment? Is it Ubuntu or
Raspbian? I am having issues setting up a distro with a "startx" that
works.
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Question about xen and Rasp 4B
2021-01-30 1:42 ` Stefano Stabellini
@ 2021-01-30 13:44 ` Jukka Kaartinen
2021-01-30 13:55 ` Julien Grall
0 siblings, 1 reply; 33+ messages in thread
From: Jukka Kaartinen @ 2021-01-30 13:44 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Xen-devel, Roman Shaposhnik, Julien Grall, Bertrand Marquis
On 30.1.2021 3.42, Stefano Stabellini wrote:
> On Wed, 27 Jan 2021, Stefano Stabellini wrote:
>> FYI I have just ordered a micro HDMI cable so I might be able to provide
>> more useful feedback in the following days.
>
> What did you use to setup the graphic environment? Is it Ubuntu or
> Raspbian? I am having issues setting up a distro with a "startx" that
> works.
>
I'm using Ubuntu mate. Plain Ubuntu seemed to be quite slow.
Btw. we noticed that cpufreq stays at 600MHz if it is not forced from
the config.txt
arm_freq=1500
force_turbo=1
This kernel trace might be related.
[ 0.746502] cpufreq-dt cpufreq-dt: failed register driver: -19
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Question about xen and Rasp 4B
2021-01-29 18:50 ` Julien Grall
2021-01-30 1:37 ` Stefano Stabellini
@ 2021-01-30 13:53 ` Jukka Kaartinen
2021-01-30 14:18 ` Julien Grall
2021-02-02 2:22 ` Roman Shaposhnik
1 sibling, 2 replies; 33+ messages in thread
From: Jukka Kaartinen @ 2021-01-30 13:53 UTC (permalink / raw)
To: Julien Grall, Stefano Stabellini
Cc: Xen-devel, Roman Shaposhnik, Bertrand Marquis
On 29.1.2021 20.50, Julien Grall wrote:
> Hi,
>
> @Jukka, would it be possible to at least configure your client to quote
> with '>'? This would make easier to understand who wrote what
> (tabulation is not great for that).
>
> If you are using gmail, then configuring it to send it as plain text
> should do the job.
Sorry about that. May it's now better. Yes it is gmail. Maybe
Thunderbird works better.
>
> On 27/01/2021 11:47, Jukka Kaartinen wrote:
>>
>>
>> On Tue, Jan 26, 2021 at 10:22 PM Stefano Stabellini
>> <sstabellini@kernel.org <mailto:sstabellini@kernel.org>> wrote:
>>
>> On Tue, 26 Jan 2021, Jukka Kaartinen wrote:
>> > On Tue, Jan 26, 2021 at 2:54 AM Stefano Stabellini
>> <sstabellini@kernel.org <mailto:sstabellini@kernel.org>> wrote:
>> > On Sat, 23 Jan 2021, Jukka Kaartinen wrote:
>> > > Thanks for the response!
>> > >
>> > > On Sat, Jan 23, 2021 at 2:27 AM Stefano Stabellini
>> <sstabellini@kernel.org <mailto:sstabellini@kernel.org>> wrote:
>> > > + xen-devel, Roman,
>> > >
>> > >
>> > > On Fri, 22 Jan 2021, Jukka Kaartinen wrote:
>> > > > Hi Stefano,
>> > > > I'm Jukka Kaartinen a SW developer working on
>> enabling hypervisors on mobile platforms. One of our HW that we
>> use on
>> > > development is
>> > > > Raspberry Pi 4B. I wonder if you could help me a
>> bit :).
>> > > >
>> > > > I'm trying to enable the GPU with Xen + Raspberry
>> Pi for
>> > > dom0.
>>
>> https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323#p1797605
>>
>> <https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323#p1797605>
>> > > >
>> > > > I got so far that GPU drivers are loaded (v3d &
>> vc4) without errors. But now Xen returns error when X is starting:
>> > > > (XEN) traps.c:1986:d0v1 HSR=0x93880045
>> pc=0x00007f97b14e70 gva=0x7f7f817000 gpa=0x0000401315d000
>> > > > I tried to debug what causes this and looks
>> like find_mmio_handler cannot find handler.
>> > > > (See more here:
>>
>> https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323&start=25#p1801691
>>
>>
>> <https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323&start=25#p1801691>
>>
>> )
>> > > >
>> > > > Any ideas why the handler is not found?
>> > >
>> > >
>> > > Hi Jukka,
>> > >
>> > > I am glad to hear that you are interested in Xen on
>> RaspberryPi :-) I
>> > > haven't tried the GPU yet, I have been using the
>> serial only.
>> > > Roman, did you ever get the GPU working?
>> > >
>> > >
>> > > The error is a data abort error: Linux is trying to
>> access an address
>> > > which is not mapped to dom0. The address seems to
>> be 0x401315d000. It is
>> > > a pretty high address; I looked in device tree but
>> couldn't spot it.
>> > >
>> > > >From the HSR (the syndrom register) it looks like
>> it is a translation
>> > > fault at EL1 on stage1. As if the Linux address
>> mapping was wrong.
>> > > Anyone has any ideas how this could happen? Maybe a
>> reserved-memory
>> > > misconfiguration?
>> > >
>> > > I had issues with loading the driver in the first place.
>> Apparently swiotlb is used, maybe it can cause this. I also tried to
>> > enable CMA.
>> > > config.txt:
>> > > dtoverlay=vc4-fkms-v3d,cma=320M@0x0-0x40000000
>> > > gpu_mem=128
>> >
>> > Also looking at your other reply and the implementation of
>> > vc4_bo_create, it looks like this is a CMA problem.
>> >
>> > It would be good to run a test with the swiotlb-xen
>> disabled:
>> >
>> > diff --git a/arch/arm/xen/mm.c b/arch/arm/xen/mm.c
>> > index 467fa225c3d0..2bdd12785d14 100644
>> > --- a/arch/arm/xen/mm.c
>> > +++ b/arch/arm/xen/mm.c
>> > @@ -138,8 +138,7 @@ void
>> xen_destroy_contiguous_region(phys_addr_t pstart, unsigned int order)
>> > static int __init xen_mm_init(void)
>> > {
>> > struct gnttab_cache_flush cflush;
>> > - if (!xen_initial_domain())
>> > - return 0;
>> > + return 0;
>> > xen_swiotlb_init(1, false);
>> >
>> > cflush.op = 0;
>> >
>> > With this change the kernel is not booting up. (btw. I'm using
>> USB SSD for my OS.)
>> > [ 0.071081] bcm2835-dma fe007000.dma: Unable to set DMA mask
>> > [ 0.076277] bcm2835-dma fe007b00.dma: Unable to set DMA mask
>> > (XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=25: not implemented
>> > (XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=15: not implemented
>> > [ 0.592695] pci 0000:00:00.0: Failed to add - passthrough or
>> MSI/MSI-X might fail!
>> > (XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=15: not implemented
>> > [ 0.606819] pci 0000:01:00.0: Failed to add - passthrough or
>> MSI/MSI-X might fail!
>> > [ 1.212820] usb 1-1: device descriptor read/64, error 18
>> > [ 1.452815] usb 1-1: device descriptor read/64, error 18
>> > [ 1.820813] usb 1-1: device descriptor read/64, error 18
>> > [ 2.060815] usb 1-1: device descriptor read/64, error 18
>> > [ 2.845548] usb 1-1: device descriptor read/8, error -61
>> > [ 2.977603] usb 1-1: device descriptor read/8, error -61
>> > [ 3.237530] usb 1-1: device descriptor read/8, error -61
>> > [ 3.369585] usb 1-1: device descriptor read/8, error -61
>> > [ 3.480765] usb usb1-port1: unable to enumerate USB device
>> >
>> > Traces stop here. I could try with a memory card. Maybe it makes
>> a difference.
>>
>> This is very surprising. Disabling swiotlb-xen should make things
>> better
>> not worse. The only reason I can think of why it could make things
>> worse
>> is if Linux runs out of low memory. Julien's patch
>> 437b0aa06a014ce174e24c0d3530b3e9ab19b18b for Xen should have
>> addressed
>> that issue though. Julien, any ideas?
>
> I think, Stefano's small patch is not enough to disable the swiotlb as
> we will still override the DMA ops. You also likely want:
>
> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
> index 8a8949174b1c..aa43e249ecdd 100644
> --- a/arch/arm/mm/dma-mapping.c
> +++ b/arch/arm/mm/dma-mapping.c
> @@ -2279,7 +2279,7 @@ void arch_setup_dma_ops(struct device *dev, u64
> dma_base, u64 size,
> set_dma_ops(dev, dma_ops);
>
> #ifdef CONFIG_XEN
> - if (xen_initial_domain())
> + if (0 || xen_initial_domain())
> dev->dma_ops = &xen_swiotlb_dma_ops;
> #endif
> dev->archdata.dma_ops_setup = true;
>
> Otherwise, you would still use the swiotlb DMA ops that would not be
> functional as we disabled the swiotlb.
>
> This would explain the following error because it will check whether the
> mask is valid using the callback dma_supported():
>
> [ 0.071081] bcm2835-dma fe007000.dma: Unable to set DMA mask
>
Good catch.
GPU works now and I can start X! Thanks! I was also able to create domU
that runs Raspian OS.
Now that swiotlb is disabled what does it mean?
And also can I pass the GPU to domU? Raspberry Pi 4 is limited HW and
doesn't have IOMMU. I'm trying to create similar OS like QubesOS where
GPU, Network, keyboard/mouse, ... are isolated to their own VMs.
>>
>> I really don't know if this is a problem but in the
>> allocate_memory_11 arch_get_dma_bitsize is called. That should return
>> the platform->dma_bitsize but at the early stage of boot platform is
>> not initialized so default 32 is returned. I tried changing the hard
>> code from 32 -> 30 but it didn't make any difference.
>>
>> static void __init allocate_memory_11(struct domain *d,
>> struct kernel_info *kinfo)
>> {
>> const unsigned int min_low_order =
>> get_order_from_bytes(min_t(paddr_t, dom0_mem, MB(128)));
>> const unsigned int min_order = get_order_from_bytes(MB(4));
>> struct page_info *pg;
>> unsigned int order = get_allocation_size(kinfo->unassigned_mem);
>> int i;
>>
>> bool lowmem = true;
>> unsigned int lowmem_bitsize = min(32U,
>> *arch_get_dma_bitsize*());
>
> Domain memory allocation will always happen after platform_init() is
> called. So the value returned here will be correct.
I see.
>
>>
>> also here the place where static dma_bitsize is set is not called for
>> dom0
>>
>>
>> void __init end_boot_allocator(void)
>> {
>> ....
>> if ( !dma_bitsize && (num_online_nodes() > 1) )
>> dma_bitsize = arch_get_dma_bitsize();
>>
>> and will lead alloc_domheap_pages not to use dma_bitsize since it is
>> not set.
>> struct page_info *alloc_domheap_pages(
>> struct domain *d, unsigned int order, unsigned int memflags)
>> {
>>
>> uses static: dma_bitsize and currently is not set for raspberry pi.
>
> Right, this is normal because we don't support NUMA on Arm so
> num_online_nodes() would always return 1.
>
> But I don't yet see the problem with the existing code. We will still
> try to allocate at least one bank below 30 bits on RPI.
>
> From the log you provided:
> (XEN) Allocating 1:1 mappings totalling 5120MB for dom0:
> (XEN) BANK[0] 0x00000008000000-0x00000028000000 (512MB)
> (XEN) BANK[1] 0x00000080000000-0x000000c0000000 (1024MB)
> (XEN) BANK[2] 0x00000100000000-0x000001e0000000 (3584MB)
>
> You have 512MB of memory allocated below the 30 bits mark.
>
> Are you expecting more memory to be allocated below the 30-bits?
No not really and looks like it is enough since I can run X. :)
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Question about xen and Rasp 4B
2021-01-30 13:44 ` Jukka Kaartinen
@ 2021-01-30 13:55 ` Julien Grall
0 siblings, 0 replies; 33+ messages in thread
From: Julien Grall @ 2021-01-30 13:55 UTC (permalink / raw)
To: Jukka Kaartinen, Stefano Stabellini
Cc: Xen-devel, Roman Shaposhnik, Bertrand Marquis
Hi,
On 30/01/2021 13:44, Jukka Kaartinen wrote:
>
>
> On 30.1.2021 3.42, Stefano Stabellini wrote:
>> On Wed, 27 Jan 2021, Stefano Stabellini wrote:
>>> FYI I have just ordered a micro HDMI cable so I might be able to provide
>>> more useful feedback in the following days.
>>
>> What did you use to setup the graphic environment? Is it Ubuntu or
>> Raspbian? I am having issues setting up a distro with a "startx" that
>> works.
>>
> I'm using Ubuntu mate. Plain Ubuntu seemed to be quite slow.
> Btw. we noticed that cpufreq stays at 600MHz if it is not forced from
> the config.txt
>
> arm_freq=1500
> force_turbo=1
>
>
> This kernel trace might be related.
> [ 0.746502] cpufreq-dt cpufreq-dt: failed register driver: -19
>
This is normal, Dom0 doesn't see the physical CPUs, instead they are all
virtual and /cpus is recreated.
For a proper solution, either dom0 would need to be modified to
understandn the different between vCPUs and pCPUs or we will want to
implement the CPUFreq in Xen.
It might be possible to have a simpler solution on some setup (for
instance if you have each pCPU dedicated to a single vCPU).
Cheers,
--
Julien Grall
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Question about xen and Rasp 4B
2021-01-30 13:53 ` Jukka Kaartinen
@ 2021-01-30 14:18 ` Julien Grall
2021-02-01 19:25 ` Stefano Stabellini
2021-02-02 2:22 ` Roman Shaposhnik
1 sibling, 1 reply; 33+ messages in thread
From: Julien Grall @ 2021-01-30 14:18 UTC (permalink / raw)
To: Jukka Kaartinen, Stefano Stabellini
Cc: Xen-devel, Roman Shaposhnik, Bertrand Marquis
Hi Jukka,
On 30/01/2021 13:53, Jukka Kaartinen wrote:
>
>
> On 29.1.2021 20.50, Julien Grall wrote:
>> Hi,
>>
>> @Jukka, would it be possible to at least configure your client to
>> quote with '>'? This would make easier to understand who wrote what
>> (tabulation is not great for that).
>>
>> If you are using gmail, then configuring it to send it as plain text
>> should do the job.
> Sorry about that. May it's now better. Yes it is gmail. Maybe
> Thunderbird works better.
It should be possible to do it with the gmail web interface. Although, I
know it has a few quirks when dealing with plain text.
Anyway, your reply is quoting with > now. Thank you for that!
>
>>
>> On 27/01/2021 11:47, Jukka Kaartinen wrote:
>>>
>>>
>>> On Tue, Jan 26, 2021 at 10:22 PM Stefano Stabellini
>>> <sstabellini@kernel.org <mailto:sstabellini@kernel.org>> wrote:
>>>
>>> On Tue, 26 Jan 2021, Jukka Kaartinen wrote:
>>> > On Tue, Jan 26, 2021 at 2:54 AM Stefano Stabellini
>>> <sstabellini@kernel.org <mailto:sstabellini@kernel.org>> wrote:
>>> > On Sat, 23 Jan 2021, Jukka Kaartinen wrote:
>>> > > Thanks for the response!
>>> > >
>>> > > On Sat, Jan 23, 2021 at 2:27 AM Stefano Stabellini
>>> <sstabellini@kernel.org <mailto:sstabellini@kernel.org>> wrote:
>>> > > + xen-devel, Roman,
>>> > >
>>> > >
>>> > > On Fri, 22 Jan 2021, Jukka Kaartinen wrote:
>>> > > > Hi Stefano,
>>> > > > I'm Jukka Kaartinen a SW developer working on
>>> enabling hypervisors on mobile platforms. One of our HW that we
>>> use on
>>> > > development is
>>> > > > Raspberry Pi 4B. I wonder if you could help me a
>>> bit :).
>>> > > >
>>> > > > I'm trying to enable the GPU with Xen + Raspberry
>>> Pi for
>>> > > dom0.
>>> https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323#p1797605
>>> <https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323#p1797605>
>>>
>>> > > >
>>> > > > I got so far that GPU drivers are loaded (v3d &
>>> vc4) without errors. But now Xen returns error when X is starting:
>>> > > > (XEN) traps.c:1986:d0v1 HSR=0x93880045
>>> pc=0x00007f97b14e70 gva=0x7f7f817000 gpa=0x0000401315d000
>>> > > > I tried to debug what causes this and looks
>>> like find_mmio_handler cannot find handler.
>>> > > > (See more here:
>>> https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323&start=25#p1801691
>>>
>>> <https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323&start=25#p1801691>
>>>
>>> )
>>> > > >
>>> > > > Any ideas why the handler is not found?
>>> > >
>>> > >
>>> > > Hi Jukka,
>>> > >
>>> > > I am glad to hear that you are interested in Xen on
>>> RaspberryPi :-) I
>>> > > haven't tried the GPU yet, I have been using the
>>> serial only.
>>> > > Roman, did you ever get the GPU working?
>>> > >
>>> > >
>>> > > The error is a data abort error: Linux is trying to
>>> access an address
>>> > > which is not mapped to dom0. The address seems to
>>> be 0x401315d000. It is
>>> > > a pretty high address; I looked in device tree but
>>> couldn't spot it.
>>> > >
>>> > > >From the HSR (the syndrom register) it looks like
>>> it is a translation
>>> > > fault at EL1 on stage1. As if the Linux address
>>> mapping was wrong.
>>> > > Anyone has any ideas how this could happen? Maybe a
>>> reserved-memory
>>> > > misconfiguration?
>>> > >
>>> > > I had issues with loading the driver in the first place.
>>> Apparently swiotlb is used, maybe it can cause this. I also tried to
>>> > enable CMA.
>>> > > config.txt:
>>> > > dtoverlay=vc4-fkms-v3d,cma=320M@0x0-0x40000000
>>> > > gpu_mem=128
>>> >
>>> > Also looking at your other reply and the implementation of
>>> > vc4_bo_create, it looks like this is a CMA problem.
>>> >
>>> > It would be good to run a test with the swiotlb-xen
>>> disabled:
>>> >
>>> > diff --git a/arch/arm/xen/mm.c b/arch/arm/xen/mm.c
>>> > index 467fa225c3d0..2bdd12785d14 100644
>>> > --- a/arch/arm/xen/mm.c
>>> > +++ b/arch/arm/xen/mm.c
>>> > @@ -138,8 +138,7 @@ void
>>> xen_destroy_contiguous_region(phys_addr_t pstart, unsigned int
>>> order)
>>> > static int __init xen_mm_init(void)
>>> > {
>>> > struct gnttab_cache_flush cflush;
>>> > - if (!xen_initial_domain())
>>> > - return 0;
>>> > + return 0;
>>> > xen_swiotlb_init(1, false);
>>> >
>>> > cflush.op = 0;
>>> >
>>> > With this change the kernel is not booting up. (btw. I'm using
>>> USB SSD for my OS.)
>>> > [ 0.071081] bcm2835-dma fe007000.dma: Unable to set DMA mask
>>> > [ 0.076277] bcm2835-dma fe007b00.dma: Unable to set DMA mask
>>> > (XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=25: not implemented
>>> > (XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=15: not implemented
>>> > [ 0.592695] pci 0000:00:00.0: Failed to add - passthrough or
>>> MSI/MSI-X might fail!
>>> > (XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=15: not implemented
>>> > [ 0.606819] pci 0000:01:00.0: Failed to add - passthrough or
>>> MSI/MSI-X might fail!
>>> > [ 1.212820] usb 1-1: device descriptor read/64, error 18
>>> > [ 1.452815] usb 1-1: device descriptor read/64, error 18
>>> > [ 1.820813] usb 1-1: device descriptor read/64, error 18
>>> > [ 2.060815] usb 1-1: device descriptor read/64, error 18
>>> > [ 2.845548] usb 1-1: device descriptor read/8, error -61
>>> > [ 2.977603] usb 1-1: device descriptor read/8, error -61
>>> > [ 3.237530] usb 1-1: device descriptor read/8, error -61
>>> > [ 3.369585] usb 1-1: device descriptor read/8, error -61
>>> > [ 3.480765] usb usb1-port1: unable to enumerate USB device
>>> >
>>> > Traces stop here. I could try with a memory card. Maybe it makes
>>> a difference.
>>>
>>> This is very surprising. Disabling swiotlb-xen should make things
>>> better
>>> not worse. The only reason I can think of why it could make
>>> things worse
>>> is if Linux runs out of low memory. Julien's patch
>>> 437b0aa06a014ce174e24c0d3530b3e9ab19b18b for Xen should have
>>> addressed
>>> that issue though. Julien, any ideas?
>>
>> I think, Stefano's small patch is not enough to disable the swiotlb as
>> we will still override the DMA ops. You also likely want:
>>
>> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
>> index 8a8949174b1c..aa43e249ecdd 100644
>> --- a/arch/arm/mm/dma-mapping.c
>> +++ b/arch/arm/mm/dma-mapping.c
>> @@ -2279,7 +2279,7 @@ void arch_setup_dma_ops(struct device *dev, u64
>> dma_base, u64 size,
>> set_dma_ops(dev, dma_ops);
>>
>> #ifdef CONFIG_XEN
>> - if (xen_initial_domain())
>> + if (0 || xen_initial_domain())
>> dev->dma_ops = &xen_swiotlb_dma_ops;
>> #endif
>> dev->archdata.dma_ops_setup = true;
>>
>> Otherwise, you would still use the swiotlb DMA ops that would not be
>> functional as we disabled the swiotlb.
>>
>> This would explain the following error because it will check whether
>> the mask is valid using the callback dma_supported():
>>
>> [ 0.071081] bcm2835-dma fe007000.dma: Unable to set DMA mask
>>
> Good catch.
> GPU works now and I can start X! Thanks! I was also able to create domU
> that runs Raspian OS.
Glad to hear it works! IIRC, the swiotlb may become necessary when
running guest if the guest memory ends up to be used for DMA transaction.
>
> Now that swiotlb is disabled what does it mean?
I can see two reasons:
1) You have limited memory below the 30 bits mark. So Swiotlb and CMA
may try to fight for the low memory.
2) We found a few conversion bugs in the swiotlb on RPI4 last year
(IIRC the DMA and physical address may be different). I looked at the
Linux branch you are using and they seem to all be there. So there might
be another bug.
I am not sure how to figure out where is the problem. Stefano, do you
have a suggestion where to start?
>
> And also can I pass the GPU to domU? Raspberry Pi 4 is limited HW and
> doesn't have IOMMU. I'm trying to create similar OS like QubesOS where
> GPU, Network, keyboard/mouse, ... are isolated to their own VMs.
Without an IOMMU or any other HW mechamisns (e.g. MPU), it would not be
safe to assign a DMA-capable device to a non-trusted VM.
If you trust the VM where you assigned a device, then a possible
approach would be to have the VM direct mapped (e.g. guest physical
address == host physical address). Although, I can foreese some issues
if you have multiple VMs requires memory below 30 bits (there seem to be
limited amount)>
If you don't trust the VM where you assigned a device, then your best
option will be to expose a PV interface of the device and have your
backend sanitizing the request and issuing it on behalf of the guest.
Cheers,
--
Julien Grall
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Question about xen and Rasp 4B
2021-01-30 14:18 ` Julien Grall
@ 2021-02-01 19:25 ` Stefano Stabellini
2021-02-02 7:47 ` Jukka Kaartinen
2021-02-04 6:58 ` Jukka Kaartinen
0 siblings, 2 replies; 33+ messages in thread
From: Stefano Stabellini @ 2021-02-01 19:25 UTC (permalink / raw)
To: Julien Grall
Cc: Jukka Kaartinen, Stefano Stabellini, Xen-devel, Roman Shaposhnik,
Bertrand Marquis
[-- Attachment #1: Type: text/plain, Size: 12849 bytes --]
On Sat, 30 Jan 2021, Julien Grall wrote:
> > > On 27/01/2021 11:47, Jukka Kaartinen wrote:
> > > >
> > > >
> > > > On Tue, Jan 26, 2021 at 10:22 PM Stefano Stabellini
> > > > <sstabellini@kernel.org <mailto:sstabellini@kernel.org>> wrote:
> > > >
> > > > On Tue, 26 Jan 2021, Jukka Kaartinen wrote:
> > > > > On Tue, Jan 26, 2021 at 2:54 AM Stefano Stabellini
> > > > <sstabellini@kernel.org <mailto:sstabellini@kernel.org>> wrote:
> > > > > On Sat, 23 Jan 2021, Jukka Kaartinen wrote:
> > > > > > Thanks for the response!
> > > > > >
> > > > > > On Sat, Jan 23, 2021 at 2:27 AM Stefano Stabellini
> > > > <sstabellini@kernel.org <mailto:sstabellini@kernel.org>> wrote:
> > > > > > + xen-devel, Roman,
> > > > > >
> > > > > >
> > > > > > On Fri, 22 Jan 2021, Jukka Kaartinen wrote:
> > > > > > > Hi Stefano,
> > > > > > > I'm Jukka Kaartinen a SW developer working on
> > > > enabling hypervisors on mobile platforms. One of our HW that we use
> > > > on
> > > > > > development is
> > > > > > > Raspberry Pi 4B. I wonder if you could help me a
> > > > bit :).
> > > > > > >
> > > > > > > I'm trying to enable the GPU with Xen + Raspberry
> > > > Pi for
> > > > > > dom0.
> > > > https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323#p1797605
> > > > <https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323#p1797605>
> > > > > > >
> > > > > > > I got so far that GPU drivers are loaded (v3d &
> > > > vc4) without errors. But now Xen returns error when X is starting:
> > > > > > > (XEN) traps.c:1986:d0v1 HSR=0x93880045
> > > > pc=0x00007f97b14e70 gva=0x7f7f817000 gpa=0x0000401315d000
> > > > > > > I tried to debug what causes this and looks
> > > > like find_mmio_handler cannot find handler.
> > > > > > > (See more here:
> > > > https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323&start=25#p1801691
> > > > <https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323&start=25#p1801691>
> > > > )
> > > > > > >
> > > > > > > Any ideas why the handler is not found?
> > > > > >
> > > > > >
> > > > > > Hi Jukka,
> > > > > >
> > > > > > I am glad to hear that you are interested in Xen on
> > > > RaspberryPi :-) I
> > > > > > haven't tried the GPU yet, I have been using the
> > > > serial only.
> > > > > > Roman, did you ever get the GPU working?
> > > > > >
> > > > > >
> > > > > > The error is a data abort error: Linux is trying to
> > > > access an address
> > > > > > which is not mapped to dom0. The address seems to
> > > > be 0x401315d000. It is
> > > > > > a pretty high address; I looked in device tree but
> > > > couldn't spot it.
> > > > > >
> > > > > > >From the HSR (the syndrom register) it looks like
> > > > it is a translation
> > > > > > fault at EL1 on stage1. As if the Linux address
> > > > mapping was wrong.
> > > > > > Anyone has any ideas how this could happen? Maybe a
> > > > reserved-memory
> > > > > > misconfiguration?
> > > > > >
> > > > > > I had issues with loading the driver in the first place.
> > > > Apparently swiotlb is used, maybe it can cause this. I also tried to
> > > > > enable CMA.
> > > > > > config.txt:
> > > > > > dtoverlay=vc4-fkms-v3d,cma=320M@0x0-0x40000000
> > > > > > gpu_mem=128
> > > > >
> > > > > Also looking at your other reply and the implementation of
> > > > > vc4_bo_create, it looks like this is a CMA problem.
> > > > >
> > > > > It would be good to run a test with the swiotlb-xen
> > > > disabled:
> > > > >
> > > > > diff --git a/arch/arm/xen/mm.c b/arch/arm/xen/mm.c
> > > > > index 467fa225c3d0..2bdd12785d14 100644
> > > > > --- a/arch/arm/xen/mm.c
> > > > > +++ b/arch/arm/xen/mm.c
> > > > > @@ -138,8 +138,7 @@ void
> > > > xen_destroy_contiguous_region(phys_addr_t pstart, unsigned int
> > > > order)
> > > > > static int __init xen_mm_init(void)
> > > > > {
> > > > > struct gnttab_cache_flush cflush;
> > > > > - if (!xen_initial_domain())
> > > > > - return 0;
> > > > > + return 0;
> > > > > xen_swiotlb_init(1, false);
> > > > >
> > > > > cflush.op = 0;
> > > > >
> > > > > With this change the kernel is not booting up. (btw. I'm using
> > > > USB SSD for my OS.)
> > > > > [ 0.071081] bcm2835-dma fe007000.dma: Unable to set DMA mask
> > > > > [ 0.076277] bcm2835-dma fe007b00.dma: Unable to set DMA mask
> > > > > (XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=25: not implemented
> > > > > (XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=15: not implemented
> > > > > [ 0.592695] pci 0000:00:00.0: Failed to add - passthrough or
> > > > MSI/MSI-X might fail!
> > > > > (XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=15: not implemented
> > > > > [ 0.606819] pci 0000:01:00.0: Failed to add - passthrough or
> > > > MSI/MSI-X might fail!
> > > > > [ 1.212820] usb 1-1: device descriptor read/64, error 18
> > > > > [ 1.452815] usb 1-1: device descriptor read/64, error 18
> > > > > [ 1.820813] usb 1-1: device descriptor read/64, error 18
> > > > > [ 2.060815] usb 1-1: device descriptor read/64, error 18
> > > > > [ 2.845548] usb 1-1: device descriptor read/8, error -61
> > > > > [ 2.977603] usb 1-1: device descriptor read/8, error -61
> > > > > [ 3.237530] usb 1-1: device descriptor read/8, error -61
> > > > > [ 3.369585] usb 1-1: device descriptor read/8, error -61
> > > > > [ 3.480765] usb usb1-port1: unable to enumerate USB device
> > > > >
> > > > > Traces stop here. I could try with a memory card. Maybe it makes
> > > > a difference.
> > > >
> > > > This is very surprising. Disabling swiotlb-xen should make things
> > > > better
> > > > not worse. The only reason I can think of why it could make things
> > > > worse
> > > > is if Linux runs out of low memory. Julien's patch
> > > > 437b0aa06a014ce174e24c0d3530b3e9ab19b18b for Xen should have
> > > > addressed
> > > > that issue though. Julien, any ideas?
> > >
> > > I think, Stefano's small patch is not enough to disable the swiotlb as we
> > > will still override the DMA ops. You also likely want:
> > >
> > > diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
> > > index 8a8949174b1c..aa43e249ecdd 100644
> > > --- a/arch/arm/mm/dma-mapping.c
> > > +++ b/arch/arm/mm/dma-mapping.c
> > > @@ -2279,7 +2279,7 @@ void arch_setup_dma_ops(struct device *dev, u64
> > > dma_base, u64 size,
> > > set_dma_ops(dev, dma_ops);
> > >
> > > #ifdef CONFIG_XEN
> > > - if (xen_initial_domain())
> > > + if (0 || xen_initial_domain())
> > > dev->dma_ops = &xen_swiotlb_dma_ops;
> > > #endif
> > > dev->archdata.dma_ops_setup = true;
> > >
> > > Otherwise, you would still use the swiotlb DMA ops that would not be
> > > functional as we disabled the swiotlb.
> > >
> > > This would explain the following error because it will check whether the
> > > mask is valid using the callback dma_supported():
> > >
> > > [ 0.071081] bcm2835-dma fe007000.dma: Unable to set DMA mask
> > >
> > Good catch.
> > GPU works now and I can start X! Thanks! I was also able to create domU that
> > runs Raspian OS.
>
> Glad to hear it works! IIRC, the swiotlb may become necessary when running
> guest if the guest memory ends up to be used for DMA transaction.
It is necessary if you are using PV network or PV disk: memory shared by
another domU could end up being used in a DMA transaction. For that, you
need swiotlb-xen.
> > Now that swiotlb is disabled what does it mean?
>
> I can see two reasons:
> 1) You have limited memory below the 30 bits mark. So Swiotlb and CMA may
> try to fight for the low memory.
> 2) We found a few conversion bugs in the swiotlb on RPI4 last year (IIRC the
> DMA and physical address may be different). I looked at the Linux branch you
> are using and they seem to all be there. So there might be another bug.
>
> I am not sure how to figure out where is the problem. Stefano, do you have a
> suggestion where to start?
Both 1) and 2) are possible. It is also possible that another driver,
probably something related to CMA or DRM, has some special dma_ops
handling that doesn't work well together with swiotlb-xen.
Given that the original error seemed to be related to vc4_bo_create,
which calls dma_alloc_wc, I would add a couple of printks to
xen_swiotlb_alloc_coherent to help us figure it out:
diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c
index 2b385c1b4a99..cac8b09af603 100644
--- a/drivers/xen/swiotlb-xen.c
+++ b/drivers/xen/swiotlb-xen.c
@@ -295,6 +295,7 @@ xen_swiotlb_alloc_coherent(struct device *hwdev, size_t size,
/* Convert the size to actually allocated. */
size = 1UL << (order + XEN_PAGE_SHIFT);
+ printk("DEBUG %s %d size=%lu flags=%lx attr=%lx\n",__func__,__LINE__,size,flags,attrs);
/* On ARM this function returns an ioremap'ped virtual address for
* which virt_to_phys doesn't return the corresponding physical
* address. In fact on ARM virt_to_phys only works for kernel direct
@@ -315,16 +316,20 @@ xen_swiotlb_alloc_coherent(struct device *hwdev, size_t size,
phys = dma_to_phys(hwdev, *dma_handle);
dev_addr = xen_phys_to_dma(hwdev, phys);
if (((dev_addr + size - 1 <= dma_mask)) &&
- !range_straddles_page_boundary(phys, size))
+ !range_straddles_page_boundary(phys, size)) {
*dma_handle = dev_addr;
- else {
+ printk("DEBUG %s %d phys=%lx dma=%lx\n",__func__,__LINE__,phys,dev_addr);
+ } else {
if (xen_create_contiguous_region(phys, order,
fls64(dma_mask), dma_handle) != 0) {
+ printk("DEBUG %s %d\n",__func__,__LINE__);
xen_free_coherent_pages(hwdev, size, ret, (dma_addr_t)phys, attrs);
return NULL;
}
*dma_handle = phys_to_dma(hwdev, *dma_handle);
SetPageXenRemapped(virt_to_page(ret));
+ printk("DEBUG %s %d dma_mask=%d page_boundary=%d phys=%lx dma=%lx\n",__func__,__LINE__,
+ ((dev_addr + size - 1 <= dma_mask)),range_straddles_page_boundary(phys, size),phys,*dma_handle);
}
memset(ret, 0, size);
return ret;
> > And also can I pass the GPU to domU? Raspberry Pi 4 is limited HW and
> > doesn't have IOMMU. I'm trying to create similar OS like QubesOS where GPU,
> > Network, keyboard/mouse, ... are isolated to their own VMs.
>
> Without an IOMMU or any other HW mechamisns (e.g. MPU), it would not be safe
> to assign a DMA-capable device to a non-trusted VM.
>
> If you trust the VM where you assigned a device, then a possible approach
> would be to have the VM direct mapped (e.g. guest physical address == host
> physical address). Although, I can foreese some issues if you have multiple
> VMs requires memory below 30 bits (there seem to be limited amount)>
>
> If you don't trust the VM where you assigned a device, then your best option
> will be to expose a PV interface of the device and have your backend
> sanitizing the request and issuing it on behalf of the guest.
FYI you could do that with the existing PVFB drivers that only support 2D
graphics.
^ permalink raw reply related [flat|nested] 33+ messages in thread
* Re: Question about xen and Rasp 4B
2021-01-30 13:53 ` Jukka Kaartinen
2021-01-30 14:18 ` Julien Grall
@ 2021-02-02 2:22 ` Roman Shaposhnik
2021-02-02 8:12 ` Jukka Kaartinen
1 sibling, 1 reply; 33+ messages in thread
From: Roman Shaposhnik @ 2021-02-02 2:22 UTC (permalink / raw)
To: Jukka Kaartinen
Cc: Julien Grall, Stefano Stabellini, Xen-devel, Bertrand Marquis
On Sat, Jan 30, 2021 at 5:53 AM Jukka Kaartinen
<jukka.kaartinen@unikie.com> wrote:
> > On 27/01/2021 11:47, Jukka Kaartinen wrote:
> >>
> >>
> >> On Tue, Jan 26, 2021 at 10:22 PM Stefano Stabellini
> >> <sstabellini@kernel.org <mailto:sstabellini@kernel.org>> wrote:
> >>
> >> On Tue, 26 Jan 2021, Jukka Kaartinen wrote:
> >> > On Tue, Jan 26, 2021 at 2:54 AM Stefano Stabellini
> >> <sstabellini@kernel.org <mailto:sstabellini@kernel.org>> wrote:
> >> > On Sat, 23 Jan 2021, Jukka Kaartinen wrote:
> >> > > Thanks for the response!
> >> > >
> >> > > On Sat, Jan 23, 2021 at 2:27 AM Stefano Stabellini
> >> <sstabellini@kernel.org <mailto:sstabellini@kernel.org>> wrote:
> >> > > + xen-devel, Roman,
> >> > >
> >> > >
> >> > > On Fri, 22 Jan 2021, Jukka Kaartinen wrote:
> >> > > > Hi Stefano,
> >> > > > I'm Jukka Kaartinen a SW developer working on
> >> enabling hypervisors on mobile platforms. One of our HW that we
> >> use on
> >> > > development is
> >> > > > Raspberry Pi 4B. I wonder if you could help me a
> >> bit :).
> >> > > >
> >> > > > I'm trying to enable the GPU with Xen + Raspberry
> >> Pi for
> >> > > dom0.
> >>
> >> https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323#p1797605
> >>
> >> <https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323#p1797605>
> >> > > >
> >> > > > I got so far that GPU drivers are loaded (v3d &
> >> vc4) without errors. But now Xen returns error when X is starting:
> >> > > > (XEN) traps.c:1986:d0v1 HSR=0x93880045
> >> pc=0x00007f97b14e70 gva=0x7f7f817000 gpa=0x0000401315d000
> >> > > > I tried to debug what causes this and looks
> >> like find_mmio_handler cannot find handler.
> >> > > > (See more here:
> >>
> >> https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323&start=25#p1801691
> >>
> >>
> >> <https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323&start=25#p1801691>
> >>
> >> )
> >> > > >
> >> > > > Any ideas why the handler is not found?
> >> > >
> >> > >
> >> > > Hi Jukka,
> >> > >
> >> > > I am glad to hear that you are interested in Xen on
> >> RaspberryPi :-) I
> >> > > haven't tried the GPU yet, I have been using the
> >> serial only.
> >> > > Roman, did you ever get the GPU working?
> >> > >
> >> > >
> >> > > The error is a data abort error: Linux is trying to
> >> access an address
> >> > > which is not mapped to dom0. The address seems to
> >> be 0x401315d000. It is
> >> > > a pretty high address; I looked in device tree but
> >> couldn't spot it.
> >> > >
> >> > > >From the HSR (the syndrom register) it looks like
> >> it is a translation
> >> > > fault at EL1 on stage1. As if the Linux address
> >> mapping was wrong.
> >> > > Anyone has any ideas how this could happen? Maybe a
> >> reserved-memory
> >> > > misconfiguration?
> >> > >
> >> > > I had issues with loading the driver in the first place.
> >> Apparently swiotlb is used, maybe it can cause this. I also tried to
> >> > enable CMA.
> >> > > config.txt:
> >> > > dtoverlay=vc4-fkms-v3d,cma=320M@0x0-0x40000000
> >> > > gpu_mem=128
> >> >
> >> > Also looking at your other reply and the implementation of
> >> > vc4_bo_create, it looks like this is a CMA problem.
> >> >
> >> > It would be good to run a test with the swiotlb-xen
> >> disabled:
> >> >
> >> > diff --git a/arch/arm/xen/mm.c b/arch/arm/xen/mm.c
> >> > index 467fa225c3d0..2bdd12785d14 100644
> >> > --- a/arch/arm/xen/mm.c
> >> > +++ b/arch/arm/xen/mm.c
> >> > @@ -138,8 +138,7 @@ void
> >> xen_destroy_contiguous_region(phys_addr_t pstart, unsigned int order)
> >> > static int __init xen_mm_init(void)
> >> > {
> >> > struct gnttab_cache_flush cflush;
> >> > - if (!xen_initial_domain())
> >> > - return 0;
> >> > + return 0;
> >> > xen_swiotlb_init(1, false);
> >> >
> >> > cflush.op = 0;
> >> >
> >> > With this change the kernel is not booting up. (btw. I'm using
> >> USB SSD for my OS.)
> >> > [ 0.071081] bcm2835-dma fe007000.dma: Unable to set DMA mask
> >> > [ 0.076277] bcm2835-dma fe007b00.dma: Unable to set DMA mask
> >> > (XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=25: not implemented
> >> > (XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=15: not implemented
> >> > [ 0.592695] pci 0000:00:00.0: Failed to add - passthrough or
> >> MSI/MSI-X might fail!
> >> > (XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=15: not implemented
> >> > [ 0.606819] pci 0000:01:00.0: Failed to add - passthrough or
> >> MSI/MSI-X might fail!
> >> > [ 1.212820] usb 1-1: device descriptor read/64, error 18
> >> > [ 1.452815] usb 1-1: device descriptor read/64, error 18
> >> > [ 1.820813] usb 1-1: device descriptor read/64, error 18
> >> > [ 2.060815] usb 1-1: device descriptor read/64, error 18
> >> > [ 2.845548] usb 1-1: device descriptor read/8, error -61
> >> > [ 2.977603] usb 1-1: device descriptor read/8, error -61
> >> > [ 3.237530] usb 1-1: device descriptor read/8, error -61
> >> > [ 3.369585] usb 1-1: device descriptor read/8, error -61
> >> > [ 3.480765] usb usb1-port1: unable to enumerate USB device
> >> >
> >> > Traces stop here. I could try with a memory card. Maybe it makes
> >> a difference.
> >>
> >> This is very surprising. Disabling swiotlb-xen should make things
> >> better
> >> not worse. The only reason I can think of why it could make things
> >> worse
> >> is if Linux runs out of low memory. Julien's patch
> >> 437b0aa06a014ce174e24c0d3530b3e9ab19b18b for Xen should have
> >> addressed
> >> that issue though. Julien, any ideas?
> >
> > I think, Stefano's small patch is not enough to disable the swiotlb as
> > we will still override the DMA ops. You also likely want:
> >
> > diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
> > index 8a8949174b1c..aa43e249ecdd 100644
> > --- a/arch/arm/mm/dma-mapping.c
> > +++ b/arch/arm/mm/dma-mapping.c
> > @@ -2279,7 +2279,7 @@ void arch_setup_dma_ops(struct device *dev, u64
> > dma_base, u64 size,
> > set_dma_ops(dev, dma_ops);
> >
> > #ifdef CONFIG_XEN
> > - if (xen_initial_domain())
> > + if (0 || xen_initial_domain())
> > dev->dma_ops = &xen_swiotlb_dma_ops;
> > #endif
> > dev->archdata.dma_ops_setup = true;
> >
> > Otherwise, you would still use the swiotlb DMA ops that would not be
> > functional as we disabled the swiotlb.
> >
> > This would explain the following error because it will check whether the
> > mask is valid using the callback dma_supported():
> >
> > [ 0.071081] bcm2835-dma fe007000.dma: Unable to set DMA mask
> >
> Good catch.
> GPU works now and I can start X! Thanks! I was also able to create domU
> that runs Raspian OS.
This is very interesting that you were able to achieve that - congrats!
Now, sorry to be a bit dense -- but since this thread went into all
sorts of interesting
directions all at once -- I just have a very particular question: what is exact
combination of versions of Xen, Linux kernel and a set of patches that went
on top that allowed you to do that? I'd love to obviously see it
productized in Xen
upstream, but for now -- I'd love to make available to Project EVE/Xen community
since there seems to be a few folks interested in EVE/Xen combo being able to
do that.
Thanks,
Roman.
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Question about xen and Rasp 4B
2021-02-01 19:25 ` Stefano Stabellini
@ 2021-02-02 7:47 ` Jukka Kaartinen
2021-02-04 6:58 ` Jukka Kaartinen
1 sibling, 0 replies; 33+ messages in thread
From: Jukka Kaartinen @ 2021-02-02 7:47 UTC (permalink / raw)
To: Stefano Stabellini, Julien Grall
Cc: Xen-devel, Roman Shaposhnik, Bertrand Marquis
>>
>> Glad to hear it works! IIRC, the swiotlb may become necessary when running
>> guest if the guest memory ends up to be used for DMA transaction.
>
> It is necessary if you are using PV network or PV disk: memory shared by
> another domU could end up being used in a DMA transaction. For that, you
> need swiotlb-xen.
Sounds bad :).
>>> Now that swiotlb is disabled what does it mean?
>>
>> I can see two reasons:
>> 1) You have limited memory below the 30 bits mark. So Swiotlb and CMA may
>> try to fight for the low memory.
>> 2) We found a few conversion bugs in the swiotlb on RPI4 last year (IIRC the
>> DMA and physical address may be different). I looked at the Linux branch you
>> are using and they seem to all be there. So there might be another bug.
>>
>> I am not sure how to figure out where is the problem. Stefano, do you have a
>> suggestion where to start?
>
> Both 1) and 2) are possible. It is also possible that another driver,
> probably something related to CMA or DRM, has some special dma_ops
> handling that doesn't work well together with swiotlb-xen.
>
> Given that the original error seemed to be related to vc4_bo_create,
> which calls dma_alloc_wc, I would add a couple of printks to
> xen_swiotlb_alloc_coherent to help us figure it out:
>
>
> diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c
> index 2b385c1b4a99..cac8b09af603 100644
> --- a/drivers/xen/swiotlb-xen.c
> +++ b/drivers/xen/swiotlb-xen.c
> @@ -295,6 +295,7 @@ xen_swiotlb_alloc_coherent(struct device *hwdev, size_t size,
> /* Convert the size to actually allocated. */
> size = 1UL << (order + XEN_PAGE_SHIFT);
>
> + printk("DEBUG %s %d size=%lu flags=%lx attr=%lx\n",__func__,__LINE__,size,flags,attrs);
> /* On ARM this function returns an ioremap'ped virtual address for
> * which virt_to_phys doesn't return the corresponding physical
> * address. In fact on ARM virt_to_phys only works for kernel direct
> @@ -315,16 +316,20 @@ xen_swiotlb_alloc_coherent(struct device *hwdev, size_t size,
> phys = dma_to_phys(hwdev, *dma_handle);
> dev_addr = xen_phys_to_dma(hwdev, phys);
> if (((dev_addr + size - 1 <= dma_mask)) &&
> - !range_straddles_page_boundary(phys, size))
> + !range_straddles_page_boundary(phys, size)) {
> *dma_handle = dev_addr;
> - else {
> + printk("DEBUG %s %d phys=%lx dma=%lx\n",__func__,__LINE__,phys,dev_addr);
> + } else {
> if (xen_create_contiguous_region(phys, order,
> fls64(dma_mask), dma_handle) != 0) {
> + printk("DEBUG %s %d\n",__func__,__LINE__);
> xen_free_coherent_pages(hwdev, size, ret, (dma_addr_t)phys, attrs);
> return NULL;
> }
> *dma_handle = phys_to_dma(hwdev, *dma_handle);
> SetPageXenRemapped(virt_to_page(ret));
> + printk("DEBUG %s %d dma_mask=%d page_boundary=%d phys=%lx dma=%lx\n",__func__,__LINE__,
> + ((dev_addr + size - 1 <= dma_mask)),range_straddles_page_boundary(phys, size),phys,*dma_handle);
> }
> memset(ret, 0, size);
> return ret;
>
Thanks I will try this.
>
>
>
>>> And also can I pass the GPU to domU? Raspberry Pi 4 is limited HW and
>>> doesn't have IOMMU. I'm trying to create similar OS like QubesOS where GPU,
>>> Network, keyboard/mouse, ... are isolated to their own VMs.
>>
>> Without an IOMMU or any other HW mechamisns (e.g. MPU), it would not be safe
>> to assign a DMA-capable device to a non-trusted VM.
>>
>> If you trust the VM where you assigned a device, then a possible approach
>> would be to have the VM direct mapped (e.g. guest physical address == host
>> physical address). Although, I can foreese some issues if you have multiple
>> VMs requires memory below 30 bits (there seem to be limited amount)>
>>
>> If you don't trust the VM where you assigned a device, then your best option
>> will be to expose a PV interface of the device and have your backend
>> sanitizing the request and issuing it on behalf of the guest.
>
> FYI you could do that with the existing PVFB drivers that only support 2D
> graphics
I'll keep that in my mind, thanks.
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Question about xen and Rasp 4B
2021-02-02 2:22 ` Roman Shaposhnik
@ 2021-02-02 8:12 ` Jukka Kaartinen
2021-02-03 0:18 ` Stefano Stabellini
0 siblings, 1 reply; 33+ messages in thread
From: Jukka Kaartinen @ 2021-02-02 8:12 UTC (permalink / raw)
To: Roman Shaposhnik
Cc: Julien Grall, Stefano Stabellini, Xen-devel, Bertrand Marquis
Hi Roman,
>>>
>> Good catch.
>> GPU works now and I can start X! Thanks! I was also able to create domU
>> that runs Raspian OS.
>
> This is very interesting that you were able to achieve that - congrats!
>
> Now, sorry to be a bit dense -- but since this thread went into all
> sorts of interesting
> directions all at once -- I just have a very particular question: what is exact
> combination of versions of Xen, Linux kernel and a set of patches that went
> on top that allowed you to do that? I'd love to obviously see it
> productized in Xen
> upstream, but for now -- I'd love to make available to Project EVE/Xen community
> since there seems to be a few folks interested in EVE/Xen combo being able to
> do that.
I have tried Xen Release 4.14.0, 4.14.1 and master (from week 4, 2021).
Kernel rpi-5.9.y and rpi-5.10.y branches from
https://github.com/raspberrypi/linux
and
U-boot (master).
For the GPU to work it was enough to disable swiotlb from the kernel(s)
as suggested in this thread.
If you use Xen master then you need to revert the
25849c8b16f2a5b7fcd0a823e80a5f1b590291f9. Apparently v3d uses same
resources and will not start.
I was able to get most of the combinations to work without any big efforts.
In case you use USB SSD U-boot needs a fix if you use 5.9 kernel.
The 5.10 works with the lates Xen master but then you need one small
workaround to the xen since there is address that Xen cannot map. Some
usb address cannot be found (address was 0 if recall correctly). I just
by passed the error:
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index e824ba34b0..3e8a175f2e 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -1409,7 +1409,7 @@ static int __init handle_device(struct domain *d,
struct dt_device_node *dev,
{
printk(XENLOG_ERR "Unable to retrieve address %u for %s\n",
i, dt_node_full_name(dev));
- return res;
+ continue; //return res;
}
res = map_range_to_domain(dev, addr, size, &mr_data);
^ permalink raw reply related [flat|nested] 33+ messages in thread
* Re: Question about xen and Rasp 4B
2021-02-02 8:12 ` Jukka Kaartinen
@ 2021-02-03 0:18 ` Stefano Stabellini
2021-02-03 1:58 ` Elliott Mitchell
2021-02-03 7:36 ` Jukka Kaartinen
0 siblings, 2 replies; 33+ messages in thread
From: Stefano Stabellini @ 2021-02-03 0:18 UTC (permalink / raw)
To: Jukka Kaartinen
Cc: Roman Shaposhnik, Julien Grall, Stefano Stabellini, Xen-devel,
Bertrand Marquis
On Tue, 2 Feb 2021, Jukka Kaartinen wrote:
> Hi Roman,
>
> > > >
> > > Good catch.
> > > GPU works now and I can start X! Thanks! I was also able to create domU
> > > that runs Raspian OS.
> >
> > This is very interesting that you were able to achieve that - congrats!
> >
> > Now, sorry to be a bit dense -- but since this thread went into all
> > sorts of interesting
> > directions all at once -- I just have a very particular question: what is
> > exact
> > combination of versions of Xen, Linux kernel and a set of patches that went
> > on top that allowed you to do that? I'd love to obviously see it
> > productized in Xen
> > upstream, but for now -- I'd love to make available to Project EVE/Xen
> > community
> > since there seems to be a few folks interested in EVE/Xen combo being able
> > to
> > do that.
>
> I have tried Xen Release 4.14.0, 4.14.1 and master (from week 4, 2021).
>
> Kernel rpi-5.9.y and rpi-5.10.y branches from
> https://github.com/raspberrypi/linux
>
> and
>
> U-boot (master).
>
> For the GPU to work it was enough to disable swiotlb from the kernel(s) as
> suggested in this thread.
How are you configuring and installing the kernel?
make bcm2711_defconfig
make Image.gz
make modules_install
?
The device tree is the one from the rpi-5.9.y build? How are you loading
the kernel and device tree with uboot? Do you have any interesting
changes to config.txt?
I am asking because I cannot get to the point of reproducing what you
are seeing: I can boot my rpi-5.9.y kernel on recent Xen but I cannot
get any graphics output on my screen. (The serial works.) I am using the
default Ubuntu Desktop rpi-install target as rootfs and uboot master.
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Question about xen and Rasp 4B
2021-02-03 0:18 ` Stefano Stabellini
@ 2021-02-03 1:58 ` Elliott Mitchell
2021-02-03 7:36 ` Jukka Kaartinen
1 sibling, 0 replies; 33+ messages in thread
From: Elliott Mitchell @ 2021-02-03 1:58 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Jukka Kaartinen, Roman Shaposhnik, Julien Grall, Xen-devel,
Bertrand Marquis
On Tue, Feb 02, 2021 at 04:18:44PM -0800, Stefano Stabellini wrote:
> How are you configuring and installing the kernel?
>
> make bcm2711_defconfig
> make Image.gz
> make modules_install
>
> ?
>
> The device tree is the one from the rpi-5.9.y build? How are you loading
> the kernel and device tree with uboot? Do you have any interesting
> changes to config.txt?
>
> I am asking because I cannot get to the point of reproducing what you
> are seeing: I can boot my rpi-5.9.y kernel on recent Xen but I cannot
> get any graphics output on my screen. (The serial works.) I am using the
> default Ubuntu Desktop rpi-install target as rootfs and uboot master.
I've been trying with various pieces from various sources trying to get
things to work. Since my goal has been a Debian-variant I use
Debian-packaged versions of things if at all possible. Sticking to
packaged versions is more maintainable over the longer run.
My starting point was SuSE Raspberry PI 4B installation medium. I'm
still using pieces from SuSE's installation. Notably SuSE's overlays
have worked rather better than RPF or kernel versions of device-tree
overlays.
Debian's u-boot-rpi:arm64 package is functional. As such that provides
u-boot.bin which is loaded via config.txt as the kernel.
Debian's grub-efi-arm64 package is also functional. Installing that is
a bit funky as U-Boot's EFI environment is incomplete. Nonetheless it
is simply an issue of having that installed in EFI/BOOT as the primary
boot entry, rather than EFI/Debian where it would normally install.
The base device-tree files from the RPF kernel function reasonably
well (unlike the overlays). I'm actually doing
`make O=<build-dir> bcm2711_defconfig menuconfig bindeb-pkg` and then
installing the resultant package. This places bcm2711-rpi-4-b.dtb in
/usr/lib/linux-image-<rev>/broadcom/bcm2711-rpi-4-b.dtb, I'm presently
copying that into the Raspberry PI boot area.
If you're unable to get graphics output, note the instruction that HDMI
MUST be plugged in *during* *boot*. Broadcom's chips have the graphics
core is control of rather more than one might expect (Qualcomm follows
this pattern by wanting their modems in control). In fact I've observed
I need my monitor displaying the input from the RP4 in order for it to
complete the handshake and have the RP4 do graphics.
--
(\___(\___(\______ --=> 8-) EHM <=-- ______/)___/)___/)
\BS ( | ehem+sigmsg@m5p.com PGP 87145445 | ) /
\_CS\ | _____ -O #include <stddisclaimer.h> O- _____ | / _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Question about xen and Rasp 4B
2021-02-03 0:18 ` Stefano Stabellini
2021-02-03 1:58 ` Elliott Mitchell
@ 2021-02-03 7:36 ` Jukka Kaartinen
2021-02-03 20:55 ` Stefano Stabellini
1 sibling, 1 reply; 33+ messages in thread
From: Jukka Kaartinen @ 2021-02-03 7:36 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Roman Shaposhnik, Julien Grall, Xen-devel, Bertrand Marquis
On 3.2.2021 2.18, Stefano Stabellini wrote:
> On Tue, 2 Feb 2021, Jukka Kaartinen wrote:
>> Hi Roman,
>>
>>>>>
>>>> Good catch.
>>>> GPU works now and I can start X! Thanks! I was also able to create domU
>>>> that runs Raspian OS.
>>>
>>> This is very interesting that you were able to achieve that - congrats!
>>>
>>> Now, sorry to be a bit dense -- but since this thread went into all
>>> sorts of interesting
>>> directions all at once -- I just have a very particular question: what is
>>> exact
>>> combination of versions of Xen, Linux kernel and a set of patches that went
>>> on top that allowed you to do that? I'd love to obviously see it
>>> productized in Xen
>>> upstream, but for now -- I'd love to make available to Project EVE/Xen
>>> community
>>> since there seems to be a few folks interested in EVE/Xen combo being able
>>> to
>>> do that.
>>
>> I have tried Xen Release 4.14.0, 4.14.1 and master (from week 4, 2021).
>>
>> Kernel rpi-5.9.y and rpi-5.10.y branches from
>> https://github.com/raspberrypi/linux
>>
>> and
>>
>> U-boot (master).
>>
>> For the GPU to work it was enough to disable swiotlb from the kernel(s) as
>> suggested in this thread.
>
> How are you configuring and installing the kernel?
>
> make bcm2711_defconfig
> make Image.gz
> make modules_install
>
> ?
>
> The device tree is the one from the rpi-5.9.y build? How are you loading
> the kernel and device tree with uboot? Do you have any interesting
> changes to config.txt?
>
> I am asking because I cannot get to the point of reproducing what you
> are seeing: I can boot my rpi-5.9.y kernel on recent Xen but I cannot
> get any graphics output on my screen. (The serial works.) I am using the
> default Ubuntu Desktop rpi-install target as rootfs and uboot master.
>
This is what I do:
make bcm2711_defconfig
cat "xen_additions" >> .config
make Image modules dtbs
make INSTALL_MOD_PATH=rootfs modules_install
depmod -a
cp arch/arm64/boot/dts/broadcom/bcm2711-rpi-4-b.dtb boot/
cp arch/arm64/boot/dts/overlays/*.dtbo boot/overlays/
config.txt:
[pi4]
max_framebuffers=2
enable_uart=1
arm_freq=1500
force_turbo=1
[all]
arm_64bit=1
kernel=u-boot.bin
start_file=start4.elf
fixup_file=fixup4.dat
# Enable the audio output, I2C and SPI interfaces on the GPIO header
dtparam=audio=on
dtparam=i2c_arm=on
dtparam=spi=on
# Enable the FKMS ("Fake" KMS) graphics overlay, enable the camera firmware
# and allocate 128Mb to the GPU memory
dtoverlay=vc4-fkms-v3d,cma-64
gpu_mem=128
# Comment out the following line if the edges of the desktop appear outside
# the edges of your display
disable_overscan=1
boot.source:
setenv serverip 10.42.0.1
setenv ipaddr 10.42.0.231
tftpb 0xC00000 boot2.scr
source 0xC00000
boot2.source:
tftpb 0xE00000 xen
tftpb 0x1000000 Image
setenv lin_size $filesize
fdt addr ${fdt_addr}
fdt resize 1024
fdt set /chosen xen,xen-bootargs "console=dtuart dtuart=serial0
sync_console dom0_mem=1024M dom0_max_vcpus=1 bootscrub=0 vwfi=native
sched=credit2"
fdt mknod /chosen dom0
# These will break the default framebuffer@3e2fe000 that
# is the same chosen -node.
#fdt set /chosen/dom0 \#address-cells <0x2>
#fdt set /chosen/dom0 \#size-cells <0x2>
fdt set /chosen/dom0 compatible "xen,linux-zimage" "xen,multiboot-module"
fdt set /chosen/dom0 reg <0x1000000 0x${lin_size}>
fdt set /chosen xen,dom0-bootargs "dwc_otg.lpm_enable=0 console=hvc0
earlycon=xen earlyprintk=xen root=/dev/sda4 elevator=deadline rootwait
fixrtc quiet splash"
setenv fdt_high 0xffffffffffffffff
fdt print /chosen
#xen
booti 0xE00000 - ${fdt_addr}
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Question about xen and Rasp 4B
2021-02-03 7:36 ` Jukka Kaartinen
@ 2021-02-03 20:55 ` Stefano Stabellini
2021-02-03 22:12 ` Elliott Mitchell
2021-02-04 6:44 ` Jukka Kaartinen
0 siblings, 2 replies; 33+ messages in thread
From: Stefano Stabellini @ 2021-02-03 20:55 UTC (permalink / raw)
To: Jukka Kaartinen
Cc: Stefano Stabellini, Roman Shaposhnik, Julien Grall, Xen-devel,
Bertrand Marquis
On Wed, 3 Feb 2021, Jukka Kaartinen wrote:
> On 3.2.2021 2.18, Stefano Stabellini wrote:
> > On Tue, 2 Feb 2021, Jukka Kaartinen wrote:
> > > > > Good catch.
> > > > > GPU works now and I can start X! Thanks! I was also able to create
> > > > > domU
> > > > > that runs Raspian OS.
> > > >
> > > > This is very interesting that you were able to achieve that - congrats!
> > > >
> > > > Now, sorry to be a bit dense -- but since this thread went into all
> > > > sorts of interesting
> > > > directions all at once -- I just have a very particular question: what
> > > > is
> > > > exact
> > > > combination of versions of Xen, Linux kernel and a set of patches that
> > > > went
> > > > on top that allowed you to do that? I'd love to obviously see it
> > > > productized in Xen
> > > > upstream, but for now -- I'd love to make available to Project EVE/Xen
> > > > community
> > > > since there seems to be a few folks interested in EVE/Xen combo being
> > > > able
> > > > to
> > > > do that.
> > >
> > > I have tried Xen Release 4.14.0, 4.14.1 and master (from week 4, 2021).
> > >
> > > Kernel rpi-5.9.y and rpi-5.10.y branches from
> > > https://github.com/raspberrypi/linux
> > >
> > > and
> > >
> > > U-boot (master).
> > >
> > > For the GPU to work it was enough to disable swiotlb from the kernel(s) as
> > > suggested in this thread.
> >
> > How are you configuring and installing the kernel?
> >
> > make bcm2711_defconfig
> > make Image.gz
> > make modules_install
> >
> > ?
> >
> > The device tree is the one from the rpi-5.9.y build? How are you loading
> > the kernel and device tree with uboot? Do you have any interesting
> > changes to config.txt?
> >
> > I am asking because I cannot get to the point of reproducing what you
> > are seeing: I can boot my rpi-5.9.y kernel on recent Xen but I cannot
> > get any graphics output on my screen. (The serial works.) I am using the
> > default Ubuntu Desktop rpi-install target as rootfs and uboot master.
> >
>
> This is what I do:
>
> make bcm2711_defconfig
> cat "xen_additions" >> .config
> make Image modules dtbs
>
> make INSTALL_MOD_PATH=rootfs modules_install
> depmod -a
>
> cp arch/arm64/boot/dts/broadcom/bcm2711-rpi-4-b.dtb boot/
> cp arch/arm64/boot/dts/overlays/*.dtbo boot/overlays/
Thanks for the detailed instructions. This helps a lot. I saw below in
boot2.source that you are using ${fdt_addr} as DTB source (instead of
loading one), which means you are using the DTB as provided by U-Boot at
runtime, instead of loading your own file.
With these two copies, I take you meant to update the first partition on
the SD card, the one where config.txt lives, right? So that Xen is
getting the DTB and overlays from the rpi-5.9.y kernel tree but passed
down by the RPi loader and U-Boot?
I think the DTB must be the issue as I wasn't applying any overlays
before. I ran a test to use the DTB and overlay from U-Boot but maybe I
haven't updated them properly because I still don't see any output.
> config.txt:
>
> [pi4]
> max_framebuffers=2
> enable_uart=1
> arm_freq=1500
> force_turbo=1
>
> [all]
> arm_64bit=1
> kernel=u-boot.bin
>
> start_file=start4.elf
> fixup_file=fixup4.dat
>
> # Enable the audio output, I2C and SPI interfaces on the GPIO header
> dtparam=audio=on
> dtparam=i2c_arm=on
> dtparam=spi=on
>
> # Enable the FKMS ("Fake" KMS) graphics overlay, enable the camera firmware
> # and allocate 128Mb to the GPU memory
> dtoverlay=vc4-fkms-v3d,cma-64
> gpu_mem=128
>
> # Comment out the following line if the edges of the desktop appear outside
> # the edges of your display
> disable_overscan=1
>
>
> boot.source:
> setenv serverip 10.42.0.1
> setenv ipaddr 10.42.0.231
> tftpb 0xC00000 boot2.scr
> source 0xC00000
>
> boot2.source:
> tftpb 0xE00000 xen
> tftpb 0x1000000 Image
> setenv lin_size $filesize
>
> fdt addr ${fdt_addr}
> fdt resize 1024
>
> fdt set /chosen xen,xen-bootargs "console=dtuart dtuart=serial0 sync_console
> dom0_mem=1024M dom0_max_vcpus=1 bootscrub=0 vwfi=native sched=credit2"
>
> fdt mknod /chosen dom0
>
> # These will break the default framebuffer@3e2fe000 that
> # is the same chosen -node.
> #fdt set /chosen/dom0 \#address-cells <0x2>
> #fdt set /chosen/dom0 \#size-cells <0x2>
>
> fdt set /chosen/dom0 compatible "xen,linux-zimage" "xen,multiboot-module"
> fdt set /chosen/dom0 reg <0x1000000 0x${lin_size}>
>
> fdt set /chosen xen,dom0-bootargs "dwc_otg.lpm_enable=0 console=hvc0
> earlycon=xen earlyprintk=xen root=/dev/sda4 elevator=deadline rootwait fixrtc
> quiet splash"
>
> setenv fdt_high 0xffffffffffffffff
>
> fdt print /chosen
>
> #xen
> booti 0xE00000 - ${fdt_addr}
>
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Question about xen and Rasp 4B
2021-02-03 20:55 ` Stefano Stabellini
@ 2021-02-03 22:12 ` Elliott Mitchell
2021-02-04 6:46 ` Jukka Kaartinen
2021-02-04 6:44 ` Jukka Kaartinen
1 sibling, 1 reply; 33+ messages in thread
From: Elliott Mitchell @ 2021-02-03 22:12 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Jukka Kaartinen, Roman Shaposhnik, Julien Grall, Xen-devel,
Bertrand Marquis
On Wed, Feb 03, 2021 at 12:55:40PM -0800, Stefano Stabellini wrote:
> On Wed, 3 Feb 2021, Jukka Kaartinen wrote:
> > On 3.2.2021 2.18, Stefano Stabellini wrote:
> > > How are you configuring and installing the kernel?
> > >
> > > make bcm2711_defconfig
> > > make Image.gz
> > > make modules_install
> > >
> > > ?
> > >
> > > The device tree is the one from the rpi-5.9.y build? How are you loading
> > > the kernel and device tree with uboot? Do you have any interesting
> > > changes to config.txt?
> > >
> > > I am asking because I cannot get to the point of reproducing what you
> > > are seeing: I can boot my rpi-5.9.y kernel on recent Xen but I cannot
> > > get any graphics output on my screen. (The serial works.) I am using the
> > > default Ubuntu Desktop rpi-install target as rootfs and uboot master.
> > >
> >
> > This is what I do:
> >
> > make bcm2711_defconfig
> > cat "xen_additions" >> .config
> > make Image modules dtbs
> >
> > make INSTALL_MOD_PATH=rootfs modules_install
> > depmod -a
> >
> > cp arch/arm64/boot/dts/broadcom/bcm2711-rpi-4-b.dtb boot/
> > cp arch/arm64/boot/dts/overlays/*.dtbo boot/overlays/
>
> Thanks for the detailed instructions. This helps a lot. I saw below in
> boot2.source that you are using ${fdt_addr} as DTB source (instead of
> loading one), which means you are using the DTB as provided by U-Boot at
> runtime, instead of loading your own file.
>
> With these two copies, I take you meant to update the first partition on
> the SD card, the one where config.txt lives, right? So that Xen is
> getting the DTB and overlays from the rpi-5.9.y kernel tree but passed
> down by the RPi loader and U-Boot?
>
> I think the DTB must be the issue as I wasn't applying any overlays
> before. I ran a test to use the DTB and overlay from U-Boot but maybe I
> haven't updated them properly because I still don't see any output.
Seeing no graphics output from U-Boot is okay. If the device-tree files
get sufficiently updated you can end up with no output from U-Boot, but
will get output once the Linux kernel's driver is operational (I've seen
this occur).
The most important part is having a HDMI display plugged in during the
early boot stages. Unless the bootloader sees the display the output
won't get initialized and the Linux driver doesn't handle that.
> > dtoverlay=vc4-fkms-v3d,cma-64
This is odd. My understanding is this is appropriate for RP3, but not
RP4. For RP4 you can have "dtoverlay=disable-vc4" and still get graphics
output (hmm, I'm starting to think I need to double-check this...).
--
(\___(\___(\______ --=> 8-) EHM <=-- ______/)___/)___/)
\BS ( | ehem+sigmsg@m5p.com PGP 87145445 | ) /
\_CS\ | _____ -O #include <stddisclaimer.h> O- _____ | / _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Question about xen and Rasp 4B
2021-02-03 20:55 ` Stefano Stabellini
2021-02-03 22:12 ` Elliott Mitchell
@ 2021-02-04 6:44 ` Jukka Kaartinen
1 sibling, 0 replies; 33+ messages in thread
From: Jukka Kaartinen @ 2021-02-04 6:44 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Roman Shaposhnik, Julien Grall, Xen-devel, Bertrand Marquis
On 3.2.2021 22.55, Stefano Stabellini wrote:
> On Wed, 3 Feb 2021, Jukka Kaartinen wrote:
>> On 3.2.2021 2.18, Stefano Stabellini wrote:
>>> On Tue, 2 Feb 2021, Jukka Kaartinen wrote:
>>>>>> Good catch.
>>>>>> GPU works now and I can start X! Thanks! I was also able to create
>>>>>> domU
>>>>>> that runs Raspian OS.
>>>>>
>>>>> This is very interesting that you were able to achieve that - congrats!
>>>>>
>>>>> Now, sorry to be a bit dense -- but since this thread went into all
>>>>> sorts of interesting
>>>>> directions all at once -- I just have a very particular question: what
>>>>> is
>>>>> exact
>>>>> combination of versions of Xen, Linux kernel and a set of patches that
>>>>> went
>>>>> on top that allowed you to do that? I'd love to obviously see it
>>>>> productized in Xen
>>>>> upstream, but for now -- I'd love to make available to Project EVE/Xen
>>>>> community
>>>>> since there seems to be a few folks interested in EVE/Xen combo being
>>>>> able
>>>>> to
>>>>> do that.
>>>>
>>>> I have tried Xen Release 4.14.0, 4.14.1 and master (from week 4, 2021).
>>>>
>>>> Kernel rpi-5.9.y and rpi-5.10.y branches from
>>>> https://github.com/raspberrypi/linux
>>>>
>>>> and
>>>>
>>>> U-boot (master).
>>>>
>>>> For the GPU to work it was enough to disable swiotlb from the kernel(s) as
>>>> suggested in this thread.
>>>
>>> How are you configuring and installing the kernel?
>>>
>>> make bcm2711_defconfig
>>> make Image.gz
>>> make modules_install
>>>
>>> ?
>>>
>>> The device tree is the one from the rpi-5.9.y build? How are you loading
>>> the kernel and device tree with uboot? Do you have any interesting
>>> changes to config.txt?
>>>
>>> I am asking because I cannot get to the point of reproducing what you
>>> are seeing: I can boot my rpi-5.9.y kernel on recent Xen but I cannot
>>> get any graphics output on my screen. (The serial works.) I am using the
>>> default Ubuntu Desktop rpi-install target as rootfs and uboot master.
>>>
>>
>> This is what I do:
>>
>> make bcm2711_defconfig
>> cat "xen_additions" >> .config
>> make Image modules dtbs
>>
>> make INSTALL_MOD_PATH=rootfs modules_install
>> depmod -a
>>
>> cp arch/arm64/boot/dts/broadcom/bcm2711-rpi-4-b.dtb boot/
>> cp arch/arm64/boot/dts/overlays/*.dtbo boot/overlays/
>
> Thanks for the detailed instructions. This helps a lot. I saw below in
> boot2.source that you are using ${fdt_addr} as DTB source (instead of
> loading one), which means you are using the DTB as provided by U-Boot at
> runtime, instead of loading your own file.
>
> With these two copies, I take you meant to update the first partition on
> the SD card, the one where config.txt lives, right? So that Xen is
> getting the DTB and overlays from the rpi-5.9.y kernel tree but passed
> down by the RPi loader and U-Boot?
>
> I think the DTB must be the issue as I wasn't applying any overlays
> before. I ran a test to use the DTB and overlay from U-Boot but maybe I
> haven't updated them properly because I still don't see any output.
I'm using ${fdt_addr} has the changes that FW will do according to your
congig.txt for example the overlay is applied. I tried to load and apply
the overlay in the u-boot but there were strange errors so I decided to
go the easy way.
I the overlay (vc4-fkms-v3d) is not applied GPU (v3d) driver is not probed.
I always see output from u-boot from the serial port. If not then there
has been something wrong with the device-tree. And it has been because
u-boot was not able to read from the SSD.
>
>
>> config.txt:
>>
>> [pi4]
>> max_framebuffers=2
>> enable_uart=1
>> arm_freq=1500
>> force_turbo=1
>>
>> [all]
>> arm_64bit=1
>> kernel=u-boot.bin
>>
>> start_file=start4.elf
>> fixup_file=fixup4.dat
>>
>> # Enable the audio output, I2C and SPI interfaces on the GPIO header
>> dtparam=audio=on
>> dtparam=i2c_arm=on
>> dtparam=spi=on
>>
>> # Enable the FKMS ("Fake" KMS) graphics overlay, enable the camera firmware
>> # and allocate 128Mb to the GPU memory
>> dtoverlay=vc4-fkms-v3d,cma-64
>> gpu_mem=128
>>
>> # Comment out the following line if the edges of the desktop appear outside
>> # the edges of your display
>> disable_overscan=1
>>
>>
>> boot.source:
>> setenv serverip 10.42.0.1
>> setenv ipaddr 10.42.0.231
>> tftpb 0xC00000 boot2.scr
>> source 0xC00000
>>
>> boot2.source:
>> tftpb 0xE00000 xen
>> tftpb 0x1000000 Image
>> setenv lin_size $filesize
>>
>> fdt addr ${fdt_addr}
>> fdt resize 1024
>>
>> fdt set /chosen xen,xen-bootargs "console=dtuart dtuart=serial0 sync_console
>> dom0_mem=1024M dom0_max_vcpus=1 bootscrub=0 vwfi=native sched=credit2"
>>
>> fdt mknod /chosen dom0
>>
>> # These will break the default framebuffer@3e2fe000 that
>> # is the same chosen -node.
>> #fdt set /chosen/dom0 \#address-cells <0x2>
>> #fdt set /chosen/dom0 \#size-cells <0x2>
>>
>> fdt set /chosen/dom0 compatible "xen,linux-zimage" "xen,multiboot-module"
>> fdt set /chosen/dom0 reg <0x1000000 0x${lin_size}>
>>
>> fdt set /chosen xen,dom0-bootargs "dwc_otg.lpm_enable=0 console=hvc0
>> earlycon=xen earlyprintk=xen root=/dev/sda4 elevator=deadline rootwait fixrtc
>> quiet splash"
>>
>> setenv fdt_high 0xffffffffffffffff
>>
>> fdt print /chosen
>>
>> #xen
>> booti 0xE00000 - ${fdt_addr}
>>
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Question about xen and Rasp 4B
2021-02-03 22:12 ` Elliott Mitchell
@ 2021-02-04 6:46 ` Jukka Kaartinen
0 siblings, 0 replies; 33+ messages in thread
From: Jukka Kaartinen @ 2021-02-04 6:46 UTC (permalink / raw)
To: Elliott Mitchell, Stefano Stabellini
Cc: Roman Shaposhnik, Julien Grall, Xen-devel, Bertrand Marquis
On 4.2.2021 0.12, Elliott Mitchell wrote:
> On Wed, Feb 03, 2021 at 12:55:40PM -0800, Stefano Stabellini wrote:
>> On Wed, 3 Feb 2021, Jukka Kaartinen wrote:
>>> On 3.2.2021 2.18, Stefano Stabellini wrote:
>>>> How are you configuring and installing the kernel?
>>>>
>>>> make bcm2711_defconfig
>>>> make Image.gz
>>>> make modules_install
>>>>
>>>> ?
>>>>
>>>> The device tree is the one from the rpi-5.9.y build? How are you loading
>>>> the kernel and device tree with uboot? Do you have any interesting
>>>> changes to config.txt?
>>>>
>>>> I am asking because I cannot get to the point of reproducing what you
>>>> are seeing: I can boot my rpi-5.9.y kernel on recent Xen but I cannot
>>>> get any graphics output on my screen. (The serial works.) I am using the
>>>> default Ubuntu Desktop rpi-install target as rootfs and uboot master.
>>>>
>>>
>>> This is what I do:
>>>
>>> make bcm2711_defconfig
>>> cat "xen_additions" >> .config
>>> make Image modules dtbs
>>>
>>> make INSTALL_MOD_PATH=rootfs modules_install
>>> depmod -a
>>>
>>> cp arch/arm64/boot/dts/broadcom/bcm2711-rpi-4-b.dtb boot/
>>> cp arch/arm64/boot/dts/overlays/*.dtbo boot/overlays/
>>
>> Thanks for the detailed instructions. This helps a lot. I saw below in
>> boot2.source that you are using ${fdt_addr} as DTB source (instead of
>> loading one), which means you are using the DTB as provided by U-Boot at
>> runtime, instead of loading your own file.
>>
>> With these two copies, I take you meant to update the first partition on
>> the SD card, the one where config.txt lives, right? So that Xen is
>> getting the DTB and overlays from the rpi-5.9.y kernel tree but passed
>> down by the RPi loader and U-Boot?
>>
>> I think the DTB must be the issue as I wasn't applying any overlays
>> before. I ran a test to use the DTB and overlay from U-Boot but maybe I
>> haven't updated them properly because I still don't see any output.
>
> Seeing no graphics output from U-Boot is okay. If the device-tree files
> get sufficiently updated you can end up with no output from U-Boot, but
> will get output once the Linux kernel's driver is operational (I've seen
> this occur).
>
> The most important part is having a HDMI display plugged in during the
> early boot stages. Unless the bootloader sees the display the output
> won't get initialized and the Linux driver doesn't handle that.
>
>
>>> dtoverlay=vc4-fkms-v3d,cma-64
>
> This is odd. My understanding is this is appropriate for RP3, but not
> RP4. For RP4 you can have "dtoverlay=disable-vc4" and still get graphics
> output (hmm, I'm starting to think I need to double-check this...).
Without the overlay GPU driver (v3d) was not probed. And you need to use
the fakekms.
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Question about xen and Rasp 4B
2021-02-01 19:25 ` Stefano Stabellini
2021-02-02 7:47 ` Jukka Kaartinen
@ 2021-02-04 6:58 ` Jukka Kaartinen
2021-02-05 1:23 ` Stefano Stabellini
1 sibling, 1 reply; 33+ messages in thread
From: Jukka Kaartinen @ 2021-02-04 6:58 UTC (permalink / raw)
To: Stefano Stabellini, Julien Grall
Cc: Xen-devel, Roman Shaposhnik, Bertrand Marquis
On 1.2.2021 21.25, Stefano Stabellini wrote:
> On Sat, 30 Jan 2021, Julien Grall wrote:
>>>> On 27/01/2021 11:47, Jukka Kaartinen wrote:
>>>>>
>>>>>
>>>>> On Tue, Jan 26, 2021 at 10:22 PM Stefano Stabellini
>>>>> <sstabellini@kernel.org <mailto:sstabellini@kernel.org>> wrote:
>>>>>
>>>>> On Tue, 26 Jan 2021, Jukka Kaartinen wrote:
>>>>> > On Tue, Jan 26, 2021 at 2:54 AM Stefano Stabellini
>>>>> <sstabellini@kernel.org <mailto:sstabellini@kernel.org>> wrote:
>>>>> > On Sat, 23 Jan 2021, Jukka Kaartinen wrote:
>>>>> > > Thanks for the response!
>>>>> > >
>>>>> > > On Sat, Jan 23, 2021 at 2:27 AM Stefano Stabellini
>>>>> <sstabellini@kernel.org <mailto:sstabellini@kernel.org>> wrote:
>>>>> > > + xen-devel, Roman,
>>>>> > >
>>>>> > >
>>>>> > > On Fri, 22 Jan 2021, Jukka Kaartinen wrote:
>>>>> > > > Hi Stefano,
>>>>> > > > I'm Jukka Kaartinen a SW developer working on
>>>>> enabling hypervisors on mobile platforms. One of our HW that we use
>>>>> on
>>>>> > > development is
>>>>> > > > Raspberry Pi 4B. I wonder if you could help me a
>>>>> bit :).
>>>>> > > >
>>>>> > > > I'm trying to enable the GPU with Xen + Raspberry
>>>>> Pi for
>>>>> > > dom0.
>>>>> https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323#p1797605
>>>>> <https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323#p1797605>
>>>>> > > >
>>>>> > > > I got so far that GPU drivers are loaded (v3d &
>>>>> vc4) without errors. But now Xen returns error when X is starting:
>>>>> > > > (XEN) traps.c:1986:d0v1 HSR=0x93880045
>>>>> pc=0x00007f97b14e70 gva=0x7f7f817000 gpa=0x0000401315d000
>>>>> > > > I tried to debug what causes this and looks
>>>>> like find_mmio_handler cannot find handler.
>>>>> > > > (See more here:
>>>>> https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323&start=25#p1801691
>>>>> <https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323&start=25#p1801691>
>>>>> )
>>>>> > > >
>>>>> > > > Any ideas why the handler is not found?
>>>>> > >
>>>>> > >
>>>>> > > Hi Jukka,
>>>>> > >
>>>>> > > I am glad to hear that you are interested in Xen on
>>>>> RaspberryPi :-) I
>>>>> > > haven't tried the GPU yet, I have been using the
>>>>> serial only.
>>>>> > > Roman, did you ever get the GPU working?
>>>>> > >
>>>>> > >
>>>>> > > The error is a data abort error: Linux is trying to
>>>>> access an address
>>>>> > > which is not mapped to dom0. The address seems to
>>>>> be 0x401315d000. It is
>>>>> > > a pretty high address; I looked in device tree but
>>>>> couldn't spot it.
>>>>> > >
>>>>> > > >From the HSR (the syndrom register) it looks like
>>>>> it is a translation
>>>>> > > fault at EL1 on stage1. As if the Linux address
>>>>> mapping was wrong.
>>>>> > > Anyone has any ideas how this could happen? Maybe a
>>>>> reserved-memory
>>>>> > > misconfiguration?
>>>>> > >
>>>>> > > I had issues with loading the driver in the first place.
>>>>> Apparently swiotlb is used, maybe it can cause this. I also tried to
>>>>> > enable CMA.
>>>>> > > config.txt:
>>>>> > > dtoverlay=vc4-fkms-v3d,cma=320M@0x0-0x40000000
>>>>> > > gpu_mem=128
>>>>> >
>>>>> > Also looking at your other reply and the implementation of
>>>>> > vc4_bo_create, it looks like this is a CMA problem.
>>>>> >
>>>>> > It would be good to run a test with the swiotlb-xen
>>>>> disabled:
>>>>> >
>>>>> > diff --git a/arch/arm/xen/mm.c b/arch/arm/xen/mm.c
>>>>> > index 467fa225c3d0..2bdd12785d14 100644
>>>>> > --- a/arch/arm/xen/mm.c
>>>>> > +++ b/arch/arm/xen/mm.c
>>>>> > @@ -138,8 +138,7 @@ void
>>>>> xen_destroy_contiguous_region(phys_addr_t pstart, unsigned int
>>>>> order)
>>>>> > static int __init xen_mm_init(void)
>>>>> > {
>>>>> > struct gnttab_cache_flush cflush;
>>>>> > - if (!xen_initial_domain())
>>>>> > - return 0;
>>>>> > + return 0;
>>>>> > xen_swiotlb_init(1, false);
>>>>> >
>>>>> > cflush.op = 0;
>>>>> >
>>>>> > With this change the kernel is not booting up. (btw. I'm using
>>>>> USB SSD for my OS.)
>>>>> > [ 0.071081] bcm2835-dma fe007000.dma: Unable to set DMA mask
>>>>> > [ 0.076277] bcm2835-dma fe007b00.dma: Unable to set DMA mask
>>>>> > (XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=25: not implemented
>>>>> > (XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=15: not implemented
>>>>> > [ 0.592695] pci 0000:00:00.0: Failed to add - passthrough or
>>>>> MSI/MSI-X might fail!
>>>>> > (XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=15: not implemented
>>>>> > [ 0.606819] pci 0000:01:00.0: Failed to add - passthrough or
>>>>> MSI/MSI-X might fail!
>>>>> > [ 1.212820] usb 1-1: device descriptor read/64, error 18
>>>>> > [ 1.452815] usb 1-1: device descriptor read/64, error 18
>>>>> > [ 1.820813] usb 1-1: device descriptor read/64, error 18
>>>>> > [ 2.060815] usb 1-1: device descriptor read/64, error 18
>>>>> > [ 2.845548] usb 1-1: device descriptor read/8, error -61
>>>>> > [ 2.977603] usb 1-1: device descriptor read/8, error -61
>>>>> > [ 3.237530] usb 1-1: device descriptor read/8, error -61
>>>>> > [ 3.369585] usb 1-1: device descriptor read/8, error -61
>>>>> > [ 3.480765] usb usb1-port1: unable to enumerate USB device
>>>>> >
>>>>> > Traces stop here. I could try with a memory card. Maybe it makes
>>>>> a difference.
>>>>>
>>>>> This is very surprising. Disabling swiotlb-xen should make things
>>>>> better
>>>>> not worse. The only reason I can think of why it could make things
>>>>> worse
>>>>> is if Linux runs out of low memory. Julien's patch
>>>>> 437b0aa06a014ce174e24c0d3530b3e9ab19b18b for Xen should have
>>>>> addressed
>>>>> that issue though. Julien, any ideas?
>>>>
>>>> I think, Stefano's small patch is not enough to disable the swiotlb as we
>>>> will still override the DMA ops. You also likely want:
>>>>
>>>> diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
>>>> index 8a8949174b1c..aa43e249ecdd 100644
>>>> --- a/arch/arm/mm/dma-mapping.c
>>>> +++ b/arch/arm/mm/dma-mapping.c
>>>> @@ -2279,7 +2279,7 @@ void arch_setup_dma_ops(struct device *dev, u64
>>>> dma_base, u64 size,
>>>> set_dma_ops(dev, dma_ops);
>>>>
>>>> #ifdef CONFIG_XEN
>>>> - if (xen_initial_domain())
>>>> + if (0 || xen_initial_domain())
>>>> dev->dma_ops = &xen_swiotlb_dma_ops;
>>>> #endif
>>>> dev->archdata.dma_ops_setup = true;
>>>>
>>>> Otherwise, you would still use the swiotlb DMA ops that would not be
>>>> functional as we disabled the swiotlb.
>>>>
>>>> This would explain the following error because it will check whether the
>>>> mask is valid using the callback dma_supported():
>>>>
>>>> [ 0.071081] bcm2835-dma fe007000.dma: Unable to set DMA mask
>>>>
>>> Good catch.
>>> GPU works now and I can start X! Thanks! I was also able to create domU that
>>> runs Raspian OS.
>>
>> Glad to hear it works! IIRC, the swiotlb may become necessary when running
>> guest if the guest memory ends up to be used for DMA transaction.
>
> It is necessary if you are using PV network or PV disk: memory shared by
> another domU could end up being used in a DMA transaction. For that, you
> need swiotlb-xen.
>
>
>>> Now that swiotlb is disabled what does it mean?
>>
>> I can see two reasons:
>> 1) You have limited memory below the 30 bits mark. So Swiotlb and CMA may
>> try to fight for the low memory.
>> 2) We found a few conversion bugs in the swiotlb on RPI4 last year (IIRC the
>> DMA and physical address may be different). I looked at the Linux branch you
>> are using and they seem to all be there. So there might be another bug.
>>
>> I am not sure how to figure out where is the problem. Stefano, do you have a
>> suggestion where to start?
>
> Both 1) and 2) are possible. It is also possible that another driver,
> probably something related to CMA or DRM, has some special dma_ops
> handling that doesn't work well together with swiotlb-xen.
>
> Given that the original error seemed to be related to vc4_bo_create,
> which calls dma_alloc_wc, I would add a couple of printks to
> xen_swiotlb_alloc_coherent to help us figure it out:
>
>
> diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c
> index 2b385c1b4a99..cac8b09af603 100644
> --- a/drivers/xen/swiotlb-xen.c
> +++ b/drivers/xen/swiotlb-xen.c
> @@ -295,6 +295,7 @@ xen_swiotlb_alloc_coherent(struct device *hwdev, size_t size,
> /* Convert the size to actually allocated. */
> size = 1UL << (order + XEN_PAGE_SHIFT);
>
> + printk("DEBUG %s %d size=%lu flags=%lx attr=%lx\n",__func__,__LINE__,size,flags,attrs);
> /* On ARM this function returns an ioremap'ped virtual address for
> * which virt_to_phys doesn't return the corresponding physical
> * address. In fact on ARM virt_to_phys only works for kernel direct
> @@ -315,16 +316,20 @@ xen_swiotlb_alloc_coherent(struct device *hwdev, size_t size,
> phys = dma_to_phys(hwdev, *dma_handle);
> dev_addr = xen_phys_to_dma(hwdev, phys);
> if (((dev_addr + size - 1 <= dma_mask)) &&
> - !range_straddles_page_boundary(phys, size))
> + !range_straddles_page_boundary(phys, size)) {
> *dma_handle = dev_addr;
> - else {
> + printk("DEBUG %s %d phys=%lx dma=%lx\n",__func__,__LINE__,phys,dev_addr);
> + } else {
> if (xen_create_contiguous_region(phys, order,
> fls64(dma_mask), dma_handle) != 0) {
> + printk("DEBUG %s %d\n",__func__,__LINE__);
> xen_free_coherent_pages(hwdev, size, ret, (dma_addr_t)phys, attrs);
> return NULL;
> }
> *dma_handle = phys_to_dma(hwdev, *dma_handle);
> SetPageXenRemapped(virt_to_page(ret));
> + printk("DEBUG %s %d dma_mask=%d page_boundary=%d phys=%lx dma=%lx\n",__func__,__LINE__,
> + ((dev_addr + size - 1 <= dma_mask)),range_straddles_page_boundary(phys, size),phys,*dma_handle);
> }
> memset(ret, 0, size);
> return ret;
>
>
>
>
>>> And also can I pass the GPU to domU? Raspberry Pi 4 is limited HW and
>>> doesn't have IOMMU. I'm trying to create similar OS like QubesOS where GPU,
>>> Network, keyboard/mouse, ... are isolated to their own VMs.
>>
>> Without an IOMMU or any other HW mechamisns (e.g. MPU), it would not be safe
>> to assign a DMA-capable device to a non-trusted VM.
>>
>> If you trust the VM where you assigned a device, then a possible approach
>> would be to have the VM direct mapped (e.g. guest physical address == host
>> physical address). Although, I can foreese some issues if you have multiple
>> VMs requires memory below 30 bits (there seem to be limited amount)>
>>
>> If you don't trust the VM where you assigned a device, then your best option
>> will be to expose a PV interface of the device and have your backend
>> sanitizing the request and issuing it on behalf of the guest.
>
> FYI you could do that with the existing PVFB drivers that only support 2D
> graphics.
>
We really need direct HW access so PVFB is not really an option. And at
this point. We can trust the VMs.
Any idea what am I missing something here is this the way to give domu
access to the memory?
# from dom0: cat /proc/iomem
# fe104000-fe104027 : fe104000.rng rng@7e104000
# to hide the above rng from the dom0 I added these to device-tree and
above line from /proc/iomem dissapiered.
boot2.scr:
fdt set /soc/rng@7e104000 xen,passthrough <0x1>
fdt set /soc/rng@7e104000 status disabled
domu.cfg:
iomem = [ 'fe104,1' ]
domu start but I cannot see that address in the domu iomem range.
Also the device-tree from the domu is quite empty.
Do I need something like:
device_tree = "rng.dtb"
like here:
https://lists.xenproject.org/archives/html/xen-devel/2018-01/msg02618.html
I tried to add
dtdev = [ "/soc/rng@7e104000" ]
but this gives be error:
"libxl: error: libxl_create.c:1107:libxl__domain_config_setdefault:
passthrough not supported on this platform"
Will this happen if generate the rng.dtb?
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: Question about xen and Rasp 4B
2021-02-04 6:58 ` Jukka Kaartinen
@ 2021-02-05 1:23 ` Stefano Stabellini
0 siblings, 0 replies; 33+ messages in thread
From: Stefano Stabellini @ 2021-02-05 1:23 UTC (permalink / raw)
To: Jukka Kaartinen
Cc: Stefano Stabellini, Julien Grall, Xen-devel, Roman Shaposhnik,
Bertrand Marquis
On Thu, 4 Feb 2021, Jukka Kaartinen wrote:
> We really need direct HW access so PVFB is not really an option. And at this
> point. We can trust the VMs.
>
>
> Any idea what am I missing something here is this the way to give domu access
> to the memory?
>
> # from dom0: cat /proc/iomem
> # fe104000-fe104027 : fe104000.rng rng@7e104000
>
> # to hide the above rng from the dom0 I added these to device-tree and above
> line from /proc/iomem dissapiered.
> boot2.scr:
> fdt set /soc/rng@7e104000 xen,passthrough <0x1>
> fdt set /soc/rng@7e104000 status disabled
Leave status to enabled or okay, just set xen,passthrough to 0x1.
> domu.cfg:
> iomem = [ 'fe104,1' ]
>
> domu start but I cannot see that address in the domu iomem range.
> Also the device-tree from the domu is quite empty.
>
> Do I need something like:
> device_tree = "rng.dtb"
>
> like here:
> https://lists.xenproject.org/archives/html/xen-devel/2018-01/msg02618.html
>
>
> I tried to add
> dtdev = [ "/soc/rng@7e104000" ]
Yes you need both dtdev and device_tree, see
https://xenbits.xenproject.org/docs/unstable/misc/arm/passthrough.txt
iomem is to remap memory regions
irqs is to remap interrupts
device_tree is to populate the DomU device tree
dtdev is to setup the IOMMU, linking to the original device in the host DT
> but this gives be error:
> "libxl: error: libxl_create.c:1107:libxl__domain_config_setdefault:
> passthrough not supported on this platform"
>
> Will this happen if generate the rng.dtb?
This usually happens because there is no IOMMU on the board, or the
IOMMU is disabled. Indeed, the Raspberry Pi 4 doesn't seem to have one.
Now this is going to make things more difficult: without the IOMMU there
can be no protection. In addition, we have a problem with address
translations: when the domU programs a device to do DMA, it uses its
"fake" pseudo-physical addresses. When the device starts the DMA
transaction with the "fake" address, the IOMMU translates it back to a
real physical address.
For this reason, there is no way to assign a device to a domU without
IOMMU on upstream Xen. However, I sent out a patch series a while back
to allow to create domUs with memory 1:1 mapped (pseudo-physical ==
physical). With those patches applied, there is no issue with address
translations anymore and you can assign a device to a domU even without
IOMMU. However, keep in mind that there is going to be no protection.
The series only works for dom0less domUs for now, but it shouldn't be
hard to make it work for any other domUs.
You can find the series here in the Xilinx Xen branch:
https://github.com/Xilinx/xen/tree/xilinx/release-2020.2
and specifically the relevant patches start here:
https://github.com/Xilinx/xen/commit/b8953d357aa095d1027156cf386ad37bd8a34da5
up until:
https://github.com/Xilinx/xen/commit/a7b332c420da40aa3192a8b77c65bcdb1935b5ab
^ permalink raw reply [flat|nested] 33+ messages in thread
end of thread, other threads:[~2021-02-05 1:23 UTC | newest]
Thread overview: 33+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <CAFnJQOouOUox_kBB66sfSuriUiUSvmhSjPxucJjgrB9DdLOwkg@mail.gmail.com>
2021-01-23 0:27 ` Question about xen and Rasp 4B Stefano Stabellini
2021-01-23 1:00 ` Roman Shaposhnik
2021-01-23 14:33 ` Jukka Kaartinen
2021-01-23 14:29 ` Jukka Kaartinen
2021-01-26 0:54 ` Stefano Stabellini
2021-01-26 10:59 ` Jukka Kaartinen
2021-01-26 13:58 ` Jukka Kaartinen
2021-01-26 20:22 ` Stefano Stabellini
2021-01-27 11:47 ` Jukka Kaartinen
2021-01-28 1:10 ` Stefano Stabellini
2021-01-28 2:19 ` Roman Shaposhnik
2021-01-30 1:42 ` Stefano Stabellini
2021-01-30 13:44 ` Jukka Kaartinen
2021-01-30 13:55 ` Julien Grall
2021-01-29 18:50 ` Julien Grall
2021-01-30 1:37 ` Stefano Stabellini
2021-01-30 13:53 ` Jukka Kaartinen
2021-01-30 14:18 ` Julien Grall
2021-02-01 19:25 ` Stefano Stabellini
2021-02-02 7:47 ` Jukka Kaartinen
2021-02-04 6:58 ` Jukka Kaartinen
2021-02-05 1:23 ` Stefano Stabellini
2021-02-02 2:22 ` Roman Shaposhnik
2021-02-02 8:12 ` Jukka Kaartinen
2021-02-03 0:18 ` Stefano Stabellini
2021-02-03 1:58 ` Elliott Mitchell
2021-02-03 7:36 ` Jukka Kaartinen
2021-02-03 20:55 ` Stefano Stabellini
2021-02-03 22:12 ` Elliott Mitchell
2021-02-04 6:46 ` Jukka Kaartinen
2021-02-04 6:44 ` Jukka Kaartinen
2021-01-23 16:48 ` Julien Grall
2021-01-24 16:23 ` Jukka Kaartinen
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.