From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dushyant Behl Subject: Re: Running Xen on Nvidia Jetson-TK1 Date: Sun, 13 Mar 2016 01:53:34 +0530 Message-ID: References: <201603080833.u288Xbfi64553164@d28relay03.in.ibm.com> <20160309133750.GD2589@citrix.com> <56E10135.8050607@arm.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============6902619615768242193==" Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" To: Meng Xu Cc: Julien Grall , Stefano Stabellini , Wei Liu , Dushyant K Behl , Xen Devel List-Id: xen-devel@lists.xenproject.org --===============6902619615768242193== Content-Type: multipart/alternative; boundary=001a11c39e729183c2052ddfd318 --001a11c39e729183c2052ddfd318 Content-Type: text/plain; charset=UTF-8 Hi Meng, On Sat, Mar 12, 2016 at 8:57 PM, Meng Xu wrote: > On Sat, Mar 12, 2016 at 9:20 AM, Dushyant Behl > wrote: > > Hi Julien, > > > > Thanks for the quick reply. > > > > On Thu, Mar 10, 2016 at 10:38 AM, Julien Grall > wrote: > >> > >> Hi Dushyant, > >> > >> On 09/03/2016 20:37, Wei Liu wrote: > >>> > >>> On Tue, Mar 08, 2016 at 08:23:29AM +0000, Dushyant K Behl wrote: > >>>> > >>>> I'm working on a research project with IBM, and I want to run Xen on > >>>> Nvidia > >>>> Tegra Jetson-tk1 board. > >>>> I looked at a post on this mailing list > >>>> (http://lists.xenproject.org/archives/ > >>>> html/xen-devel/2015-03/msg01122.html), > >>>> and I am using this git tree - > >>>> > >>>> git://xenbits.xen.org/people/ianc/xen.git > >>>> and branch - tegra-tk1-jetson-v1 > >>>> > >>>> But when I try to boot Xen on the board I am not able to see any > output > >>>> (even > >>>> with earlyprintk enabled). > >>>> After jumping to Xen the board just resets without showing any output. > >>>> > >>>> I am using upstream u-boot with non secure mode enabled. I have also > >>>> tested > >>>> booting the Linux kernel on the same setup > >>>> and Linux 4.0 is able to boot with all 4 cores in HYP mode and kvm > >>>> enabled. > >>>> > >>>> Can anyone help me as to what I might have done wrong while using Xen? > >> > >> > >> I never tried to boot Xen on Jetson-TK1 myself. > >> > >> Could you provide the command line you use to compile Xen (with > >> earlyprintk enabled) and the U-boot runes? > > > > > > I am running a ubuntu trusty schroot and I am using the Ubuntu/Linaro > > arm-linux-gnueabihf-gcc 4.7.3 > > as the cross compiler to compile everything. The tree which I pointed > > towards (in my previous mail) > > contains the earlyprintk configurations for Jetson-TK1, so I am using > this > > command to compile Xen with earlyprintk - > > > > make dist-xen debug=y CONFIG_EARLY_PRINTK=jetson XEN_TARGET_ARCH=arm32 > > > > I want to update my current stage with Xen, after using this toolchain I > am > > able to get Xen to print output > > on the Jetson-TK1's serial console and Xen is able to boot correctly with > > all 4 cores in HYP mode. > > > > But the problem is when Xen tries to boot the dom0 kernel and transfer > > control, I receive no output on the serial port. > > I am using Linux 4.1.0 as the dom0 kernel and the kernel is compiled with > > Xen specific options enabled. > > Also the same linux kernel is able to boot on the board without Xen. > > > > I am passing the dom0 kernel argument to Xen in the /chosen node in the > > device tree using these commands - > > > > fdt mknod /chosen module > > fdt set /chosen/module compatible "xen,linux-zimage" > "xen,multiboot-module" > > fdt set /chosen/module reg <0x0 $kernel_addr_r 0x0 0x$filesize> > > fdt set /chosen/module bootargs "$bootargs" > > I haven't run Xen on ARM yet. But just a random thought: > Is it possible that you need to provide the console=hvc0 for the boot > cmdline of dom0? > > dom0 needs share the serial port with Xen. I have added the serial port in the bootargs passed to the kernel. The bootargs passed to the dom0 when executing with Xen are the same bootargs which I pass when I run the kernel standalone without Xen and at that time I am able to see the kernel bootlog and a tty console on the serial port. > > > > I am loading Xen at top of the ram and linux and device tree at their > > default locations (near starting of ram). > > > > This is the Xen BootLog which I receive through the earlyprintk serial > port > > - > > > > - UART enabled - > > - CPU 00000000 booting - > > - Xen starting in Hyp mode - > > - Zero BSS - > > - Setting up control registers - > > - CPU Init Done -- Turning on paging - > > - Ready - > > (XEN) Checking for initrd in /chosen > > (XEN) RAM: 0000000080000000 - 00000000ffefffff > > (XEN) > > (XEN) MODULE[0]: 0000000082000000 - 0000000082010000 Device Tree > > (XEN) MODULE[1]: 0000000081000000 - 0000000081544448 Kernel > > (XEN) RESVD[0]: 0000000082000000 - 0000000082010000 > > (XEN) > > (XEN) Command line: > > (XEN) Placing Xen at 0x00000000ffc00000-0x00000000ffe00000 > > (XEN) Update BOOTMOD_XEN from 00000000fd000000-00000000fd0f9701 => > > 00000000ffc00000-00000000ffcf9701 > > (XEN) Xen heap: 00000000fa000000-00000000fe000000 (16384 pages) > > (XEN) Dom heap: 507648 pages > > (XEN) Domain heap initialised > > (XEN) Platform: TEGRA124 > > (XEN) No dtuart path configured > > (XEN) Bad console= option 'dtuart' > > Xen 4.6-unstable > > (XEN) Xen version 4.6-unstable (root@) (arm-linux-gnueabihf-gcc > > (Ubuntu/Linaro 4.7.3-11ubuntu1) 4.7.3) debug=y Fri Mar 11 10:09:07 UTC > 2016 > > (XEN) Latest ChangeSet: > > (XEN) Processor: 413fc0f3: "ARM Limited", variant: 0x3, part 0xc0f, rev > 0x3 > > (XEN) 32-bit Execution: > > (XEN) Processor Features: 00001131:00011011 > > (XEN) Instruction Sets: AArch32 A32 Thumb Thumb-2 ThumbEE Jazelle > > (XEN) Extensions: GenericTimer Security > > (XEN) Debug Features: 02010555 > > (XEN) Auxiliary Features: 00000000 > > (XEN) Memory Model Features: 10201105 40000000 01240000 02102211 > > (XEN) ISA Features: 02101110 13112111 21232041 11112131 10011142 > 00000000 > > (XEN) Using PSCI-0.1 for SMP bringup > > (XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27 Freq: 12000 KHz > > (XEN) GICv2 initialization: > > (XEN) gic_dist_addr=0000000050041000 > > (XEN) gic_cpu_addr=0000000050042000 > > (XEN) gic_hyp_addr=0000000050044000 > > (XEN) gic_vcpu_addr=0000000050046000 > > (XEN) gic_maintenance_irq=25 > > (XEN) GICv2: 192 lines, 4 cpus, secure (IID 0000043b). > > (XEN) Using scheduler: SMP Credit Scheduler (credit) > > (XEN) I/O virtualisation disabled > > (XEN) Allocated console ring of 32 KiB. > > (XEN) VFP implementer 0x41 architecture 4 part 0x30 variant 0xf rev 0x0 > > (XEN) Bringing up CPU1 > > - CPU 00000001 booting - > > - Xen starting in Hyp mode - > > - Setting up control registers - > > - CPU Init Done -- Turning on paging - > > - Ready - > > (XEN) CPU 1 booted. > > (XEN) Bringing up CPU2 > > - CPU 00000002 booting - > > - Xen starting in Hyp mode - > > - Setting up control registers - > > - CPU Init Done -- Turning on paging - > > - Ready - > > (XEN) CPU 2 booted. > > (XEN) Bringing up CPU3 > > - CPU 00000003 booting - > > - Xen starting in Hyp mode - > > - Setting up control registers - > > - CPU Init Done -- Turning on paging - > > - Ready - > > (XEN) CPU 3 booted. > > (XEN) Brought up 4 CPUs > > (XEN) P2M: 40-bit IPA > > (XEN) P2M: 3 levels with order-1 root, VTCR 0x80003558 > > (XEN) *** LOADING DOMAIN 0 *** > > (XEN) Loading kernel from boot module @ 0000000081000000 > > (XEN) Allocating 1:1 mappings totalling 128MB for dom0: > > (XEN) BANK[0] 0x00000088000000-0x00000090000000 (128MB) > > (XEN) Additional MMIO 40000-40040 (IRAM) > > (XEN) Additional MMIO 54200-54240 (Display A) > > (XEN) Additional MMIO 54240-54280 (Display B) > > (XEN) Additional MMIO 6000f-60010 (EXCEPTION VECTORS) > > (XEN) Additional MMIO 6000c-6000d (SYSREG) > > (XEN) Additional MMIO 1000-1001 (PCI CFG0) > > (XEN) Additional MMIO 1001-1002 (PCI CFG1) > > (XEN) Additional MMIO 12000-12010 (PCI IO) > > (XEN) Additional MMIO 13000-20000 (PCI MEM) > > (XEN) Additional MMIO 20000-40000 (PCI MEM (PREFETCH)) > > (XEN) Additional IRQ 105 (DISPLAY) > > (XEN) TEGRA: Routing IRQ105 to dom0, ICTLR2, mask 0x000200 > > (XEN) Additional IRQ 106 (DISPLAY B) > > (XEN) TEGRA: Routing IRQ106 to dom0, ICTLR2, mask 0x000400 > > (XEN) Loading zImage from 0000000081000000 to > > 000000008fa00000-000000008ff44448 > > (XEN) Allocating PPI 16 for event channel interrupt > > (XEN) Loading dom0 DTB to 0x000000008f800000-0x000000008f80f4a4 > > (XEN) Scrubbing Free RAM on 1 nodes using 4 CPUs > > (XEN) ....done. > > (XEN) Initial low memory virq threshold set at 0x4000 pages. > > (XEN) Std. Loglevel: All > > (XEN) Guest Loglevel: All > > (XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch > input to > > Xen) > > (XEN) Freed 272kB init memory. > > > > After this last line the system just hangs with no output on the serial > > console. > > But is dom0 booted up? > If you have network, can you connect to it? > If dom0 did boot up, then at least we know it's some configuration issue. > Again, I haven't tried on ARM yet, but I'd love to... :-) Yes, I have network on the board and I have configured the board to get fixed ip from my local network but even after waiting sometime I am not able to see any network response from the board. So I guessed maybe the dom0 is not booting correctly. Thanks, Dushyant > > Best, > > Meng > > -- > Meng Xu > PhD Student in Computer and Information Science > University of Pennsylvania > http://www.cis.upenn.edu/~mengxu/ > --001a11c39e729183c2052ddfd318 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Meng,

On Sat, Mar 12, 2016 at 8:57 PM, Meng Xu <= mengxu@cis.upenn.= edu> wrote:
On Sat, Mar 12, 2016 at 9:20 AM, Dushyant Behl
<myselfdushyantbehl@gmai= l.com> wrote:
> Hi Julien,
>
> Thanks for the quick reply.
>
> On Thu, Mar 10, 2016 at 10:38 AM, Julien Grall <julien.grall@arm.com> wrote:
>>
>> Hi Dushyant,
>>
>> On 09/03/2016 20:37, Wei Liu wrote:
>>>
>>> On Tue, Mar 08, 2016 at 08:23:29AM +0000, Dushyant K Behl wrot= e:
>>>>
>>>> I'm working on a research project with IBM, and I want= to run Xen on
>>>> Nvidia
>>>> Tegra Jetson-tk1 board.
>>>> I looked at a post on this mailing list
>>>> (http://lists.xenproject.org/archives/
>>>> html/xen-devel/2015-03/msg01122.html),
>>>> and I am using this git tree -
>>>>
>>>> git://xenbits.xen.org/people/ianc/xen.git=
>>>> and branch=C2=A0 - tegra-tk1-jetson-v1
>>>>
>>>> But when I try to boot Xen on the board I am not able to s= ee any output
>>>> (even
>>>> with earlyprintk enabled).
>>>> After jumping to Xen the board just resets without showing= any output.
>>>>
>>>> I am using upstream u-boot with non secure mode enabled. I= have also
>>>> tested
>>>> booting the Linux kernel on the same setup
>>>> and Linux 4.0 is able to boot with all 4 cores in HYP mode= and kvm
>>>> enabled.
>>>>
>>>> Can anyone help me as to what I might have done wrong whil= e using Xen?
>>
>>
>> I never tried to boot Xen on Jetson-TK1 myself.
>>
>> Could you provide the command line you use to compile Xen (with >> earlyprintk enabled) and the U-boot runes?
>
>
> I am running a ubuntu trusty schroot and I am using the Ubuntu/Linaro<= br> > arm-linux-gnueabihf-gcc 4.7.3
> as the cross compiler to compile everything. The tree which I pointed<= br> > towards (in my previous mail)
> contains the earlyprintk configurations for Jetson-TK1, so I am using = this
> command to compile Xen with earlyprintk -
>
> make dist-xen debug=3Dy CONFIG_EARLY_PRINTK=3Djetson XEN_TARGET_ARCH= =3Darm32
>
> I want to update my current stage with Xen, after using this toolchain= I am
> able to get Xen to print output
> on the Jetson-TK1's serial console and Xen is able to boot correct= ly with
> all 4 cores in HYP mode.
>
> But the problem is when Xen tries to boot the dom0 kernel and transfer=
> control, I receive no output on the serial port.
> I am using Linux 4.1.0 as the dom0 kernel and the kernel is compiled w= ith
> Xen specific options enabled.
> Also the same linux kernel is able to boot on the board without Xen. >
> I am passing the dom0 kernel argument to Xen in the /chosen node in th= e
> device tree using these commands=C2=A0 -
>
> fdt mknod /chosen module
> fdt set /chosen/module compatible "xen,linux-zimage" "x= en,multiboot-module"
> fdt set /chosen/module reg <0x0 $kernel_addr_r 0x0 0x$filesize><= br> > fdt set /chosen/module bootargs "$bootargs"

I haven't run Xen on ARM yet. But just a random thought: Is it possible that you need to provide the console=3Dhvc0 for the boot
cmdline of dom0?

dom0 needs share the serial port with Xen.

= I have added the serial port in the bootargs passed to the kernel. The boot= args
passed to the dom0 when executing with Xen are the same boot= args which I pass=C2=A0
when I run the kernel standalone without = Xen and at that time I am able to see the=C2=A0
kernel bootlog an= d a tty console on the serial port.
=C2=A0
>
> I am loading Xen at top of the ram and linux and device tree at their<= br> > default locations (near starting of ram).
>
> This is the Xen BootLog which I receive through the earlyprintk serial= port
> -
>
> - UART enabled -
> - CPU 00000000 booting -
> - Xen starting in Hyp mode -
> - Zero BSS -
> - Setting up control registers -
> - CPU Init Done -- Turning on paging -
> - Ready -
> (XEN) Checking for initrd in /chosen
> (XEN) RAM: 0000000080000000 - 00000000ffefffff
> (XEN)
> (XEN) MODULE[0]: 0000000082000000 - 0000000082010000 Device Tree
> (XEN) MODULE[1]: 0000000081000000 - 0000000081544448 Kernel
> (XEN)=C2=A0 RESVD[0]: 0000000082000000 - 0000000082010000
> (XEN)
> (XEN) Command line: <NULL>
> (XEN) Placing Xen at 0x00000000ffc00000-0x00000000ffe00000
> (XEN) Update BOOTMOD_XEN from 00000000fd000000-00000000fd0f9701 =3D>= ;
> 00000000ffc00000-00000000ffcf9701
> (XEN) Xen heap: 00000000fa000000-00000000fe000000 (16384 pages)
> (XEN) Dom heap: 507648 pages
> (XEN) Domain heap initialised
> (XEN) Platform: TEGRA124
> (XEN) No dtuart path configured
> (XEN) Bad console=3D option 'dtuart'
>=C2=A0 Xen 4.6-unstable
> (XEN) Xen version 4.6-unstable (root@) (arm-linux-gnueabihf-gcc
> (Ubuntu/Linaro 4.7.3-11ubuntu1) 4.7.3) debug=3Dy Fri Mar 11 10:09:07 U= TC 2016
> (XEN) Latest ChangeSet:
> (XEN) Processor: 413fc0f3: "ARM Limited", variant: 0x3, part= 0xc0f, rev 0x3
> (XEN) 32-bit Execution:
> (XEN)=C2=A0 =C2=A0Processor Features: 00001131:00011011
> (XEN)=C2=A0 =C2=A0 =C2=A0Instruction Sets: AArch32 A32 Thumb Thumb-2 T= humbEE Jazelle
> (XEN)=C2=A0 =C2=A0 =C2=A0Extensions: GenericTimer Security
> (XEN)=C2=A0 =C2=A0Debug Features: 02010555
> (XEN)=C2=A0 =C2=A0Auxiliary Features: 00000000
> (XEN)=C2=A0 =C2=A0Memory Model Features: 10201105 40000000 01240000 02= 102211
> (XEN)=C2=A0 ISA Features: 02101110 13112111 21232041 11112131 10011142= 00000000
> (XEN) Using PSCI-0.1 for SMP bringup
> (XEN) Generic Timer IRQ: phys=3D30 hyp=3D26 virt=3D27 Freq: 12000 KHz<= br> > (XEN) GICv2 initialization:
> (XEN)=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0gic_dist_addr=3D000000005004100= 0
> (XEN)=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0gic_cpu_addr=3D0000000050042000=
> (XEN)=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0gic_hyp_addr=3D0000000050044000=
> (XEN)=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0gic_vcpu_addr=3D000000005004600= 0
> (XEN)=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0gic_maintenance_irq=3D25
> (XEN) GICv2: 192 lines, 4 cpus, secure (IID 0000043b).
> (XEN) Using scheduler: SMP Credit Scheduler (credit)
> (XEN) I/O virtualisation disabled
> (XEN) Allocated console ring of 32 KiB.
> (XEN) VFP implementer 0x41 architecture 4 part 0x30 variant 0xf rev 0x= 0
> (XEN) Bringing up CPU1
> - CPU 00000001 booting -
> - Xen starting in Hyp mode -
> - Setting up control registers -
> - CPU Init Done -- Turning on paging -
> - Ready -
> (XEN) CPU 1 booted.
> (XEN) Bringing up CPU2
> - CPU 00000002 booting -
> - Xen starting in Hyp mode -
> - Setting up control registers -
> - CPU Init Done -- Turning on paging -
> - Ready -
> (XEN) CPU 2 booted.
> (XEN) Bringing up CPU3
> - CPU 00000003 booting -
> - Xen starting in Hyp mode -
> - Setting up control registers -
> - CPU Init Done -- Turning on paging -
> - Ready -
> (XEN) CPU 3 booted.
> (XEN) Brought up 4 CPUs
> (XEN) P2M: 40-bit IPA
> (XEN) P2M: 3 levels with order-1 root, VTCR 0x80003558
> (XEN) *** LOADING DOMAIN 0 ***
> (XEN) Loading kernel from boot module @ 0000000081000000
> (XEN) Allocating 1:1 mappings totalling 128MB for dom0:
> (XEN) BANK[0] 0x00000088000000-0x00000090000000 (128MB)
> (XEN) Additional MMIO 40000-40040 (IRAM)
> (XEN) Additional MMIO 54200-54240 (Display A)
> (XEN) Additional MMIO 54240-54280 (Display B)
> (XEN) Additional MMIO 6000f-60010 (EXCEPTION VECTORS)
> (XEN) Additional MMIO 6000c-6000d (SYSREG)
> (XEN) Additional MMIO 1000-1001 (PCI CFG0)
> (XEN) Additional MMIO 1001-1002 (PCI CFG1)
> (XEN) Additional MMIO 12000-12010 (PCI IO)
> (XEN) Additional MMIO 13000-20000 (PCI MEM)
> (XEN) Additional MMIO 20000-40000 (PCI MEM (PREFETCH))
> (XEN) Additional IRQ 105 (DISPLAY)
> (XEN) TEGRA: Routing IRQ105 to dom0, ICTLR2, mask 0x000200
> (XEN) Additional IRQ 106 (DISPLAY B)
> (XEN) TEGRA: Routing IRQ106 to dom0, ICTLR2, mask 0x000400
> (XEN) Loading zImage from 0000000081000000 to
> 000000008fa00000-000000008ff44448
> (XEN) Allocating PPI 16 for event channel interrupt
> (XEN) Loading dom0 DTB to 0x000000008f800000-0x000000008f80f4a4
> (XEN) Scrubbing Free RAM on 1 nodes using 4 CPUs
> (XEN) ....done.
> (XEN) Initial low memory virq threshold set at 0x4000 pages.
> (XEN) Std. Loglevel: All
> (XEN) Guest Loglevel: All
> (XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times t= o switch input to
> Xen)
> (XEN) Freed 272kB init memory.
>
> After this last line the system just hangs with no output on the seria= l
> console.

But is dom0 booted up?
If you have network, can you connect to it?
If dom0 did boot up, then at least we know it's some configuration issu= e.
Again, I haven't tried on ARM yet, but I'd love to... :-)

Yes, I have network on the board and I have configur= ed the board to get fixed ip from my local=C2=A0
network but even= after waiting sometime I am not able to see any network response from the = board.
So I guessed maybe the dom0 is not booting correctly.=C2= =A0

Thanks,
Dushyant
=C2=A0

Best,

Meng

--
Meng Xu
PhD Student in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/

--001a11c39e729183c2052ddfd318-- --===============6902619615768242193== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: inline X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KWGVuLWRldmVs IG1haWxpbmcgbGlzdApYZW4tZGV2ZWxAbGlzdHMueGVuLm9yZwpodHRwOi8vbGlzdHMueGVuLm9y Zy94ZW4tZGV2ZWwK --===============6902619615768242193==--