* Re: Xen ARM and rtds real-time scheduler: local_irq_is_enabled() assert
2014-09-26 14:02 Xen ARM and rtds real-time scheduler: local_irq_is_enabled() assert Leonardo Taccari
@ 2014-09-26 14:17 ` Konrad Rzeszutek Wilk
2014-09-26 15:42 ` Meng Xu
2014-09-29 11:48 ` Dario Faggioli
2 siblings, 0 replies; 6+ messages in thread
From: Konrad Rzeszutek Wilk @ 2014-09-26 14:17 UTC (permalink / raw)
To: Leonardo Taccari; +Cc: Meng Xu, Sisu Xi, xen-devel
On Fri, Sep 26, 2014 at 04:02:39PM +0200, Leonardo Taccari wrote:
> Good afternoon to the entire Xen community!
> In the previous months I have succesfully installed and run xen-4.4 on a
> cubieboard2 (running Debian GNU/Linux testing armhf).
>
> Now I am interested on trying and testing the rtds real-time scheduler
> and so I have compiled xen-devel (master branch ~1-2 days ago) via an
> armhf chroot generated via debootstrap[0] (Debian testing, the same
> version of the one I have installed in the microsd of cubieboard2),
> running everything through QEMU user mode emulation (dynamic binary
> translation).
> I have used this setup in order to compile xen toolstack on an amd64
> machine (and thanks to the armhf chroot and QEMU user mode emulation I
> avoided the cross-compilation).
>
> I have applied the four patches regarding rtds real-time scheduler
> appeared on xen-devel@ ML with the subject "[PATCH for 4.5 v4 [1-4]/4]"
> (I've also needed to apply a patch that fixes a printf(3) syntax in
> burn_budget() from Ian Campbell[1]).
Would it be possible to use Xen 4.5 for your testing? It has
these patches already in?
>
> I tried to reproduce the scenario described by Meng Xu.
> Here it is the complete output on the cubieboard2 (connected via an USB
> TTL UART), the problem regards local_irq_is_enabled() assertion is at the
> end with the Xen stack trace and call trace:
>
>
> [...]
> root@cubieboard:~# /etc/init.d/xencommons start
> Starting /usr/local/sbin/xenstored...
> Setting domain 0 name, domid and JSON config...
> Done setting up Dom0
> Starting xenconsoled...
> Starting QEMU as disk backend for dom0
> /etc/init.d/xencommons: line 114: /usr/local/lib/xen/bin/qemu-system-i386: No
> such file or directory
> root@cubieboard:~# xl sched-rtds
> root@cubieboard:~# xl sched-rtds -d Domain-0 -p 20000 -b 10000
> libxl: error: libxl.c:5481:sched_rtds_domain_set: getting domain sched rtds:
> Invalid argument libxl_domain_sched_params_set failed.
> root@cubieboard:~# xl cpupool-list
> Name CPUs Sched Active Domain count
> Pool-0 2 credit y 1
> root@cubieboard:~# xl cpupool-cpu-remove Pool-0 1
> root@cubieboard:~# xl cpupool-list
> Name CPUs Sched Active Domain count
> Pool-0 1 credit y 1
> root@cubieboard:~# xl cpupool-create name=\"rt\" sched=\"rtds\"
> Using config file "command line"
> cpupool name: rt
> scheduler: rtds
> number of cpus: 0
> (XEN) Initializing RTDS scheduler
> (XEN) WARNING: This is experimental software in development.
> (XEN) Use at your own risk.
> root@cubieboard:~# xl cpupool-cpu-add rt 1
> root@cubieboard:~# cd xen/debian-testing
> root@cubieboard:~/xen/debian-testing# losetup /dev/loop0 /root/xen/debian-testing/debian-testing.img
> root@cubieboard:~/xen/debian-testing# xl create -c debian-testing.cfg
> root@cubieboard:~# xl list
> Name ID Mem VCPUs State Time(s)
> Domain-0 0 128 2 r----- 14.3
> debian-testing 1 128 1 ------ 6.5
> root@cubieboard:~# xl cpupool-list
> Name CPUs Sched Active Domain count
> Pool-0 1 credit y 2
> rt 1 rtds y 0
> root@cubieboard:~# xl cpupool-migrate debian-testing rt
> (XEN) Assertion 'local_irq_is_enabled()' failed, line 137, file spinlock.c
> root@cubieboard:~/xen/debian-testing# (XEN) Xen BUG at spinlock.c:137
> (XEN) CPU1: Unexpected Trap: Undefined Instruction
> (XEN) ----[ Xen-4.5-unstable arm32 debug=y Not tainted ]----
> (XEN) CPU: 1
> (XEN) PC: 0024b590 __bug+0x2c/0x44
> (XEN) CPSR: 200f01da MODE:Hypervisor
> (XEN) R0: 0027b6d4 R1: 3fd11d80 R2: 00000000 R3: 00000000
> (XEN) R4: 00000089 R5: 00272664 R6: 002fe020 R7: 00000049
> (XEN) R8: 4002f9f8 R9: 4002fa78 R10:40038ec0 R11:47fefe3c R12:3fd11d80
> (XEN) HYP: SP: 47fefe34 LR: 0024b590
> (XEN)
> (XEN) VTCR_EL2: 80003558
> (XEN) VTTBR_EL2: 000200007ff04000
> (XEN)
> (XEN) SCTLR_EL2: 30cd187f
> (XEN) HCR_EL2: 000000000038643f
> (XEN) TTBR0_EL2: 0000000076010000
> (XEN)
> (XEN) ESR_EL2: 00000000
> (XEN) HPFAR_EL2: 000000000001c810
> (XEN) HDFAR: c8800f00
> (XEN) HIFAR: 3ba1661c
> (XEN)
> (XEN) Xen stack trace from sp=47fefe34:
> (XEN) 00000000 47fefe4c 00231f1c 003002a8 47fdd000 47fefe7c 0022c2c4 47fdd000
> (XEN) 4002fa78 40011000 47fdd000 40011000 9d305069 00000049 00000001 40012040
> (XEN) 00000001 47fefe8c 0023082c 47fdd000 40011000 47fefe9c 00250758 47fdd000
> (XEN) 40011000 47fefeac 00250e18 40011000 47fdd000 47feff14 0022d5a4 00000000
> (XEN) 002650bc 00000000 4002fa78 000f4240 00000000 47fefef8 00301594 00300324
> (XEN) 00000000 47fdd000 00000000 000f4240 00000000 00000000 00000000 47fefef8
> (XEN) 002c9ff4 00000001 00301594 002c6000 00000000 0027bb80 7fc00000 47feff34
> (XEN) 002310c8 00301594 00301594 00300324 00000000 00000000 7fe00000 47feff3c
> (XEN) 00231190 47feff54 00250ce0 00000001 00000000 10011142 00000000 47feff3c
> (XEN) 0025d6c0 00000000 00000000 00000000 00000000 00000000 00000000 00000000
> (XEN) 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
> (XEN) 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
> (XEN) 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
> (XEN) 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
> (XEN) 00000000 00000000 00000000
> (XEN) Xen call trace:
> (XEN) [<0024b590>] __bug+0x2c/0x44 (PC)
> (XEN) [<0024b590>] __bug+0x2c/0x44 (LR)
> (XEN) [<00231f1c>] _spin_lock_irq+0x4c/0xdc
> (XEN) [<0022c2c4>] rt_context_saved+0x54/0x1d0
> (XEN) [<0023082c>] context_saved+0x7c/0xa4
> (XEN) [<00250758>] schedule_tail+0x150/0x31c
> (XEN) [<00250e18>] context_switch+0x114/0x128
> (XEN) [<0022d5a4>] schedule+0x8e4/0x940
> (XEN) [<002310c8>] __do_softirq+0xf8/0x118
> (XEN) [<00231190>] do_softirq+0x18/0x28
> (XEN) [<00250ce0>] idle_loop+0x1a4/0x1c8
> (XEN) [<0025d6c0>] start_secondary+0x1a0/0x1bc
> (XEN)
> (XEN)
> (XEN) ****************************************
> (XEN) Panic on CPU 1:
> (XEN) CPU1: Unexpected Trap: Undefined Instruction
> (XEN)
> (XEN) ****************************************
> (XEN)
> (XEN) Reboot in five seconds...
> [...]
>
>
> [0]: https://wiki.debian.org/Debootstrap
> [1]: http://lists.xen.org/archives/html/xen-devel/2014-09/msg03352.html
>
>
> Thank you very much for your attention!
> Ciao,
> L.
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Xen ARM and rtds real-time scheduler: local_irq_is_enabled() assert
2014-09-26 14:02 Xen ARM and rtds real-time scheduler: local_irq_is_enabled() assert Leonardo Taccari
2014-09-26 14:17 ` Konrad Rzeszutek Wilk
@ 2014-09-26 15:42 ` Meng Xu
2014-09-27 11:04 ` Leonardo Taccari
2014-09-29 11:48 ` Dario Faggioli
2 siblings, 1 reply; 6+ messages in thread
From: Meng Xu @ 2014-09-26 15:42 UTC (permalink / raw)
To: Leonardo Taccari; +Cc: Meng Xu, Sisu Xi, xen-devel
2014-09-26 10:02 GMT-04:00 Leonardo Taccari <iamleot@gmail.com>:
> Good afternoon to the entire Xen community!
> In the previous months I have succesfully installed and run xen-4.4 on a
> cubieboard2 (running Debian GNU/Linux testing armhf).
>
> Now I am interested on trying and testing the rtds real-time scheduler
> and so I have compiled xen-devel (master branch ~1-2 days ago) via an
> armhf chroot generated via debootstrap[0] (Debian testing, the same
> version of the one I have installed in the microsd of cubieboard2),
> running everything through QEMU user mode emulation (dynamic binary
> translation).
> I have used this setup in order to compile xen toolstack on an amd64
> machine (and thanks to the armhf chroot and QEMU user mode emulation I
> avoided the cross-compilation).
>
> I have applied the four patches regarding rtds real-time scheduler
> appeared on xen-devel@ ML with the subject "[PATCH for 4.5 v4 [1-4]/4]"
> (I've also needed to apply a patch that fixes a printf(3) syntax in
> burn_budget() from Ian Campbell[1]).
>
> I tried to reproduce the scenario described by Meng Xu.
> Here it is the complete output on the cubieboard2 (connected via an USB
> TTL UART), the problem regards local_irq_is_enabled() assertion is at the
> end with the Xen stack trace and call trace:
>
>
> [...]
> root@cubieboard:~# /etc/init.d/xencommons start
> Starting /usr/local/sbin/xenstored...
> Setting domain 0 name, domid and JSON config...
> Done setting up Dom0
> Starting xenconsoled...
> Starting QEMU as disk backend for dom0
> /etc/init.d/xencommons: line 114: /usr/local/lib/xen/bin/qemu-system-i386: No
> such file or directory
> root@cubieboard:~# xl sched-rtds
> root@cubieboard:~# xl sched-rtds -d Domain-0 -p 20000 -b 10000
> libxl: error: libxl.c:5481:sched_rtds_domain_set: getting domain sched rtds:
> Invalid argument libxl_domain_sched_params_set failed.
> root@cubieboard:~# xl cpupool-list
> Name CPUs Sched Active Domain count
> Pool-0 2 credit y 1
> root@cubieboard:~# xl cpupool-cpu-remove Pool-0 1
> root@cubieboard:~# xl cpupool-list
> Name CPUs Sched Active Domain count
> Pool-0 1 credit y 1
> root@cubieboard:~# xl cpupool-create name=\"rt\" sched=\"rtds\"
> Using config file "command line"
> cpupool name: rt
> scheduler: rtds
> number of cpus: 0
> (XEN) Initializing RTDS scheduler
> (XEN) WARNING: This is experimental software in development.
> (XEN) Use at your own risk.
> root@cubieboard:~# xl cpupool-cpu-add rt 1
> root@cubieboard:~# cd xen/debian-testing
> root@cubieboard:~/xen/debian-testing# losetup /dev/loop0 /root/xen/debian-testing/debian-testing.img
> root@cubieboard:~/xen/debian-testing# xl create -c debian-testing.cfg
> root@cubieboard:~# xl list
> Name ID Mem VCPUs State Time(s)
> Domain-0 0 128 2 r----- 14.3
> debian-testing 1 128 1 ------ 6.5
> root@cubieboard:~# xl cpupool-list
> Name CPUs Sched Active Domain count
> Pool-0 1 credit y 2
> rt 1 rtds y 0
> root@cubieboard:~# xl cpupool-migrate debian-testing rt
> (XEN) Assertion 'local_irq_is_enabled()' failed, line 137, file spinlock.c
> root@cubieboard:~/xen/debian-testing# (XEN) Xen BUG at spinlock.c:137
> (XEN) CPU1: Unexpected Trap: Undefined Instruction
> (XEN) ----[ Xen-4.5-unstable arm32 debug=y Not tainted ]----
> (XEN) CPU: 1
> (XEN) PC: 0024b590 __bug+0x2c/0x44
> (XEN) CPSR: 200f01da MODE:Hypervisor
> (XEN) R0: 0027b6d4 R1: 3fd11d80 R2: 00000000 R3: 00000000
> (XEN) R4: 00000089 R5: 00272664 R6: 002fe020 R7: 00000049
> (XEN) R8: 4002f9f8 R9: 4002fa78 R10:40038ec0 R11:47fefe3c R12:3fd11d80
> (XEN) HYP: SP: 47fefe34 LR: 0024b590
> (XEN)
> (XEN) VTCR_EL2: 80003558
> (XEN) VTTBR_EL2: 000200007ff04000
> (XEN)
> (XEN) SCTLR_EL2: 30cd187f
> (XEN) HCR_EL2: 000000000038643f
> (XEN) TTBR0_EL2: 0000000076010000
> (XEN)
> (XEN) ESR_EL2: 00000000
> (XEN) HPFAR_EL2: 000000000001c810
> (XEN) HDFAR: c8800f00
> (XEN) HIFAR: 3ba1661c
> (XEN)
> (XEN) Xen stack trace from sp=47fefe34:
> (XEN) 00000000 47fefe4c 00231f1c 003002a8 47fdd000 47fefe7c 0022c2c4 47fdd000
> (XEN) 4002fa78 40011000 47fdd000 40011000 9d305069 00000049 00000001 40012040
> (XEN) 00000001 47fefe8c 0023082c 47fdd000 40011000 47fefe9c 00250758 47fdd000
> (XEN) 40011000 47fefeac 00250e18 40011000 47fdd000 47feff14 0022d5a4 00000000
> (XEN) 002650bc 00000000 4002fa78 000f4240 00000000 47fefef8 00301594 00300324
> (XEN) 00000000 47fdd000 00000000 000f4240 00000000 00000000 00000000 47fefef8
> (XEN) 002c9ff4 00000001 00301594 002c6000 00000000 0027bb80 7fc00000 47feff34
> (XEN) 002310c8 00301594 00301594 00300324 00000000 00000000 7fe00000 47feff3c
> (XEN) 00231190 47feff54 00250ce0 00000001 00000000 10011142 00000000 47feff3c
> (XEN) 0025d6c0 00000000 00000000 00000000 00000000 00000000 00000000 00000000
> (XEN) 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
> (XEN) 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
> (XEN) 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
> (XEN) 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
> (XEN) 00000000 00000000 00000000
> (XEN) Xen call trace:
> (XEN) [<0024b590>] __bug+0x2c/0x44 (PC)
> (XEN) [<0024b590>] __bug+0x2c/0x44 (LR)
> (XEN) [<00231f1c>] _spin_lock_irq+0x4c/0xdc
> (XEN) [<0022c2c4>] rt_context_saved+0x54/0x1d0
> (XEN) [<0023082c>] context_saved+0x7c/0xa4
> (XEN) [<00250758>] schedule_tail+0x150/0x31c
> (XEN) [<00250e18>] context_switch+0x114/0x128
> (XEN) [<0022d5a4>] schedule+0x8e4/0x940
> (XEN) [<002310c8>] __do_softirq+0xf8/0x118
> (XEN) [<00231190>] do_softirq+0x18/0x28
> (XEN) [<00250ce0>] idle_loop+0x1a4/0x1c8
> (XEN) [<0025d6c0>] start_secondary+0x1a0/0x1bc
> (XEN)
> (XEN)
> (XEN) ****************************************
> (XEN) Panic on CPU 1:
> (XEN) CPU1: Unexpected Trap: Undefined Instruction
> (XEN)
> (XEN) ****************************************
> (XEN)
> (XEN) Reboot in five seconds...
> [...]
>
Could you please have a try: git://xenbits.xen.org/xen.git, staging
branch and let me know if this problem still exist?
I didn't test it on arm, but I think the difference between ARM and
Intel x86 should not make much difference in schedulers.
Thanks,
Meng
--
-----------
Meng Xu
PhD Student in Computer and Information Science
University of Pennsylvania
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Xen ARM and rtds real-time scheduler: local_irq_is_enabled() assert
2014-09-26 14:02 Xen ARM and rtds real-time scheduler: local_irq_is_enabled() assert Leonardo Taccari
2014-09-26 14:17 ` Konrad Rzeszutek Wilk
2014-09-26 15:42 ` Meng Xu
@ 2014-09-29 11:48 ` Dario Faggioli
2 siblings, 0 replies; 6+ messages in thread
From: Dario Faggioli @ 2014-09-29 11:48 UTC (permalink / raw)
To: Leonardo Taccari; +Cc: Meng Xu, Sisu Xi, xen-devel
[-- Attachment #1.1: Type: text/plain, Size: 5754 bytes --]
On ven, 2014-09-26 at 16:02 +0200, Leonardo Taccari wrote:
> Good afternoon to the entire Xen community!
>
Hi Leonardo,
> In the previous months I have succesfully installed and run xen-4.4 on a
> cubieboard2 (running Debian GNU/Linux testing armhf).
>
Cool!
> Now I am interested on trying and testing the rtds real-time scheduler
>
Purely out of curiosity, may I ask why? :-)
> I have applied the four patches regarding rtds real-time scheduler
> appeared on xen-devel@ ML with the subject "[PATCH for 4.5 v4 [1-4]/4]"
> (I've also needed to apply a patch that fixes a printf(3) syntax in
> burn_budget() from Ian Campbell[1]).
>
Right. As other said, you should
> [...]
> root@cubieboard:~# /etc/init.d/xencommons start
> Starting /usr/local/sbin/xenstored...
> Setting domain 0 name, domid and JSON config...
> Done setting up Dom0
> Starting xenconsoled...
> Starting QEMU as disk backend for dom0
> /etc/init.d/xencommons: line 114: /usr/local/lib/xen/bin/qemu-system-i386: No
> such file or directory
> root@cubieboard:~# xl sched-rtds
> root@cubieboard:~# xl sched-rtds -d Domain-0 -p 20000 -b 10000
> libxl: error: libxl.c:5481:sched_rtds_domain_set: getting domain sched rtds:
> Invalid argument libxl_domain_sched_params_set failed.
>
This is correct, you can't change the scheduler for just a domain. What
you must do is what you actually try to do below, i.e., move it in a
cpupool the scheduler for which is the one you want.
> root@cubieboard:~# xl cpupool-list
> Name CPUs Sched Active Domain count
> Pool-0 2 credit y 1
> root@cubieboard:~# xl cpupool-cpu-remove Pool-0 1
> root@cubieboard:~# xl cpupool-list
> Name CPUs Sched Active Domain count
> Pool-0 1 credit y 1
> root@cubieboard:~# xl cpupool-create name=\"rt\" sched=\"rtds\"
> Using config file "command line"
> cpupool name: rt
> scheduler: rtds
> number of cpus: 0
>
Mmm... so, you created a cpupool without any cpu assigned to it.
That's legal, as you can see, but then I don't think you should/could
move domain to it.
And in fact, if you look at xen/common/cpupool.c, where
XEN_SYSCTL_CPUPOOL_OP_MOVEDOMAIN is handled, you'll find this:
if ( (c != NULL) && cpumask_weight(c->cpu_valid) )
{
d->cpupool->n_dom--;
ret = sched_move_domain(d, c);
if ( ret )
d->cpupool->n_dom++;
else
c->n_dom++;
}
In your case, I think cpumask_weight(c->cpu_valid) is 0
> (XEN) Initializing RTDS scheduler
> (XEN) WARNING: This is experimental software in development.
> (XEN) Use at your own risk.
> root@cubieboard:~# xl cpupool-cpu-add rt 1
> root@cubieboard:~# cd xen/debian-testing
> root@cubieboard:~/xen/debian-testing# losetup /dev/loop0 /root/xen/debian-testing/debian-testing.img
> root@cubieboard:~/xen/debian-testing# xl create -c debian-testing.cfg
> root@cubieboard:~# xl list
> Name ID Mem VCPUs State Time(s)
> Domain-0 0 128 2 r----- 14.3
> debian-testing 1 128 1 ------ 6.5
> root@cubieboard:~# xl cpupool-list
> Name CPUs Sched Active Domain count
> Pool-0 1 credit y 2
> rt 1 rtds y 0
> root@cubieboard:~# xl cpupool-migrate debian-testing rt
>
I've only tried on x86, but what I see confirms what I saw above:
root@tg03:~# xl list
Name ID Mem VCPUs State Time(s)
Domain-0 0 1023 24 r----- 20.7
debian.guest.osstest 1 19256 2 -b---- 5.3
root@tg03:~# xl -vvv cpupool-migrate 1 rt
libxl: error: libxl.c:6215:libxl_cpupool_movedomain: Error moving domain to cpupool
root@tg03:~# xl sched-rtds
Cpupool rt: sched=RTDS
Name ID Period Budget
root@tg03:~# xl sched-credit
Cpupool Pool-0: tslice=30ms ratelimit=1000us
Name ID Weight Cap
Domain-0 0 256 0
debian.guest.osstest 1 256 0
> (XEN) Assertion 'local_irq_is_enabled()' failed, line 137, file spinlock.c
> root@cubieboard:~/xen/debian-testing# (XEN) Xen BUG at spinlock.c:137
> (XEN) CPU1: Unexpected Trap: Undefined Instruction
> (XEN) ----[ Xen-4.5-unstable arm32 debug=y Not tainted ]----
> (XEN) Xen call trace:
> (XEN) [<0024b590>] __bug+0x2c/0x44 (PC)
> (XEN) [<0024b590>] __bug+0x2c/0x44 (LR)
> (XEN) [<00231f1c>] _spin_lock_irq+0x4c/0xdc
> (XEN) [<0022c2c4>] rt_context_saved+0x54/0x1d0
> (XEN) [<0023082c>] context_saved+0x7c/0xa4
> (XEN) [<00250758>] schedule_tail+0x150/0x31c
> (XEN) [<00250e18>] context_switch+0x114/0x128
> (XEN) [<0022d5a4>] schedule+0x8e4/0x940
> (XEN) [<002310c8>] __do_softirq+0xf8/0x118
> (XEN) [<00231190>] do_softirq+0x18/0x28
> (XEN) [<00250ce0>] idle_loop+0x1a4/0x1c8
> (XEN) [<0025d6c0>] start_secondary+0x1a0/0x1bc
> (XEN)
>
Interesting... As I said, the domain shouldn't even be there, so I don't
immediately see the root cause for this.
If you try it with latest staging, let us know how it goes.
Regards,
Dario
--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 6+ messages in thread