Dom0 kernel 4.14 with SMP randomly crashing

Dom0 kernel 4.14 with SMP randomly crashing

[Rishi]

[-- Attachment #1.1: Type: text/plain, Size: 678 bytes --]

I've built a dom0 kernel 4.14 with SMP support. The dom0 kernel crashes
when I'm downloading a large file on host. It does not crash if I have
nosmp boot option on xen command line.

my .config SMP options are

[root@f6029920339a wip-kernel-4.14.78]# grep SMP .config

CONFIG_X86_64_*SMP*=y

CONFIG_GENERIC_*SMP*_IDLE_THREAD=y

CONFIG_*SMP*=y

CONFIG_XEN_PV_*SMP*=y

CONFIG_XEN_PVHVM_*SMP*=y

# CONFIG_MAX*SMP* is not set

CONFIG_PM_SLEEP_*SMP*=y

CONFIG_SCSI_SAS_HOST_*SMP*=y


While comparing it to earlier kernels the options

CONFIG_XEN_PV_*SMP*=y

CONFIG_XEN_PVHVM_*SMP*=y


are newly introduced. Does xen need any update to better handle them?

My xen versions is : 4.7.6

[-- Attachment #1.2: Type: text/html, Size: 7440 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Dom0 kernel 4.14 with SMP randomly crashing

[Rishi]

[-- Attachment #1.1: Type: text/plain, Size: 1072 bytes --]

Found the references of these in
https://lists.xenproject.org/archives/html/xen-devel/2017-02/msg03120.html
proposal to split PV, PVHVM and PVH code in kernel.

There is no mention of any change or requirement from Xen perspective. Any
other way to track this problem?

On Sun, Nov 4, 2018 at 6:37 PM Rishi <2rushikeshj@gmail.com> wrote:

> I've built a dom0 kernel 4.14 with SMP support. The dom0 kernel crashes
> when I'm downloading a large file on host. It does not crash if I have
> nosmp boot option on xen command line.
>
> my .config SMP options are
>
> [root@f6029920339a wip-kernel-4.14.78]# grep SMP .config
>
> CONFIG_X86_64_*SMP*=y
>
> CONFIG_GENERIC_*SMP*_IDLE_THREAD=y
>
> CONFIG_*SMP*=y
>
> CONFIG_XEN_PV_*SMP*=y
>
> CONFIG_XEN_PVHVM_*SMP*=y
>
> # CONFIG_MAX*SMP* is not set
>
> CONFIG_PM_SLEEP_*SMP*=y
>
> CONFIG_SCSI_SAS_HOST_*SMP*=y
>
>
> While comparing it to earlier kernels the options
>
> CONFIG_XEN_PV_*SMP*=y
>
> CONFIG_XEN_PVHVM_*SMP*=y
>
>
> are newly introduced. Does xen need any update to better handle them?
>
> My xen versions is : 4.7.6
>

[-- Attachment #1.2: Type: text/html, Size: 8165 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Dom0 kernel 4.14 with SMP randomly crashing

[Wei Liu]

On Sun, Nov 04, 2018 at 06:37:36PM +0530, Rishi wrote:
> I've built a dom0 kernel 4.14 with SMP support. The dom0 kernel crashes
> when I'm downloading a large file on host. It does not crash if I have
> nosmp boot option on xen command line.
> 
> my .config SMP options are
> 
> [root@f6029920339a wip-kernel-4.14.78]# grep SMP .config
> 
> CONFIG_X86_64_*SMP*=y
> 
> CONFIG_GENERIC_*SMP*_IDLE_THREAD=y
> 
> CONFIG_*SMP*=y
> 
> CONFIG_XEN_PV_*SMP*=y
> 
> CONFIG_XEN_PVHVM_*SMP*=y
> 
> # CONFIG_MAX*SMP* is not set
> 
> CONFIG_PM_SLEEP_*SMP*=y
> 
> CONFIG_SCSI_SAS_HOST_*SMP*=y
> 
> 
> While comparing it to earlier kernels the options
> 
> CONFIG_XEN_PV_*SMP*=y
> 
> CONFIG_XEN_PVHVM_*SMP*=y
> 
> 
> are newly introduced. Does xen need any update to better handle them?
> 

Please post the kernel log when it crashed. I'm afraid Xen developers
can't read your mind to figure out what went wrong. :-)

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Dom0 kernel 4.14 with SMP randomly crashing

[Rishi]

[-- Attachment #1.1: Type: text/plain, Size: 1405 bytes --]

Nothing is getting logged. I suspect its because the kernel just goes in
halt state.

/var/log/kern.log /var/log/messages /var/log/xen/hypervisor.log

All of these files have nothing appended.

I can fetch info from kernel debugfs, if you point me to required area,


The dom0 kernel works fine without spilt of SMP code or with "nosmp" option
at xen command line.

On Mon, Nov 5, 2018 at 3:49 PM Wei Liu <wei.liu2@citrix.com> wrote:

> On Sun, Nov 04, 2018 at 06:37:36PM +0530, Rishi wrote:
> > I've built a dom0 kernel 4.14 with SMP support. The dom0 kernel crashes
> > when I'm downloading a large file on host. It does not crash if I have
> > nosmp boot option on xen command line.
> >
> > my .config SMP options are
> >
> > [root@f6029920339a wip-kernel-4.14.78]# grep SMP .config
> >
> > CONFIG_X86_64_*SMP*=y
> >
> > CONFIG_GENERIC_*SMP*_IDLE_THREAD=y
> >
> > CONFIG_*SMP*=y
> >
> > CONFIG_XEN_PV_*SMP*=y
> >
> > CONFIG_XEN_PVHVM_*SMP*=y
> >
> > # CONFIG_MAX*SMP* is not set
> >
> > CONFIG_PM_SLEEP_*SMP*=y
> >
> > CONFIG_SCSI_SAS_HOST_*SMP*=y
> >
> >
> > While comparing it to earlier kernels the options
> >
> > CONFIG_XEN_PV_*SMP*=y
> >
> > CONFIG_XEN_PVHVM_*SMP*=y
> >
> >
> > are newly introduced. Does xen need any update to better handle them?
> >
>
> Please post the kernel log when it crashed. I'm afraid Xen developers
> can't read your mind to figure out what went wrong. :-)
>
> Wei.
>

[-- Attachment #1.2: Type: text/html, Size: 3284 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Dom0 kernel 4.14 with SMP randomly crashing

[Wei Liu]

On Mon, Nov 05, 2018 at 04:26:03PM +0530, Rishi wrote:
> Nothing is getting logged. I suspect its because the kernel just goes in
> halt state.
> 
> /var/log/kern.log /var/log/messages /var/log/xen/hypervisor.log
> 
> All of these files have nothing appended.
> 
> I can fetch info from kernel debugfs, if you point me to required area,
> 
> 
> The dom0 kernel works fine without spilt of SMP code or with "nosmp" option
> at xen command line.

I think you will need to set up a serial console. I wouldn't expect to
see anything in /var/log if the system has crashed.

Without serial log there is nothing we can do to help.

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Dom0 kernel 4.14 with SMP randomly crashing

[Rishi]

[-- Attachment #1.1: Type: text/plain, Size: 3404 bytes --]

Alright, I got the serial console and following is the crash log. Thank you
for pointing that out.

[  133.594852] watchdog: BUG: soft lockup - CPU#2 stuck for 22s!
[ksoftirqd/2:22]

[  133.599232] Kernel panic - not syncing: softlockup: hung tasks

[  133.602275] CPU: 2 PID: 22 Comm: ksoftirqd/2 Tainted: G
L    4.19.1
#1

[  133.606620] Hardware name: Xen HVM domU, BIOS 4.4.1-xs132257 12/12/2016

[  133.610376] Call Trace:

[  133.611827]  <IRQ>

[  133.612942]  dump_stack+0x5a/0x73

[  133.614676]  panic+0xe8/0x249

[  133.616183]  watchdog_timer_fn+0x200/0x230

[  133.618366]  ? softlockup_fn+0x40/0x40

[  133.620561]  __hrtimer_run_queues+0x133/0x270

[  133.623056]  hrtimer_interrupt+0xfb/0x260

[  133.625580]  xen_timer_interrupt+0x1b/0x30

[  133.628011]  __handle_irq_event_percpu+0x69/0x1a0

[  133.630642]  handle_irq_event_percpu+0x30/0x70

[  133.633826]  handle_percpu_irq+0x34/0x50

[  133.636012]  generic_handle_irq+0x1e/0x30

[  133.638573]  __evtchn_fifo_handle_events+0x163/0x1a0

[  133.641444]  __xen_evtchn_do_upcall+0x41/0x70

[  133.643772]  xen_evtchn_do_upcall+0x27/0x50

[  133.645868]  xen_do_hypervisor_callback+0x29/0x40

[  133.648231]  </IRQ>

[  133.649279] RIP: e030:xen_hypercall_xen_version+0xa/0x20

[  133.652259] Code: 51 41 53 b8 10 00 00 00 0f 05 41 5b 59 c3 cc cc cc cc
cc cc cc cc cc cc cc cc cc cc cc cc cc cc 51 41 53 b8 11 00 00 00 0f 05
<41> 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc

[  133.662574] RSP: e02b:ffffc900400dbd18 EFLAGS: 00000246

[  133.665557] RAX: 0000000000040007 RBX: ffff88003315a898 RCX:
ffffffff8100122a

[  133.670005] RDX: ffffc900400080ff RSI: 0000000000000000 RDI:
0000000000000000

[  133.674606] RBP: ffff88003315a890 R08: 0000000000000000 R09:
0000000000000000

[  133.678567] R10: 0000000000000000 R11: 0000000000000246 R12:
0000000080000600

[  133.682552] R13: 000000000000002a R14: 00000000000002a0 R15:
000000000000002a

[  133.686851]  ? xen_hypercall_xen_version+0xa/0x20

[  133.689681]  ? xen_force_evtchn_callback+0x9/0x10

[  133.692189]  ? check_events+0x12/0x20

[  133.694227]  ? xen_restore_fl_direct+0x1f/0x20

[  133.696878]  ? _raw_spin_unlock_irqrestore+0x14/0x20

[  133.699894]  ? cp_rx_poll+0x427/0x4d0 [8139cp]

[  133.702421]  ? net_rx_action+0x171/0x3a0

[  133.704693]  ? __do_softirq+0x11e/0x295

[  133.706672]  ? run_ksoftirqd+0x1e/0x40

[  133.708867]  ? smpboot_thread_fn+0x13c/0x160

[  133.711320]  ? kthread+0x115/0x120

[  133.713350]  ? sort_range+0x20/0x20

[  133.715421]  ? __kthread_cancel_work+0x80/0x80

[  133.717969]  ? ret_from_fork+0x35/0x40

[  133.720238] Kernel Offset: disabled



I'll try to solve it ahead.



On Mon, Nov 5, 2018 at 4:28 PM Wei Liu <wei.liu2@citrix.com> wrote:

> On Mon, Nov 05, 2018 at 04:26:03PM +0530, Rishi wrote:
> > Nothing is getting logged. I suspect its because the kernel just goes in
> > halt state.
> >
> > /var/log/kern.log /var/log/messages /var/log/xen/hypervisor.log
> >
> > All of these files have nothing appended.
> >
> > I can fetch info from kernel debugfs, if you point me to required area,
> >
> >
> > The dom0 kernel works fine without spilt of SMP code or with "nosmp"
> option
> > at xen command line.
>
> I think you will need to set up a serial console. I wouldn't expect to
> see anything in /var/log if the system has crashed.
>
> Without serial log there is nothing we can do to help.
>
> Wei.
>

[-- Attachment #1.2: Type: text/html, Size: 21024 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Dom0 kernel 4.14 with SMP randomly crashing

[Wei Liu]

On Mon, Nov 05, 2018 at 04:58:35PM +0530, Rishi wrote:
> Alright, I got the serial console and following is the crash log. Thank you
> for pointing that out.
> 
> [  133.594852] watchdog: BUG: soft lockup - CPU#2 stuck for 22s!
> [ksoftirqd/2:22]
> 
> [  133.599232] Kernel panic - not syncing: softlockup: hung tasks
> 
> [  133.602275] CPU: 2 PID: 22 Comm: ksoftirqd/2 Tainted: G
> L    4.19.1
> #1
> 
> [  133.606620] Hardware name: Xen HVM domU, BIOS 4.4.1-xs132257 12/12/2016

Is this serial log from the host? It says it is running as a HVM DomU.
Maybe you have mistaken guest serial log with host serial log?

This indicates your machine runs XenServer, which has its own patch
queues on top of upstream Xen. You may also want to report to xs-devel
mailing list.

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Dom0 kernel 4.14 with SMP randomly crashing

[Rishi]

[-- Attachment #1.1: Type: text/plain, Size: 950 bytes --]

Yes, I'm running it in a HVM domU for development purpose.

On Mon, Nov 5, 2018 at 5:11 PM Wei Liu <wei.liu2@citrix.com> wrote:

> On Mon, Nov 05, 2018 at 04:58:35PM +0530, Rishi wrote:
> > Alright, I got the serial console and following is the crash log. Thank
> you
> > for pointing that out.
> >
> > [  133.594852] watchdog: BUG: soft lockup - CPU#2 stuck for 22s!
> > [ksoftirqd/2:22]
> >
> > [  133.599232] Kernel panic - not syncing: softlockup: hung tasks
> >
> > [  133.602275] CPU: 2 PID: 22 Comm: ksoftirqd/2 Tainted: G
> > L    4.19.1
> > #1
> >
> > [  133.606620] Hardware name: Xen HVM domU, BIOS 4.4.1-xs132257
> 12/12/2016
>
> Is this serial log from the host? It says it is running as a HVM DomU.
> Maybe you have mistaken guest serial log with host serial log?
>
> This indicates your machine runs XenServer, which has its own patch
> queues on top of upstream Xen. You may also want to report to xs-devel
> mailing list.
>
> Wei.
>

[-- Attachment #1.2: Type: text/html, Size: 1315 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Dom0 kernel 4.14 with SMP randomly crashing

[Wei Liu]

On Mon, Nov 05, 2018 at 05:18:43PM +0530, Rishi wrote:
> Yes, I'm running it in a HVM domU for development purpose.

What is your exact setup?

Wei.

> 
> On Mon, Nov 5, 2018 at 5:11 PM Wei Liu <wei.liu2@citrix.com> wrote:
> 
> > On Mon, Nov 05, 2018 at 04:58:35PM +0530, Rishi wrote:
> > > Alright, I got the serial console and following is the crash log. Thank
> > you
> > > for pointing that out.
> > >
> > > [  133.594852] watchdog: BUG: soft lockup - CPU#2 stuck for 22s!
> > > [ksoftirqd/2:22]
> > >
> > > [  133.599232] Kernel panic - not syncing: softlockup: hung tasks
> > >
> > > [  133.602275] CPU: 2 PID: 22 Comm: ksoftirqd/2 Tainted: G
> > > L    4.19.1
> > > #1
> > >
> > > [  133.606620] Hardware name: Xen HVM domU, BIOS 4.4.1-xs132257
> > 12/12/2016
> >
> > Is this serial log from the host? It says it is running as a HVM DomU.
> > Maybe you have mistaken guest serial log with host serial log?
> >
> > This indicates your machine runs XenServer, which has its own patch
> > queues on top of upstream Xen. You may also want to report to xs-devel
> > mailing list.
> >
> > Wei.
> >

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Dom0 kernel 4.14 with SMP randomly crashing

[Rishi]

[-- Attachment #1.1: Type: text/plain, Size: 1481 bytes --]

I'm using a XenServer Host and XCP-NG on it as HVM. I used xencons=tty
console=ttyS0 on XCP-NG dom0 kernel line, to obtain serial console.
I'm working on to build a more recent dom0 kernel for improved support of
Ceph in XenServer/XCP-NG.



On Mon, Nov 5, 2018 at 5:28 PM Wei Liu <wei.liu2@citrix.com> wrote:

> On Mon, Nov 05, 2018 at 05:18:43PM +0530, Rishi wrote:
> > Yes, I'm running it in a HVM domU for development purpose.
>
> What is your exact setup?
>
> Wei.
>
> >
> > On Mon, Nov 5, 2018 at 5:11 PM Wei Liu <wei.liu2@citrix.com> wrote:
> >
> > > On Mon, Nov 05, 2018 at 04:58:35PM +0530, Rishi wrote:
> > > > Alright, I got the serial console and following is the crash log.
> Thank
> > > you
> > > > for pointing that out.
> > > >
> > > > [  133.594852] watchdog: BUG: soft lockup - CPU#2 stuck for 22s!
> > > > [ksoftirqd/2:22]
> > > >
> > > > [  133.599232] Kernel panic - not syncing: softlockup: hung tasks
> > > >
> > > > [  133.602275] CPU: 2 PID: 22 Comm: ksoftirqd/2 Tainted: G
> > > > L    4.19.1
> > > > #1
> > > >
> > > > [  133.606620] Hardware name: Xen HVM domU, BIOS 4.4.1-xs132257
> > > 12/12/2016
> > >
> > > Is this serial log from the host? It says it is running as a HVM DomU.
> > > Maybe you have mistaken guest serial log with host serial log?
> > >
> > > This indicates your machine runs XenServer, which has its own patch
> > > queues on top of upstream Xen. You may also want to report to xs-devel
> > > mailing list.
> > >
> > > Wei.
> > >
>

[-- Attachment #1.2: Type: text/html, Size: 2273 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Dom0 kernel 4.14 with SMP randomly crashing

[Wei Liu]

I forgot to say: please don't top-post.

On Mon, Nov 05, 2018 at 06:00:10PM +0530, Rishi wrote:
> I'm using a XenServer Host and XCP-NG on it as HVM. I used xencons=tty
> console=ttyS0 on XCP-NG dom0 kernel line, to obtain serial console.
> I'm working on to build a more recent dom0 kernel for improved support of
> Ceph in XenServer/XCP-NG.

This is an interesting setup. I don't think you can expect to just drop
in a new kernel to XenServer/XCP-NG and then it works flawlessly. What
did you do to the patch queue XenServer carries for 4.4?

Also, have you got a working baseline? I.e. did the stock 4.4 kernel
work?

Wei.

> 
> 
> 
> On Mon, Nov 5, 2018 at 5:28 PM Wei Liu <wei.liu2@citrix.com> wrote:
> 
> > On Mon, Nov 05, 2018 at 05:18:43PM +0530, Rishi wrote:
> > > Yes, I'm running it in a HVM domU for development purpose.
> >
> > What is your exact setup?
> >
> > Wei.
> >
> > >
> > > On Mon, Nov 5, 2018 at 5:11 PM Wei Liu <wei.liu2@citrix.com> wrote:
> > >
> > > > On Mon, Nov 05, 2018 at 04:58:35PM +0530, Rishi wrote:
> > > > > Alright, I got the serial console and following is the crash log.
> > Thank
> > > > you
> > > > > for pointing that out.
> > > > >
> > > > > [  133.594852] watchdog: BUG: soft lockup - CPU#2 stuck for 22s!
> > > > > [ksoftirqd/2:22]
> > > > >
> > > > > [  133.599232] Kernel panic - not syncing: softlockup: hung tasks
> > > > >
> > > > > [  133.602275] CPU: 2 PID: 22 Comm: ksoftirqd/2 Tainted: G
> > > > > L    4.19.1
> > > > > #1
> > > > >
> > > > > [  133.606620] Hardware name: Xen HVM domU, BIOS 4.4.1-xs132257
> > > > 12/12/2016
> > > >
> > > > Is this serial log from the host? It says it is running as a HVM DomU.
> > > > Maybe you have mistaken guest serial log with host serial log?
> > > >
> > > > This indicates your machine runs XenServer, which has its own patch
> > > > queues on top of upstream Xen. You may also want to report to xs-devel
> > > > mailing list.
> > > >
> > > > Wei.
> > > >
> >

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Dom0 kernel 4.14 with SMP randomly crashing

[Rishi]

[-- Attachment #1.1: Type: text/plain, Size: 2731 bytes --]

Yes, I'm taking out patches from 4.4 and actually do have a working 4.9
kernel along with blktap. Tested networking and disk IO in it.

There are roughly 415 patches to 4.4 out of which some ~210+ are already
applied in 4.9 and ~220+ are already applied in 4.14. I dont have numbers
for 4.19 yet.

Essentially I'm down to single digit number of patches atm to have a
working setup for kernel 4.9. I know there would be mishaps since I'm not
applying all patches but my experiment is to see how close can we stay near
mainline kernel + what can be the patches that kernel.org can accept.



On Mon, Nov 5, 2018 at 6:19 PM Wei Liu <wei.liu2@citrix.com> wrote:

> I forgot to say: please don't top-post.
>
> On Mon, Nov 05, 2018 at 06:00:10PM +0530, Rishi wrote:
> > I'm using a XenServer Host and XCP-NG on it as HVM. I used xencons=tty
> > console=ttyS0 on XCP-NG dom0 kernel line, to obtain serial console.
> > I'm working on to build a more recent dom0 kernel for improved support of
> > Ceph in XenServer/XCP-NG.
>
> This is an interesting setup. I don't think you can expect to just drop
> in a new kernel to XenServer/XCP-NG and then it works flawlessly. What
> did you do to the patch queue XenServer carries for 4.4?
>
> Also, have you got a working baseline? I.e. did the stock 4.4 kernel
> work?
>
> Wei.
>
> >
> >
> >
> > On Mon, Nov 5, 2018 at 5:28 PM Wei Liu <wei.liu2@citrix.com> wrote:
> >
> > > On Mon, Nov 05, 2018 at 05:18:43PM +0530, Rishi wrote:
> > > > Yes, I'm running it in a HVM domU for development purpose.
> > >
> > > What is your exact setup?
> > >
> > > Wei.
> > >
> > > >
> > > > On Mon, Nov 5, 2018 at 5:11 PM Wei Liu <wei.liu2@citrix.com> wrote:
> > > >
> > > > > On Mon, Nov 05, 2018 at 04:58:35PM +0530, Rishi wrote:
> > > > > > Alright, I got the serial console and following is the crash log.
> > > Thank
> > > > > you
> > > > > > for pointing that out.
> > > > > >
> > > > > > [  133.594852] watchdog: BUG: soft lockup - CPU#2 stuck for 22s!
> > > > > > [ksoftirqd/2:22]
> > > > > >
> > > > > > [  133.599232] Kernel panic - not syncing: softlockup: hung tasks
> > > > > >
> > > > > > [  133.602275] CPU: 2 PID: 22 Comm: ksoftirqd/2 Tainted: G
> > > > > > L    4.19.1
> > > > > > #1
> > > > > >
> > > > > > [  133.606620] Hardware name: Xen HVM domU, BIOS 4.4.1-xs132257
> > > > > 12/12/2016
> > > > >
> > > > > Is this serial log from the host? It says it is running as a HVM
> DomU.
> > > > > Maybe you have mistaken guest serial log with host serial log?
> > > > >
> > > > > This indicates your machine runs XenServer, which has its own patch
> > > > > queues on top of upstream Xen. You may also want to report to
> xs-devel
> > > > > mailing list.
> > > > >
> > > > > Wei.
> > > > >
> > >
>

[-- Attachment #1.2: Type: text/html, Size: 3854 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Dom0 kernel 4.14 with SMP randomly crashing

[Rishi]

[-- Attachment #1.1: Type: text/plain, Size: 2952 bytes --]

On Mon, Nov 5, 2018 at 6:29 PM Rishi <2rushikeshj@gmail.com> wrote:

> Yes, I'm taking out patches from 4.4 and actually do have a working 4.9
> kernel along with blktap. Tested networking and disk IO in it.
>
> There are roughly 415 patches to 4.4 out of which some ~210+ are already
> applied in 4.9 and ~220+ are already applied in 4.14. I dont have numbers
> for 4.19 yet.
>
> Essentially I'm down to single digit number of patches atm to have a
> working setup for kernel 4.9. I know there would be mishaps since I'm not
> applying all patches but my experiment is to see how close can we stay near
> mainline kernel + what can be the patches that kernel.org can accept.
>
>
>
> On Mon, Nov 5, 2018 at 6:19 PM Wei Liu <wei.liu2@citrix.com> wrote:
>
>> I forgot to say: please don't top-post.
>>
>> On Mon, Nov 05, 2018 at 06:00:10PM +0530, Rishi wrote:
>> > I'm using a XenServer Host and XCP-NG on it as HVM. I used xencons=tty
>> > console=ttyS0 on XCP-NG dom0 kernel line, to obtain serial console.
>> > I'm working on to build a more recent dom0 kernel for improved support
>> of
>> > Ceph in XenServer/XCP-NG.
>>
>> This is an interesting setup. I don't think you can expect to just drop
>> in a new kernel to XenServer/XCP-NG and then it works flawlessly. What
>> did you do to the patch queue XenServer carries for 4.4?
>>
>> Also, have you got a working baseline? I.e. did the stock 4.4 kernel
>> work?
>>
>> Wei.
>>
>> >
>> >
>> >
>> > On Mon, Nov 5, 2018 at 5:28 PM Wei Liu <wei.liu2@citrix.com> wrote:
>> >
>> > > On Mon, Nov 05, 2018 at 05:18:43PM +0530, Rishi wrote:
>> > > > Yes, I'm running it in a HVM domU for development purpose.
>> > >
>> > > What is your exact setup?
>> > >
>> > > Wei.
>> > >
>> > > >
>> > > > On Mon, Nov 5, 2018 at 5:11 PM Wei Liu <wei.liu2@citrix.com> wrote:
>> > > >
>> > > > > On Mon, Nov 05, 2018 at 04:58:35PM +0530, Rishi wrote:
>> > > > > > Alright, I got the serial console and following is the crash
>> log.
>> > > Thank
>> > > > > you
>> > > > > > for pointing that out.
>> > > > > >
>> > > > > > [  133.594852] watchdog: BUG: soft lockup - CPU#2 stuck for 22s!
>> > > > > > [ksoftirqd/2:22]
>> > > > > >
>> > > > > > [  133.599232] Kernel panic - not syncing: softlockup: hung
>> tasks
>> > > > > >
>> > > > > > [  133.602275] CPU: 2 PID: 22 Comm: ksoftirqd/2 Tainted: G
>> > > > > > L    4.19.1
>> > > > > > #1
>> > > > > >
>> > > > > > [  133.606620] Hardware name: Xen HVM domU, BIOS 4.4.1-xs132257
>> > > > > 12/12/2016
>> > > > >
>> > > > > Is this serial log from the host? It says it is running as a HVM
>> DomU.
>> > > > > Maybe you have mistaken guest serial log with host serial log?
>> > > > >
>> > > > > This indicates your machine runs XenServer, which has its own
>> patch
>> > > > > queues on top of upstream Xen. You may also want to report to
>> xs-devel
>> > > > > mailing list.
>> > > > >
>> > > > > Wei.
>> > > > >
>> > >
>>
>

Sorry, I'll take care of top post from onwards.

[-- Attachment #1.2: Type: text/html, Size: 4272 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Dom0 kernel 4.14 with SMP randomly crashing

[Rishi]

[-- Attachment #1.1: Type: text/plain, Size: 6310 bytes --]

On Mon, Nov 5, 2018 at 6:32 PM Rishi <2rushikeshj@gmail.com> wrote:

>
>
> On Mon, Nov 5, 2018 at 6:29 PM Rishi <2rushikeshj@gmail.com> wrote:
>
>> Yes, I'm taking out patches from 4.4 and actually do have a working 4.9
>> kernel along with blktap. Tested networking and disk IO in it.
>>
>> There are roughly 415 patches to 4.4 out of which some ~210+ are already
>> applied in 4.9 and ~220+ are already applied in 4.14. I dont have numbers
>> for 4.19 yet.
>>
>> Essentially I'm down to single digit number of patches atm to have a
>> working setup for kernel 4.9. I know there would be mishaps since I'm not
>> applying all patches but my experiment is to see how close can we stay near
>> mainline kernel + what can be the patches that kernel.org can accept.
>>
>>
>>
>> On Mon, Nov 5, 2018 at 6:19 PM Wei Liu <wei.liu2@citrix.com> wrote:
>>
>>> I forgot to say: please don't top-post.
>>>
>>> On Mon, Nov 05, 2018 at 06:00:10PM +0530, Rishi wrote:
>>> > I'm using a XenServer Host and XCP-NG on it as HVM. I used xencons=tty
>>> > console=ttyS0 on XCP-NG dom0 kernel line, to obtain serial console.
>>> > I'm working on to build a more recent dom0 kernel for improved support
>>> of
>>> > Ceph in XenServer/XCP-NG.
>>>
>>> This is an interesting setup. I don't think you can expect to just drop
>>> in a new kernel to XenServer/XCP-NG and then it works flawlessly. What
>>> did you do to the patch queue XenServer carries for 4.4?
>>>
>>> Also, have you got a working baseline? I.e. did the stock 4.4 kernel
>>> work?
>>>
>>> Wei.
>>>
>>> >
>>> >
>>> >
>>> > On Mon, Nov 5, 2018 at 5:28 PM Wei Liu <wei.liu2@citrix.com> wrote:
>>> >
>>> > > On Mon, Nov 05, 2018 at 05:18:43PM +0530, Rishi wrote:
>>> > > > Yes, I'm running it in a HVM domU for development purpose.
>>> > >
>>> > > What is your exact setup?
>>> > >
>>> > > Wei.
>>> > >
>>> > > >
>>> > > > On Mon, Nov 5, 2018 at 5:11 PM Wei Liu <wei.liu2@citrix.com>
>>> wrote:
>>> > > >
>>> > > > > On Mon, Nov 05, 2018 at 04:58:35PM +0530, Rishi wrote:
>>> > > > > > Alright, I got the serial console and following is the crash
>>> log.
>>> > > Thank
>>> > > > > you
>>> > > > > > for pointing that out.
>>> > > > > >
>>> > > > > > [  133.594852] watchdog: BUG: soft lockup - CPU#2 stuck for
>>> 22s!
>>> > > > > > [ksoftirqd/2:22]
>>> > > > > >
>>> > > > > > [  133.599232] Kernel panic - not syncing: softlockup: hung
>>> tasks
>>> > > > > >
>>> > > > > > [  133.602275] CPU: 2 PID: 22 Comm: ksoftirqd/2 Tainted: G
>>> > > > > > L    4.19.1
>>> > > > > > #1
>>> > > > > >
>>> > > > > > [  133.606620] Hardware name: Xen HVM domU, BIOS 4.4.1-xs132257
>>> > > > > 12/12/2016
>>> > > > >
>>> > > > > Is this serial log from the host? It says it is running as a HVM
>>> DomU.
>>> > > > > Maybe you have mistaken guest serial log with host serial log?
>>> > > > >
>>> > > > > This indicates your machine runs XenServer, which has its own
>>> patch
>>> > > > > queues on top of upstream Xen. You may also want to report to
>>> xs-devel
>>> > > > > mailing list.
>>> > > > >
>>> > > > > Wei.
>>> > > > >
>>> > >
>>>
>>
>
> Sorry, I'll take care of top post from onwards.
>

So after knowing the stack trace, it appears that the CPU was getting stuck
for xen_hypercall_xen_version

watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [swapper/0:0]


[30569.582740] watchdog: BUG: soft lockup - CPU#0 stuck for 23s!
[swapper/0:0]

[30569.588186] Kernel panic - not syncing: softlockup: hung tasks

[30569.591307] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G             L    4.19.1
#1

[30569.595110] Hardware name: Xen HVM domU, BIOS 4.4.1-xs132257 12/12/2016

[30569.598356] Call Trace:

[30569.599597]  <IRQ>

[30569.600920]  dump_stack+0x5a/0x73

[30569.602998]  panic+0xe8/0x249

[30569.604806]  watchdog_timer_fn+0x200/0x230

[30569.607029]  ? softlockup_fn+0x40/0x40

[30569.609246]  __hrtimer_run_queues+0x133/0x270

[30569.611712]  hrtimer_interrupt+0xfb/0x260

[30569.613800]  xen_timer_interrupt+0x1b/0x30

[30569.616972]  __handle_irq_event_percpu+0x69/0x1a0

[30569.619831]  handle_irq_event_percpu+0x30/0x70

[30569.622382]  handle_percpu_irq+0x34/0x50

[30569.625048]  generic_handle_irq+0x1e/0x30

[30569.627216]  __evtchn_fifo_handle_events+0x163/0x1a0

[30569.629955]  __xen_evtchn_do_upcall+0x41/0x70

[30569.632612]  xen_evtchn_do_upcall+0x27/0x50

[30569.635136]  xen_do_hypervisor_callback+0x29/0x40

[30569.638181] RIP: e030:xen_hypercall_xen_version+0xa/0x20

[30569.641302] Code: 51 41 53 b8 10 00 00 00 0f 05 41 5b 59 c3 cc cc cc cc
cc cc cc cc cc cc cc cc cc cc cc cc cc cc 51 41 53 b8 11 00 00 00 0f 05
<41> 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc

[30569.651998] RSP: e02b:ffff8800b6203e10 EFLAGS: 00000246

[30569.655077] RAX: 0000000000040007 RBX: ffff8800ae41a898 RCX:
ffffffff8100122a

[30569.659226] RDX: ffffc900400080ff RSI: 0000000000000000 RDI:
0000000000000000

[30569.663480] RBP: ffff8800ae41a890 R08: 0000000000000000 R09:
0000000000000000

[30569.667943] R10: 0000000000000000 R11: 0000000000000246 R12:
0000000080000600

[30569.672057] R13: 000000000000001d R14: 00000000000001d0 R15:
000000000000001d

[30569.675911]  ? xen_hypercall_xen_version+0xa/0x20

[30569.678470]  ? xen_force_evtchn_callback+0x9/0x10

[30569.681495]  ? check_events+0x12/0x20

[30569.683738]  ? xen_restore_fl_direct+0x1f/0x20

[30569.686632]  ? _raw_spin_unlock_irqrestore+0x14/0x20

[30569.689166]  ? cp_rx_poll+0x427/0x4d0 [8139cp]

[30569.691519]  ? net_rx_action+0x171/0x3a0

[30569.694219]  ? __do_softirq+0x11e/0x295

[30569.696442]  ? irq_exit+0x62/0xb0

[30569.698251]  ? xen_evtchn_do_upcall+0x2c/0x50

[30569.701037]  ? xen_do_hypervisor_callback+0x29/0x40

[30569.704439]  </IRQ>

[30569.705731]  ? xen_hypercall_sched_op+0xa/0x20

[30569.708766]  ? xen_hypercall_sched_op+0xa/0x20

[30569.711344]  ? xen_safe_halt+0xc/0x20

[30569.713353]  ? default_idle+0x80/0x140

[30569.715345]  ? do_idle+0x13a/0x250

[30569.717216]  ? cpu_startup_entry+0x6f/0x80

[30569.719511]  ? start_kernel+0x4f6/0x516

[30569.721681]  ? set_init_arg+0x57/0x57

[30569.723985]  ? xen_start_kernel+0x575/0x57f

[30569.726453] Kernel Offset: disabled



So I wrote a kernel module to try to call the function
xen_hypercall_xen_version through it, it could successfully run and return
the version.


What else should I be checking for?

[-- Attachment #1.2: Type: text/html, Size: 26059 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Dom0 kernel 4.14 with SMP randomly crashing

[Wei Liu]

On Tue, Nov 06, 2018 at 03:31:31PM +0530, Rishi wrote:
> 
> So after knowing the stack trace, it appears that the CPU was getting stuck
> for xen_hypercall_xen_version

That hypercall is used when a PV kernel (re-)enables interrupts. See
xen_irq_enable. The purpose is to force the kernel to switch to
hypervisor.

> 
> watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [swapper/0:0]
> 
> 
> [30569.582740] watchdog: BUG: soft lockup - CPU#0 stuck for 23s!
> [swapper/0:0]
> 
> [30569.588186] Kernel panic - not syncing: softlockup: hung tasks
> 
> [30569.591307] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G             L    4.19.1
> #1
> 
> [30569.595110] Hardware name: Xen HVM domU, BIOS 4.4.1-xs132257 12/12/2016
> 
> [30569.598356] Call Trace:
> 
> [30569.599597]  <IRQ>
> 
> [30569.600920]  dump_stack+0x5a/0x73
> 
> [30569.602998]  panic+0xe8/0x249
> 
> [30569.604806]  watchdog_timer_fn+0x200/0x230
> 
> [30569.607029]  ? softlockup_fn+0x40/0x40
> 
> [30569.609246]  __hrtimer_run_queues+0x133/0x270
> 
> [30569.611712]  hrtimer_interrupt+0xfb/0x260
> 
> [30569.613800]  xen_timer_interrupt+0x1b/0x30
> 
> [30569.616972]  __handle_irq_event_percpu+0x69/0x1a0
> 
> [30569.619831]  handle_irq_event_percpu+0x30/0x70
> 
> [30569.622382]  handle_percpu_irq+0x34/0x50
> 
> [30569.625048]  generic_handle_irq+0x1e/0x30
> 
> [30569.627216]  __evtchn_fifo_handle_events+0x163/0x1a0
> 
> [30569.629955]  __xen_evtchn_do_upcall+0x41/0x70
> 
> [30569.632612]  xen_evtchn_do_upcall+0x27/0x50
> 
> [30569.635136]  xen_do_hypervisor_callback+0x29/0x40
> 
> [30569.638181] RIP: e030:xen_hypercall_xen_version+0xa/0x20

What is the asm code for this RIP?


Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Dom0 kernel 4.14 with SMP randomly crashing

[Rishi]

[-- Attachment #1.1: Type: text/plain, Size: 2394 bytes --]

On Tue, Nov 6, 2018 at 5:47 PM Wei Liu <wei.liu2@citrix.com> wrote:

> On Tue, Nov 06, 2018 at 03:31:31PM +0530, Rishi wrote:
> >
> > So after knowing the stack trace, it appears that the CPU was getting
> stuck
> > for xen_hypercall_xen_version
>
> That hypercall is used when a PV kernel (re-)enables interrupts. See
> xen_irq_enable. The purpose is to force the kernel to switch to
> hypervisor.
>
> >
> > watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [swapper/0:0]
> >
> >
> > [30569.582740] watchdog: BUG: soft lockup - CPU#0 stuck for 23s!
> > [swapper/0:0]
> >
> > [30569.588186] Kernel panic - not syncing: softlockup: hung tasks
> >
> > [30569.591307] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G             L
> 4.19.1
> > #1
> >
> > [30569.595110] Hardware name: Xen HVM domU, BIOS 4.4.1-xs132257
> 12/12/2016
> >
> > [30569.598356] Call Trace:
> >
> > [30569.599597]  <IRQ>
> >
> > [30569.600920]  dump_stack+0x5a/0x73
> >
> > [30569.602998]  panic+0xe8/0x249
> >
> > [30569.604806]  watchdog_timer_fn+0x200/0x230
> >
> > [30569.607029]  ? softlockup_fn+0x40/0x40
> >
> > [30569.609246]  __hrtimer_run_queues+0x133/0x270
> >
> > [30569.611712]  hrtimer_interrupt+0xfb/0x260
> >
> > [30569.613800]  xen_timer_interrupt+0x1b/0x30
> >
> > [30569.616972]  __handle_irq_event_percpu+0x69/0x1a0
> >
> > [30569.619831]  handle_irq_event_percpu+0x30/0x70
> >
> > [30569.622382]  handle_percpu_irq+0x34/0x50
> >
> > [30569.625048]  generic_handle_irq+0x1e/0x30
> >
> > [30569.627216]  __evtchn_fifo_handle_events+0x163/0x1a0
> >
> > [30569.629955]  __xen_evtchn_do_upcall+0x41/0x70
> >
> > [30569.632612]  xen_evtchn_do_upcall+0x27/0x50
> >
> > [30569.635136]  xen_do_hypervisor_callback+0x29/0x40
> >
> > [30569.638181] RIP: e030:xen_hypercall_xen_version+0xa/0x20
>
> What is the asm code for this RIP?
>
>
> Wei.
>

The issue of crash is getting resolved with appending "noirqbalance" at xen
command line. This way all dom0 cpus are available but not irq balanced at
xen.

Even though I'm running irqbalance service in dom0 the irqs seems to be not
moving. <- this is dom0 perspective, I do not know yet, if it follows Xen
irq.

I tried objdump, while I have  have the function in out but there is no asm
code of it. Its just "..."

ffffffff81001220 <xen_hypercall_xen_version>:

        ...


ffffffff81001240 <xen_hypercall_console_io>:

        ...

All "hypercalls" appear similarly.

[-- Attachment #1.2: Type: text/html, Size: 4831 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Dom0 kernel 4.14 with SMP randomly crashing

[Rishi]

[-- Attachment #1.1: Type: text/plain, Size: 2995 bytes --]

On Tue, Nov 6, 2018 at 10:41 PM Rishi <2rushikeshj@gmail.com> wrote:

>
>
> On Tue, Nov 6, 2018 at 5:47 PM Wei Liu <wei.liu2@citrix.com> wrote:
>
>> On Tue, Nov 06, 2018 at 03:31:31PM +0530, Rishi wrote:
>> >
>> > So after knowing the stack trace, it appears that the CPU was getting
>> stuck
>> > for xen_hypercall_xen_version
>>
>> That hypercall is used when a PV kernel (re-)enables interrupts. See
>> xen_irq_enable. The purpose is to force the kernel to switch to
>> hypervisor.
>>
>> >
>> > watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [swapper/0:0]
>> >
>> >
>> > [30569.582740] watchdog: BUG: soft lockup - CPU#0 stuck for 23s!
>> > [swapper/0:0]
>> >
>> > [30569.588186] Kernel panic - not syncing: softlockup: hung tasks
>> >
>> > [30569.591307] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G             L
>>   4.19.1
>> > #1
>> >
>> > [30569.595110] Hardware name: Xen HVM domU, BIOS 4.4.1-xs132257
>> 12/12/2016
>> >
>> > [30569.598356] Call Trace:
>> >
>> > [30569.599597]  <IRQ>
>> >
>> > [30569.600920]  dump_stack+0x5a/0x73
>> >
>> > [30569.602998]  panic+0xe8/0x249
>> >
>> > [30569.604806]  watchdog_timer_fn+0x200/0x230
>> >
>> > [30569.607029]  ? softlockup_fn+0x40/0x40
>> >
>> > [30569.609246]  __hrtimer_run_queues+0x133/0x270
>> >
>> > [30569.611712]  hrtimer_interrupt+0xfb/0x260
>> >
>> > [30569.613800]  xen_timer_interrupt+0x1b/0x30
>> >
>> > [30569.616972]  __handle_irq_event_percpu+0x69/0x1a0
>> >
>> > [30569.619831]  handle_irq_event_percpu+0x30/0x70
>> >
>> > [30569.622382]  handle_percpu_irq+0x34/0x50
>> >
>> > [30569.625048]  generic_handle_irq+0x1e/0x30
>> >
>> > [30569.627216]  __evtchn_fifo_handle_events+0x163/0x1a0
>> >
>> > [30569.629955]  __xen_evtchn_do_upcall+0x41/0x70
>> >
>> > [30569.632612]  xen_evtchn_do_upcall+0x27/0x50
>> >
>> > [30569.635136]  xen_do_hypervisor_callback+0x29/0x40
>> >
>> > [30569.638181] RIP: e030:xen_hypercall_xen_version+0xa/0x20
>>
>> What is the asm code for this RIP?
>>
>>
>> Wei.
>>
>
> The issue of crash is getting resolved with appending "noirqbalance" at
> xen command line. This way all dom0 cpus are available but not irq balanced
> at xen.
>
> Even though I'm running irqbalance service in dom0 the irqs seems to be
> not moving. <- this is dom0 perspective, I do not know yet, if it follows
> Xen irq.
>
> I tried objdump, while I have  have the function in out but there is no
> asm code of it. Its just "..."
>
> ffffffff81001220 <xen_hypercall_xen_version>:
>
>         ...
>
>
> ffffffff81001240 <xen_hypercall_console_io>:
>
>         ...
>
> All "hypercalls" appear similarly.
>

How frequent can be that hypercall/xen_irq_enable()? Like n/s or once a
while?
During my tests, the system runs stable unless I'm downloading a large
file. Files around a GB size are getting downloaded without crash, but
system crash comes when its above it. I'm using a 2.1GB file & wget to
download.

Is there a way I can simulate PV kernel (re-)enable of interrupt using a
kernel module with a controlled fashion?

[-- Attachment #1.2: Type: text/html, Size: 5926 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Dom0 kernel 4.14 with SMP randomly crashing

[Rishi]

[-- Attachment #1.1: Type: text/plain, Size: 4548 bytes --]

On Wed, Nov 7, 2018 at 12:16 AM Rishi <2rushikeshj@gmail.com> wrote:

>
>
> On Tue, Nov 6, 2018 at 10:41 PM Rishi <2rushikeshj@gmail.com> wrote:
>
>>
>>
>> On Tue, Nov 6, 2018 at 5:47 PM Wei Liu <wei.liu2@citrix.com> wrote:
>>
>>> On Tue, Nov 06, 2018 at 03:31:31PM +0530, Rishi wrote:
>>> >
>>> > So after knowing the stack trace, it appears that the CPU was getting
>>> stuck
>>> > for xen_hypercall_xen_version
>>>
>>> That hypercall is used when a PV kernel (re-)enables interrupts. See
>>> xen_irq_enable. The purpose is to force the kernel to switch to
>>> hypervisor.
>>>
>>> >
>>> > watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [swapper/0:0]
>>> >
>>> >
>>> > [30569.582740] watchdog: BUG: soft lockup - CPU#0 stuck for 23s!
>>> > [swapper/0:0]
>>> >
>>> > [30569.588186] Kernel panic - not syncing: softlockup: hung tasks
>>> >
>>> > [30569.591307] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G             L
>>>   4.19.1
>>> > #1
>>> >
>>> > [30569.595110] Hardware name: Xen HVM domU, BIOS 4.4.1-xs132257
>>> 12/12/2016
>>> >
>>> > [30569.598356] Call Trace:
>>> >
>>> > [30569.599597]  <IRQ>
>>> >
>>> > [30569.600920]  dump_stack+0x5a/0x73
>>> >
>>> > [30569.602998]  panic+0xe8/0x249
>>> >
>>> > [30569.604806]  watchdog_timer_fn+0x200/0x230
>>> >
>>> > [30569.607029]  ? softlockup_fn+0x40/0x40
>>> >
>>> > [30569.609246]  __hrtimer_run_queues+0x133/0x270
>>> >
>>> > [30569.611712]  hrtimer_interrupt+0xfb/0x260
>>> >
>>> > [30569.613800]  xen_timer_interrupt+0x1b/0x30
>>> >
>>> > [30569.616972]  __handle_irq_event_percpu+0x69/0x1a0
>>> >
>>> > [30569.619831]  handle_irq_event_percpu+0x30/0x70
>>> >
>>> > [30569.622382]  handle_percpu_irq+0x34/0x50
>>> >
>>> > [30569.625048]  generic_handle_irq+0x1e/0x30
>>> >
>>> > [30569.627216]  __evtchn_fifo_handle_events+0x163/0x1a0
>>> >
>>> > [30569.629955]  __xen_evtchn_do_upcall+0x41/0x70
>>> >
>>> > [30569.632612]  xen_evtchn_do_upcall+0x27/0x50
>>> >
>>> > [30569.635136]  xen_do_hypervisor_callback+0x29/0x40
>>> >
>>> > [30569.638181] RIP: e030:xen_hypercall_xen_version+0xa/0x20
>>>
>>> What is the asm code for this RIP?
>>>
>>>
>>> Wei.
>>>
>>
>> The issue of crash is getting resolved with appending "noirqbalance" at
>> xen command line. This way all dom0 cpus are available but not irq balanced
>> at xen.
>>
>> Even though I'm running irqbalance service in dom0 the irqs seems to be
>> not moving. <- this is dom0 perspective, I do not know yet, if it follows
>> Xen irq.
>>
>> I tried objdump, while I have  have the function in out but there is no
>> asm code of it. Its just "..."
>>
>> ffffffff81001220 <xen_hypercall_xen_version>:
>>
>>         ...
>>
>>
>> ffffffff81001240 <xen_hypercall_console_io>:
>>
>>         ...
>>
>> All "hypercalls" appear similarly.
>>
>
> How frequent can be that hypercall/xen_irq_enable()? Like n/s or once a
> while?
> During my tests, the system runs stable unless I'm downloading a large
> file. Files around a GB size are getting downloaded without crash, but
> system crash comes when its above it. I'm using a 2.1GB file & wget to
> download.
>
> Is there a way I can simulate PV kernel (re-)enable of interrupt using a
> kernel module with a controlled fashion?
>

If this is on right track

ffffffff8101ab70 <xen_force_evtchn_callback>:

ffffffff8101ab70:       31 ff                   xor    %edi,%edi

ffffffff8101ab72:       31 f6                   xor    %esi,%esi

ffffffff8101ab74:       e8 a7 66 fe ff          callq  ffffffff81001220
<xen_hypercall_xen_version>

ffffffff8101ab79:       c3                      retq

ffffffff8101ab7a:       66 0f 1f 44 00 00       nopw   0x0(%rax,%rax,1)

It seems I'm hitting following code from xen_irq_enable

        barrier(); /* unmask then check (avoid races) */

        if (unlikely(vcpu->evtchn_upcall_pending))
                xen_force_evtchn_callback();

The code says unlikely yet, it is being called, And I got following
structure

struct vcpu_info {

        /*

         * 'evtchn_upcall_pending' is written non-zero by Xen to indicate

         * a pending notification for a particular VCPU. It is then cleared

         * by the guest OS /before/ checking for pending work, thus avoiding

         * a set-and-check race. Note that the mask is only accessed by Xen

         * on the CPU that is currently hosting the VCPU. This means that
the

         * pending and mask flags can be updated by the guest without
special

         * synchronisation (i.e., no need for the x86 LOCK prefix).


 Let me know if the thread is being spammed with such intermediates.

[-- Attachment #1.2: Type: text/html, Size: 15490 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel