All of lore.kernel.org
 help / color / mirror / Atom feed
From: "Michał Leszczyński" <michal.leszczynski@cert.pl>
To: xen-devel@lists.xenproject.org
Subject: BUG: credit=sched2 machine hang when using DRAKVUF
Date: Fri, 23 Oct 2020 00:59:19 +0200 (CEST)	[thread overview]
Message-ID: <157653679.6164.1603407559737.JavaMail.zimbra@nask.pl> (raw)

Hello,

when using DRAKVUF against a Windows 7 x64 DomU, the whole machine hangs after a few minutes.

The chance for a hang seems to be correlated with number of PCPUs, in this case we have 14 PCPUs and hang is very easily reproducible, while on other machines with 2-4 PCPUs it's very rare (but still occurring sometimes). The issue is observed with the default sched=credit2 and is no longer reproducible once sched=credit is set.


Enclosed: panic log from my Dom0.

Best regards,
Michał Leszczyński
CERT Polska


paź 22 12:20:50 hostname kernel: rcu: INFO: rcu_sched self-detected stall on CPU
paź 22 12:20:50 hostname kernel: rcu:         3-....: (21002 ticks this GP) idle=7e2/1/0x4000000000000002 softirq=61729/61729 fqs=10490
paź 22 12:20:50 hostname kernel: rcu:          (t=21003 jiffies g=36437 q=9406)
paź 22 12:20:50 hostname kernel: NMI backtrace for cpu 3
paź 22 12:20:50 hostname kernel: CPU: 3 PID: 4153 Comm: drakvuf Tainted: P           OEL    4.19.0-6-amd64 #1 Debian 4.19.67-2+deb10u2
paź 22 12:20:50 hostname kernel: Hardware name: Dell Inc. PowerEdge R640/08HT8T, BIOS 2.1.8 04/30/2019
paź 22 12:20:50 hostname kernel: Call Trace:
paź 22 12:20:50 hostname kernel:  <IRQ>
paź 22 12:20:50 hostname kernel:  dump_stack+0x5c/0x80
paź 22 12:20:50 hostname kernel:  nmi_cpu_backtrace.cold.4+0x13/0x50
paź 22 12:20:50 hostname kernel:  ? lapic_can_unplug_cpu.cold.29+0x3b/0x3b
paź 22 12:20:50 hostname kernel:  nmi_trigger_cpumask_backtrace+0xf9/0xfb
paź 22 12:20:50 hostname kernel:  rcu_dump_cpu_stacks+0x9b/0xcb
paź 22 12:20:50 hostname kernel:  rcu_check_callbacks.cold.81+0x1db/0x335
paź 22 12:20:50 hostname kernel:  ? tick_sched_do_timer+0x60/0x60
paź 22 12:20:50 hostname kernel:  update_process_times+0x28/0x60
paź 22 12:20:50 hostname kernel:  tick_sched_handle+0x22/0x60
paź 22 12:20:50 hostname kernel:  tick_sched_timer+0x37/0x70
paź 22 12:20:50 hostname kernel:  __hrtimer_run_queues+0x100/0x280
paź 22 12:20:50 hostname kernel:  hrtimer_interrupt+0x100/0x220
paź 22 12:20:50 hostname kernel:  xen_timer_interrupt+0x1e/0x30
paź 22 12:20:50 hostname kernel:  __handle_irq_event_percpu+0x46/0x190
paź 22 12:20:50 hostname kernel:  handle_irq_event_percpu+0x30/0x80
paź 22 12:20:50 hostname kernel:  handle_percpu_irq+0x40/0x60
paź 22 12:20:50 hostname kernel:  generic_handle_irq+0x27/0x30
paź 22 12:20:50 hostname kernel:  __evtchn_fifo_handle_events+0x17d/0x190
paź 22 12:20:50 hostname kernel:  __xen_evtchn_do_upcall+0x42/0x80
paź 22 12:20:50 hostname kernel:  xen_evtchn_do_upcall+0x27/0x40
paź 22 12:20:50 hostname kernel:  xen_do_hypervisor_callback+0x29/0x40
paź 22 12:20:50 hostname kernel:  </IRQ>
paź 22 12:20:50 hostname kernel: RIP: e030:smp_call_function_single+0xce/0xf0
paź 22 12:20:50 hostname kernel: Code: 8b 4c 24 38 65 48 33 0c 25 28 00 00 00 75 34 c9 c3 48 89 d1 48 89 f2 48 89 e6 e8 6d fe ff ff 8b 54 24 18 83 e2 01 74 0b f3 90 <8b> 54 24 18 8
3 e2 01 75 f5 eb ca 8b 05 b9 99 4d 01 85 c0 75 88 0f
paź 22 12:20:50 hostname kernel: RSP: e02b:ffffc9004713bd00 EFLAGS: 00000202
paź 22 12:20:50 hostname kernel: RAX: 0000000000000000 RBX: ffff888b0b6eea40 RCX: 0000000000000200
paź 22 12:20:50 hostname kernel: RDX: 0000000000000001 RSI: ffffffff8212e4a0 RDI: ffffffff81c2dec0
paź 22 12:20:50 hostname kernel: RBP: ffffc9004713bd50 R08: 0000000000000000 R09: ffff888c54052480
paź 22 12:20:50 hostname kernel: R10: ffff888c540524a8 R11: 0000000000000000 R12: ffffc9004713bd60
paź 22 12:20:50 hostname kernel: R13: 0000000080000000 R14: ffffffff80000000 R15: ffff888b0b6eeab0
paź 22 12:20:50 hostname kernel:  ? xen_pgd_alloc+0x110/0x110
paź 22 12:20:50 hostname kernel:  xen_exit_mmap+0xaa/0x100
paź 22 12:20:50 hostname kernel:  exit_mmap+0x64/0x180
paź 22 12:20:50 hostname kernel:  ? __raw_spin_unlock+0x5/0x10
paź 22 12:20:50 hostname kernel:  ? __handle_mm_fault+0x1090/0x1270
paź 22 12:20:50 hostname kernel:  ? _raw_spin_unlock_irqrestore+0x14/0x20
paź 22 12:20:50 hostname kernel:  ? exit_robust_list+0x5b/0x130
paź 22 12:20:50 hostname kernel:  mmput+0x54/0x130
paź 22 12:20:50 hostname kernel:  do_exit+0x290/0xb90
paź 22 12:20:50 hostname kernel:  ? handle_mm_fault+0xd6/0x200
paź 22 12:20:50 hostname kernel:  do_group_exit+0x3a/0xa0
paź 22 12:20:50 hostname kernel:  __x64_sys_exit_group+0x14/0x20
paź 22 12:20:50 hostname kernel:  do_syscall_64+0x53/0x110
paź 22 12:20:50 hostname kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xa9
paź 22 12:20:50 hostname kernel: RIP: 0033:0x7f98d23ec9d6
paź 22 12:20:50 hostname kernel: Code: Bad RIP value.
paź 22 12:20:50 hostname kernel: RSP: 002b:00007ffc4a0327f8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
paź 22 12:20:50 hostname kernel: RAX: ffffffffffffffda RBX: 00007f98d24dd760 RCX: 00007f98d23ec9d6
paź 22 12:20:50 hostname kernel: RDX: 0000000000000000 RSI: 000000000000003c RDI: 0000000000000000
paź 22 12:20:50 hostname kernel: RBP: 0000000000000000 R08: 00000000000000e7 R09: ffffffffffffff60
paź 22 12:20:50 hostname kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 00007f98d24dd760
paź 22 12:20:50 hostname kernel: R13: 000000000000005a R14: 00007f98d24e6428 R15: 0000000000000000


             reply	other threads:[~2020-10-22 22:59 UTC|newest]

Thread overview: 7+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-10-22 22:59 Michał Leszczyński [this message]
2020-10-23  4:47 ` BUG: credit=sched2 machine hang when using DRAKVUF Jürgen Groß
2020-10-23 23:36   ` Michał Leszczyński
2020-10-28  2:04   ` Michał Leszczyński
2020-10-28  7:45     ` Jan Beulich
2020-10-31  0:42       ` Dario Faggioli
2020-10-30 18:50     ` Dario Faggioli

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=157653679.6164.1603407559737.JavaMail.zimbra@nask.pl \
    --to=michal.leszczynski@cert.pl \
    --cc=xen-devel@lists.xenproject.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.