From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 046A7C63697 for ; Thu, 26 Nov 2020 17:21:07 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2F41B21D1A for ; Thu, 26 Nov 2020 17:21:06 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2F41B21D1A Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=antioche.eu.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.38967.71724 (Exim 4.92) (envelope-from ) id 1kiKwt-0007Vj-Ll; Thu, 26 Nov 2020 17:20:51 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 38967.71724; Thu, 26 Nov 2020 17:20:51 +0000 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1kiKwt-0007Vc-IT; Thu, 26 Nov 2020 17:20:51 +0000 Received: by outflank-mailman (input) for mailman id 38967; Thu, 26 Nov 2020 17:20:50 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1kiKws-0007VW-EC for xen-devel@lists.xenproject.org; Thu, 26 Nov 2020 17:20:50 +0000 Received: from chassiron.antioche.eu.org (unknown [2001:41d0:fe9d:1101::1]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id e960c967-7462-4c05-a27d-1635be85e312; Thu, 26 Nov 2020 17:20:47 +0000 (UTC) Received: from sandettie.soc.lip6.fr (82-64-3-41.subs.proxad.net [82.64.3.41]) by chassiron.antioche.eu.org (8.15.2/8.15.2) with ESMTPS id 0AQHKeOf012144 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=OK); Thu, 26 Nov 2020 18:20:41 +0100 (MET) Received: by sandettie.soc.lip6.fr (Postfix, from userid 373) id F12EC2E9CAC; Thu, 26 Nov 2020 18:20:34 +0100 (MET) Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1kiKws-0007VW-EC for xen-devel@lists.xenproject.org; Thu, 26 Nov 2020 17:20:50 +0000 X-Inumbo-ID: e960c967-7462-4c05-a27d-1635be85e312 Received: from chassiron.antioche.eu.org (unknown [2001:41d0:fe9d:1101::1]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id e960c967-7462-4c05-a27d-1635be85e312; Thu, 26 Nov 2020 17:20:47 +0000 (UTC) Received: from sandettie.soc.lip6.fr (82-64-3-41.subs.proxad.net [82.64.3.41]) by chassiron.antioche.eu.org (8.15.2/8.15.2) with ESMTPS id 0AQHKeOf012144 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=OK); Thu, 26 Nov 2020 18:20:41 +0100 (MET) Received: by sandettie.soc.lip6.fr (Postfix, from userid 373) id F12EC2E9CAC; Thu, 26 Nov 2020 18:20:34 +0100 (MET) Date: Thu, 26 Nov 2020 18:20:34 +0100 From: Manuel Bouyer To: Roger Pau =?iso-8859-1?Q?Monn=E9?= Cc: Jan Beulich , xen-devel@lists.xenproject.org Subject: Re: NetBSD dom0 PVH: hardware interrupts stalls Message-ID: <20201126172034.GA7642@antioche.eu.org> References: <6d6a77cf-58de-4e4d-ed75-e9365be060b7@suse.com> <20201124142713.GM2020@antioche.eu.org> <20201124150842.GN2020@antioche.eu.org> <20201124154917.l3jwa6w4ejumjuqw@Air-de-Roger> <20201124160914.GQ2020@antioche.eu.org> <20201126133444.r2oi24i3umh7shb3@Air-de-Roger> <20201126141608.GA4123@antioche.eu.org> <20201126142635.uzi643co3mxp5h42@Air-de-Roger> <20201126150937.jhbfp7iefkmtedx7@Air-de-Roger> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="uAKRQypu60I7Lcqm" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20201126150937.jhbfp7iefkmtedx7@Air-de-Roger> X-Greylist: Sender succeeded STARTTLS authentication, not delayed by milter-greylist-4.4.3 (chassiron.antioche.eu.org [151.127.5.145]); Thu, 26 Nov 2020 18:20:41 +0100 (MET) --uAKRQypu60I7Lcqm Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit On Thu, Nov 26, 2020 at 04:09:37PM +0100, Roger Pau Monné wrote: > > > > Oh, that's actually very useful. The interrupt is being constantly > > injected from the hardware and received by Xen, it's just not then > > injected into dom0 - that's the bit we are missing. Let me look into > > adding some more debug to that path, hopefully it will tell us where > > things are getting blocked. > > So I have yet one more patch for you to try, this one has more > debugging and a slight change in the emulated IO-APIC behavior. > Depending on the result I might have to find a way to mask the > interrupt so it doesn't spam the whole buffer in order for us to see > exactly what triggered this scenario you are in. OK, here it is: http://www-soc.lip6.fr/~bouyer/xen-log9.txt I had to restart from a clean source tree to apply this patch, so to make sure we're in sync I attached the diff from my sources -- Manuel Bouyer NetBSD: 26 ans d'experience feront toujours la difference -- --uAKRQypu60I7Lcqm Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="xen.diff" diff --git a/xen/arch/x86/hvm/irq.c b/xen/arch/x86/hvm/irq.c index 38ac5fb6c7..9db3dcc957 100644 --- a/xen/arch/x86/hvm/irq.c +++ b/xen/arch/x86/hvm/irq.c @@ -187,6 +187,10 @@ void hvm_gsi_assert(struct domain *d, unsigned int gsi) * to know if the GSI is pending or not. */ spin_lock(&d->arch.hvm.irq_lock); + if ( gsi == TRACK_IRQ ) + debugtrace_printk("hvm_gsi_assert irq %u trig %u assert count %u\n", + gsi, trig, hvm_irq->gsi_assert_count[gsi]); + if ( trig == VIOAPIC_EDGE_TRIG || !hvm_irq->gsi_assert_count[gsi] ) { if ( trig == VIOAPIC_LEVEL_TRIG ) diff --git a/xen/arch/x86/hvm/vioapic.c b/xen/arch/x86/hvm/vioapic.c index 67d4a6237f..aeff9c7687 100644 --- a/xen/arch/x86/hvm/vioapic.c +++ b/xen/arch/x86/hvm/vioapic.c @@ -257,7 +257,17 @@ static void vioapic_write_redirent( vlapic_adjust_i8259_target(d); } else if ( ent.fields.trig_mode == VIOAPIC_EDGE_TRIG ) + { + if ( gsi == TRACK_IRQ ) + debugtrace_printk("vIO-APIC set edge trigger irq %u\n", gsi); pent->fields.remote_irr = 0; + if ( is_iommu_enabled(d) ) + { + spin_unlock(&d->arch.hvm.irq_lock); + hvm_dpci_eoi(d, gsi, pent); + spin_lock(&d->arch.hvm.irq_lock); + } + } else if ( !ent.fields.mask && !ent.fields.remote_irr && hvm_irq->gsi_assert_count[idx] ) @@ -278,6 +288,10 @@ static void vioapic_write_redirent( */ int ret = vioapic_hwdom_map_gsi(gsi, ent.fields.trig_mode, ent.fields.polarity); + + if ( gsi == TRACK_IRQ ) + debugtrace_printk("vIO-APIC UNMASK irq %u\n", gsi); + if ( ret ) { gprintk(XENLOG_ERR, @@ -285,6 +299,9 @@ static void vioapic_write_redirent( unmasked = 0; } } + else if ( is_hardware_domain(d) && gsi == TRACK_IRQ ) + debugtrace_printk("vIO-APIC MASK irq %u\n", gsi); + if ( gsi == 0 || unmasked ) pt_may_unmask_irq(d, NULL); @@ -405,6 +422,10 @@ static void vioapic_deliver(struct hvm_vioapic *vioapic, unsigned int pin) ASSERT(spin_is_locked(&d->arch.hvm.irq_lock)); + if ( irq == TRACK_IRQ ) + debugtrace_printk("vIO-APIC deliver irq %u vector %u\n", + irq, vector); + HVM_DBG_LOG(DBG_LEVEL_IOAPIC, "dest=%x dest_mode=%x delivery_mode=%x " "vector=%x trig_mode=%x", diff --git a/xen/arch/x86/io_apic.c b/xen/arch/x86/io_apic.c index e66fa99ec7..c28025657d 100644 --- a/xen/arch/x86/io_apic.c +++ b/xen/arch/x86/io_apic.c @@ -1641,6 +1641,9 @@ static void mask_and_ack_level_ioapic_irq(struct irq_desc *desc) unsigned long v; int i; + if ( desc->irq == TRACK_IRQ ) + debugtrace_printk("ACK irq %u\n", desc->irq); + irq_complete_move(desc); if ( !directed_eoi_enabled ) @@ -1688,6 +1691,9 @@ static void mask_and_ack_level_ioapic_irq(struct irq_desc *desc) static void end_level_ioapic_irq_old(struct irq_desc *desc, u8 vector) { + if ( desc->irq == TRACK_IRQ ) + debugtrace_printk("END irq %u\n", desc->irq); + if ( directed_eoi_enabled ) { if ( !(desc->status & (IRQ_DISABLED|IRQ_MOVE_PENDING)) ) diff --git a/xen/arch/x86/irq.c b/xen/arch/x86/irq.c index 93c4fb9a79..cc5a0e2a21 100644 --- a/xen/arch/x86/irq.c +++ b/xen/arch/x86/irq.c @@ -1109,6 +1109,10 @@ static void irq_guest_eoi_timer_fn(void *data) unsigned int i, irq = desc - irq_desc; irq_guest_action_t *action; + if ( desc->irq == TRACK_IRQ ) + debugtrace_printk("irq_guest_eoi_timer_fn irq %u status %x\n", + desc->irq, desc->status); + spin_lock_irq(&desc->lock); if ( !(desc->status & IRQ_GUEST) ) @@ -1118,6 +1122,10 @@ static void irq_guest_eoi_timer_fn(void *data) ASSERT(action->ack_type != ACKTYPE_NONE); + if ( desc->irq == TRACK_IRQ ) + debugtrace_printk("ack_type %u in_flight %u\n", + action->ack_type, action->in_flight); + /* * Is no IRQ in flight at all, or another instance of this timer already * running? Skip everything to avoid forcing an EOI early. @@ -1837,6 +1845,10 @@ static void do_IRQ_guest(struct irq_desc *desc, unsigned int vector) unsigned int i; struct pending_eoi *peoi = this_cpu(pending_eoi); + if ( desc->irq == TRACK_IRQ ) + debugtrace_printk("do_IRQ_guest irq %u\n", desc->irq); + + if ( unlikely(!action->nr_guests) ) { /* An interrupt may slip through while freeing an ACKTYPE_EOI irq. */ diff --git a/xen/drivers/passthrough/io.c b/xen/drivers/passthrough/io.c index 6b1305a3e5..86c2db9da0 100644 --- a/xen/drivers/passthrough/io.c +++ b/xen/drivers/passthrough/io.c @@ -174,7 +174,6 @@ static void pt_irq_time_out(void *data) * In the identity mapped case the EOI can also be done now, this way * the iteration over the list of domain pirqs is avoided. */ - hvm_gsi_deassert(irq_map->dom, dpci_pirq(irq_map)->pirq); irq_map->flags |= HVM_IRQ_DPCI_EOI_LATCH; pt_irq_guest_eoi(irq_map->dom, irq_map, NULL); spin_unlock(&irq_map->dom->event_lock); @@ -828,6 +827,9 @@ int hvm_do_IRQ_dpci(struct domain *d, struct pirq *pirq) !pirq_dpci || !(pirq_dpci->flags & HVM_IRQ_DPCI_MAPPED) ) return 0; + if ( pirq->pirq == TRACK_IRQ ) + debugtrace_printk("hvm_do_IRQ_dpci irq %u\n", pirq->pirq); + pirq_dpci->masked = 1; raise_softirq_for(pirq_dpci); return 1; @@ -1010,6 +1012,9 @@ void hvm_dpci_eoi(struct domain *d, unsigned int guest_gsi, if ( !is_iommu_enabled(d) ) return; + if ( guest_gsi == TRACK_IRQ ) + debugtrace_printk("hvm_dpci_eoi irq %u\n", guest_gsi); + if ( is_hardware_domain(d) ) { spin_lock(&d->event_lock); diff --git a/xen/drivers/vpci/msix.c b/xen/drivers/vpci/msix.c index 64dd0a929c..3eb6102a61 100644 --- a/xen/drivers/vpci/msix.c +++ b/xen/drivers/vpci/msix.c @@ -370,7 +370,7 @@ static int msix_write(struct vcpu *v, unsigned long addr, unsigned int len, entry->updated = false; } - else + else if ( msix->enabled ) vpci_msix_arch_mask_entry(entry, pdev, entry->masked); break; diff --git a/xen/include/xen/irq.h b/xen/include/xen/irq.h index 43d567fe44..871810134f 100644 --- a/xen/include/xen/irq.h +++ b/xen/include/xen/irq.h @@ -174,4 +174,6 @@ unsigned int arch_hwdom_irqs(domid_t); void arch_evtchn_bind_pirq(struct domain *, int pirq); #endif +#define TRACK_IRQ 34 + #endif /* __XEN_IRQ_H__ */ --uAKRQypu60I7Lcqm--