From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.3 required=3.0 tests=DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0021EC433E0 for ; Mon, 8 Jun 2020 12:33:50 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B64E32074B for ; Mon, 8 Jun 2020 12:33:50 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="b1SPYhIW" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B64E32074B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jiGy0-00089Q-61; Mon, 08 Jun 2020 12:33:28 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jiGxz-00089L-16 for xen-devel@lists.xenproject.org; Mon, 08 Jun 2020 12:33:27 +0000 X-Inumbo-ID: 3f98c2d2-a984-11ea-96fb-bc764e2007e4 Received: from mail-ej1-x643.google.com (unknown [2a00:1450:4864:20::643]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 3f98c2d2-a984-11ea-96fb-bc764e2007e4; Mon, 08 Jun 2020 12:33:26 +0000 (UTC) Received: by mail-ej1-x643.google.com with SMTP id y13so18095614eju.2 for ; Mon, 08 Jun 2020 05:33:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=M70e72UCuxkCW4wbSfecZcBgKmPhNTcBwAXxM5ZKJVw=; b=b1SPYhIWbq6L1x48hOz0xLwx5Ys1QstrHafH2c9TjA1E6s9/RPLagwvArmFyd8osct ZlDAnf7Dj1HNBjtNJMxE3HHRJrbFNZp+W0hqQRs4JnyyL4RCJSxjOuXCpvhjJhl2cL5k eUq0tCRx+uwRSfoINH3chhKeEdErrZKym1OrF5ts73P/w1h8KClh4jYFmUTtlieaAWZu NX5nhpRRK3yy/X8k5Nt24Fms7dOPMLV/4pTGSSeOTki0/BZp4RqvzalnHYhyCM0on+5c /J0M8ixD55kmvtrBttFORkpruNtitme1OGVfBnjZMW3PNZlhzepExSBZ/M1ITonDMRKE wWrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=M70e72UCuxkCW4wbSfecZcBgKmPhNTcBwAXxM5ZKJVw=; b=bleyh55dgpRYa+5LS0XN+mtNzyljm1QXRHW/jx6yVYPN/7nIq/p1OeyZNlZq8Op9EF AjC4ZxfHOG4YQ8NVEUZ+mRFMLgbmsnoJ9xuFl6jDe4lmjDT0gmRBFGgIk/weK7+3o+Cg YG1KrNauTgWWp8Kfn2iyWdrxdTWCZtRRf26vmoDImo91yS13a4J81ftHy5V7pHIGuFK9 vfK8T6u6/FB8meHkslLgcJm20iXZR6vHiIrwzB/bgsKDgXTuflx6qGOlkd2MOe5B/y5w wqL51ko7sg+KxQV8AX31EUsJCr5l5wixoZ3EGao2RdP6ItOMSUgqApchXDllmQnLXtl2 u6hA== X-Gm-Message-State: AOAM532gYMQPJ/qDBmopUTj7Gz52s/H34NHlqcFyFJjxoc2tVhnSHJJL bm6s5yZ2B85/5nWdZ4lIY+T2LZXd/aqblF1FrqZ4y9cf X-Google-Smtp-Source: ABdhPJyWvZlyJQravMnlR+nb313WS7nccHlMrboOPQ1e+mgIurZ+Qwescx0YJqjTdkbGXiQKYwA99XGjmNKMVZv8qlg= X-Received: by 2002:a17:906:5785:: with SMTP id k5mr20139782ejq.494.1591619605187; Mon, 08 Jun 2020 05:33:25 -0700 (PDT) MIME-Version: 1.0 References: <25953300-f69d-19a4-9215-49cfedbd16ed@xen.org> <8C39F9D0-8351-4671-9A39-D5D4BFF02BD6@arm.com> <3ff17aa9-0aae-d598-40ce-4e90d4e50cc7@xen.org> <00E14EAD-BD23-4A3A-872E-0C47C26B7B41@arm.com> <8C6A23AE-6C2B-411F-ACAD-F5574211E8ED@arm.com> <14244e49-e1ac-a29d-bbd9-bd4c202bf186@xen.org> <77006AAF-BC3B-4C6E-BDFC-577CF87DE64E@arm.com> <99E77330-049F-4471-ABF9-13F9AB4E95B5@arm.com> In-Reply-To: <99E77330-049F-4471-ABF9-13F9AB4E95B5@arm.com> From: CodeWiz2280 Date: Mon, 8 Jun 2020 08:33:12 -0400 Message-ID: Subject: Re: Keystone Issue To: Bertrand Marquis Content-Type: text/plain; charset="UTF-8" X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: xen-devel , nd , Stefano Stabellini , Julien Grall Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" It actually shows only 1 interrupt for any of the devices in that list (e.g. spi, ttyS0, ethernet) so you're probably right on the money with it being an interrupt acknowledge issue. Any help you can provide is greatly appreciated. On Mon, Jun 8, 2020 at 4:40 AM Bertrand Marquis wrote: > > > > > On 5 Jun 2020, at 20:12, CodeWiz2280 wrote: > > > > On Fri, Jun 5, 2020 at 11:05 AM CodeWiz2280 wrote: > >> > >> On Fri, Jun 5, 2020 at 8:47 AM Bertrand Marquis > >> wrote: > >>> > >>> > >>> > >>>> On 5 Jun 2020, at 13:42, CodeWiz2280 wrote: > >>>> > >>>> On Fri, Jun 5, 2020 at 8:30 AM Julien Grall wrote: > >>>>> > >>>>> Hi, > >>>>> > >>>>> On 05/06/2020 13:25, CodeWiz2280 wrote: > >>>>>> The Keystone uses the netcp driver, which has interrupts from 40-79 > >>>>>> listed in the device tree (arch/arm/boot/keystone-k2e-netcp.dtsi). > >>>>>> I'm using the same device tree between my non-xen standalone kernel > >>>>>> and my dom0 kernel booted by xen. In the standalone (non-xen) kernel > >>>>>> the ethernet works fine, but I don't see any of its interrupts in the > >>>>>> output of /proc/iomem. I'm not seeing them in /proc/iomem when > >>>>>> running dom0 under Xen either. When booting with Xen I get this > >>>>>> behavior where the ifconfig output shows 1 RX message and 1 TX > >>>>>> message, and then nothing else. > >>>>> > >>>>> I am not sure whether this is a typo in the e-mail. /proc/iomem is > >>>>> listing the list of the MMIO regions. You want to use /proc/interrupts. > >>>>> > >>>>> Can you confirm which path you are dumping? > >>>> Yes, that was a typo. Sorry about that. I meant that I am dumping > >>>> /proc/interrupts and do not > >>>> see them under the non-xen kernel or xen booted dom0. > >>> > >>> Could you post both /proc/interrupts content ? > >> > >> Standalone non-xen kernel (Ethernet works) > >> # cat /proc/interrupts > >> CPU0 CPU1 CPU2 CPU3 > >> 17: 0 0 0 0 GICv2 29 Level > >> arch_timer > >> 18: 9856 1202 457 650 GICv2 30 Level > >> arch_timer > >> 21: 0 0 0 0 GICv2 142 Edge > >> timer-keystone > >> 22: 0 0 0 0 GICv2 52 Edge arm-pmu > >> 23: 0 0 0 0 GICv2 53 Edge arm-pmu > >> 24: 0 0 0 0 GICv2 54 Edge arm-pmu > >> 25: 0 0 0 0 GICv2 55 Edge arm-pmu > >> 26: 0 0 0 0 GICv2 36 Edge > >> 26202a0.keystone_irq > >> 27: 1435 0 0 0 GICv2 309 Edge ttyS0 > >> 29: 0 0 0 0 GICv2 315 Edge > >> 2530000.i2c > >> 30: 1 0 0 0 GICv2 318 Edge > >> 2530400.i2c > >> 31: 0 0 0 0 GICv2 321 Edge > >> 2530800.i2c > >> 32: 69 0 0 0 GICv2 324 Edge > >> 21000400.spi > >> 33: 0 0 0 0 GICv2 328 Edge > >> 21000600.spi > >> 34: 0 0 0 0 GICv2 332 Edge > >> 21000800.spi > >> 70: 0 0 0 0 GICv2 417 Edge > >> ks-pcie-error-irq > >> 79: 0 0 0 0 PCI-MSI 0 Edge > >> PCIe PME, aerdrv > >> 88: 57 0 0 0 GICv2 80 Level > >> hwqueue-528 > >> 89: 57 0 0 0 GICv2 81 Level > >> hwqueue-529 > >> 90: 47 0 0 0 GICv2 82 Level > >> hwqueue-530 > >> 91: 41 0 0 0 GICv2 83 Level > >> hwqueue-531 > >> IPI0: 0 0 0 0 CPU wakeup interrupts > >> IPI1: 0 0 0 0 Timer broadcast interrupts > >> IPI2: 730 988 1058 937 Rescheduling interrupts > >> IPI3: 2 3 4 6 Function call interrupts > >> IPI4: 0 0 0 0 CPU stop interrupts > >> IPI5: 0 0 0 0 IRQ work interrupts > >> IPI6: 0 0 0 0 completion interrupts > >> > >> Xen dom0 (Ethernet stops) > >> # cat /proc/interrupts > >> CPU0 > >> 18: 10380 GIC-0 27 Level arch_timer > >> 19: 0 GIC-0 142 Edge timer-keystone > >> 20: 88 GIC-0 16 Level events > >> 21: 0 xen-dyn Edge -event xenbus > >> 22: 0 GIC-0 36 Edge 26202a0.keystone_irq > >> 23: 1 GIC-0 312 Edge ttyS0 > >> 25: 1 GIC-0 318 Edge > >> 27: 1 GIC-0 324 Edge 21000400.spi > >> 28: 0 GIC-0 328 Edge 21000600.spi > >> 29: 0 GIC-0 332 Edge 21000800.spi > >> 65: 0 GIC-0 417 Edge ks-pcie-error-irq > >> 74: 0 PCI-MSI 0 Edge PCIe PME, aerdrv > >> 83: 1 GIC-0 80 Level hwqueue-528 > >> 84: 1 GIC-0 81 Level hwqueue-529 > >> 85: 1 GIC-0 82 Level hwqueue-530 > >> 86: 1 GIC-0 83 Level hwqueue-531 > >> 115: 87 xen-dyn Edge -virq hvc_console > >> IPI0: 0 CPU wakeup interrupts > >> IPI1: 0 Timer broadcast interrupts > >> IPI2: 0 Rescheduling interrupts > >> IPI3: 0 Function call interrupts > >> IPI4: 0 CPU stop interrupts > >> IPI5: 0 IRQ work interrupts > >> IPI6: 0 completion interrupts > >> Err: 0 > > After getting a chance to look at this a little more, I believe the > > TX/RX interrupts for the ethernets map like this: > > > > eth0 Rx - hwqueue-528 > > eth1 Rx - hwqueue-529 > > eth0 Tx - hwqueue-530 > > eth1 Tx - hwqueue-531 > >> > > The interrupt counts in the standlone working kernel seem to roughly > > correspond to the counts of Tx/Rx messages in ifconfig. Going on > > that, its clear that only 1 interrupt has been received for Tx and 1 > > for Rx in the Xen Dom0 equivalent. Any thoughts on this? > > This definitely look like an interrupt acknowledgement issue. > This could be caused by 2 things I remember of: > - front vs level interrupts > - a problem with forwarded interrupt acknowledgement. > I think there was something related to that where the vcpu ack was not properly > handled on a keystone and I had to change the way the interrupt was acked for > forwarded hardware interrupts. > > I will try to get more info on that one as I have no access to the code anymore. > > Regards > Bertrand > > > > >