From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B9C9EC10F27 for ; Wed, 11 Mar 2020 14:20:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 8D82020727 for ; Wed, 11 Mar 2020 14:20:40 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="SD5UJtg9" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729729AbgCKOUj (ORCPT ); Wed, 11 Mar 2020 10:20:39 -0400 Received: from mail-ed1-f67.google.com ([209.85.208.67]:45694 "EHLO mail-ed1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729584AbgCKOUj (ORCPT ); Wed, 11 Mar 2020 10:20:39 -0400 Received: by mail-ed1-f67.google.com with SMTP id h62so3027536edd.12 for ; Wed, 11 Mar 2020 07:20:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=9tl1iLU38egfZZhVKhZocJxiDFhBVJ4sdLJ75mYBMTg=; b=SD5UJtg9J2lvJOl1Zy/rj4qjbUj21+4W0+L/d8CTr/MZv0+6e5iPxIuW0cLKgrQQ6h LOm+aVH5nH9b1QMhojb1j/mLN5ltf0ySN74NZ5coasqw1qlD10srFtvFSNLAAsxre6a/ 52Y6l5cHs0zJoWteD11msQm0LNZ2p5lYbWuCRVM3z8Vit5wCs6NiRIDcdBcnCenKHNNL ZlPa1fKIEHb3uCYPdZGc7srSrlV9JAomcTCMJn1FCZZt/Sgx1hqSkXCethHXzMTc0RMX lhQUsGNbZj+u7Wyx/fFlDzyrtVLbf9WSavzdoTjxG6K9yf0uH1YepJZnCQbgsMuXLnWx P7/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=9tl1iLU38egfZZhVKhZocJxiDFhBVJ4sdLJ75mYBMTg=; b=E0MXcojNHvo9Ng2u71oA9Fg2vOY98hQuKFxeuvkk9fRv+E4YOt+WJR1FEp9vHR6ksk w15KY4Lq8kMfXRQimtnP1s77PpJjsuqgdG62h1a0lsb7f/lKm/V4iiJ9PUVUUys2UfnR HnEU/jmNOypa8fkGvScp0UXhLOpeH0X+sLX7Z8HtbpOlNMQGiuAnJt3xBOdNudxBJ7Dz pir5t1yvgrKXETNJn97cZNPG6OHdxO5VbBYAV39McyPNiUTGk6lqgPEP3PKcFbM7rgk9 /2vMsXMZtyKmQaU1it42nMMmjSft7PJiURseYj4phFGh8lvkz+ba5UQNaQ5W6XsaM/pU Oz5w== X-Gm-Message-State: ANhLgQ0OU9vvQR2A5N6wgsJ0j/NN1aHAsgpMCmWTh+0RBPqpX/qpG67n LSSQiK07vkF4T3jtIhxDHG5qHvQkz9MwEDP8jlYm9w== X-Google-Smtp-Source: ADFU+vv8g/7qhoeTIHJ81NUyoMBbdJNoqQewHY8yvtwiLsjRorCZhuuvggIj6sKvW5SMLE/8WKjKvRB2ftfGGgIHov8= X-Received: by 2002:aa7:d9d9:: with SMTP id v25mr2984333eds.291.1583936437845; Wed, 11 Mar 2020 07:20:37 -0700 (PDT) MIME-Version: 1.0 References: <20200307213853.GA208095@google.com> In-Reply-To: From: =?UTF-8?B?THXDrXMgTWVuZGVz?= Date: Wed, 11 Mar 2020 14:20:26 +0000 Message-ID: Subject: Re: Problem with PCIe enumeration of Google/Coral TPU Edge module on Linux To: Nicholas Johnson Cc: Bjorn Helgaas , Linux PCI , Thomas Petazzoni , Jason Cooper , Benjamin Herrenschmidt Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Re-sending... after 48 hours the previous message still didn't hit the mailing list... On Mon, Mar 9, 2020 at 11:21 AM Lu=C3=ADs Mendes = wrote: > > Hi Nicholas, > > Thanks for your help. > Replies follow below. > > Kind Regards, > Lu=C3=ADs > > On Sun, Mar 8, 2020 at 5:51 AM Nicholas Johnson > wrote: > > > > Hi, > > > > On Sat, Mar 7, 2020 at 12:11 PM Lu=C3=ADs Mendes wrote: > > > > > This issue seems to happen only with the Coral Edge TPU device, b= ut it > > > > > happens independently of whether the gasket/apex driver module is > > > > > loaded or not. The BAR 0 of the Coral device is not assigned eith= er > > > > > way. > > > > > > > > > > Lu=C3=ADs > > > > So the problem only occurs with the Coral Edge TPU device, so there is = a > > possibility that it is not a problem with the platform, or something > > caused by the combination of the TPU and platform. Is it possible to pu= t > > the TPU into an X86 system with the same kernel version(s) to add more > > evidence to this theory? If it works on X86 then we can focus on the > > differences between X86 and ARM. > > I've tested two Coral TPUs on two x86_64 platforms and the BARs seem > to be assigned, however the driver fails to load, during probe and the > system is unable to shutdown cleanly, but I think that is a driver > issue when setting up sysfs entries. I can blacklist gasket/apex, if > it helps in some way, or try an older kernel. > Dmesg log for one of the x86_64 system is here: > https://pastebin.ubuntu.com/p/FHhHNN6XTF/ > lspci -vvv for the same x86_64 system is here: > https://pastebin.ubuntu.com/p/xbSNWFQ9TS/ > > > > > Also, please revert c13704f5685d "PCI: Avoid double hpmemsize MMIO > > window assignment" or try with Linux v5.4 which does not have this > > commit, just to rule out the possibility of it causing issues. This > > patch helps me and also solved the problem of one other person using an > > ARM computer who came to us regarding a problem. However, it could also > > adversely affect unknown use cases - it is impossible to completely rul= e > > out, due to the nature of how drivers/pci/setup-bus.c is written. > > On armhf with 5.4.14 the problem remains, BAR 0 and BAR 2 are not > assigned: https://pastebin.ubuntu.com/p/9H2qqqMNJN/ > I've also tried kernel 4.20.11 and the problem also exists. > I've got JTAG on this system with OpenOCD. I believe I can debug the > kernel, if needed. > > > > > Kind regards, > > Nicholas