From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.1 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 07D38C04AAC for ; Thu, 23 May 2019 04:43:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id C8CE720B1F for ; Thu, 23 May 2019 04:43:23 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="sTN5QPdL" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1725811AbfEWEnX (ORCPT ); Thu, 23 May 2019 00:43:23 -0400 Received: from mail-ot1-f44.google.com ([209.85.210.44]:42754 "EHLO mail-ot1-f44.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725792AbfEWEnW (ORCPT ); Thu, 23 May 2019 00:43:22 -0400 Received: by mail-ot1-f44.google.com with SMTP id i2so4211735otr.9 for ; Wed, 22 May 2019 21:43:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=YuqufgVVeAOGCpxkKbTgC8JrNpTgMtgIwu/Qt17UiVA=; b=sTN5QPdLFC1/bJ5CUEpz5WqxG7VWGDdPmVdCk+gJ6bc6qv3TAxtZz57Q420Y0qLVgF /0Qpudh0b/cAQGCHdHaX02LmGWj01uokQd4KiTY7SvsjLcgiB4fBzYKeN6YRGb1UmYCj 79wTBfnax6RmKUljHNFZGb0m0GUP+XKVbD8p6V3LOF5yyfjg2ELw4avcs8rI0W45BEBn 5fFTw2tFUYNX8RhTPqJ5Ch9KOZmrfzSKQBoOjk66paYSTdTWdl/PVi7mH6VxN8dDjoDx 32S9+rwqfAOzdKu1RLjP4nnuD8kQFqV7jN41thVW3NgtG/XPQoqTSWs1puL/ciS6NDeu Ezgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=YuqufgVVeAOGCpxkKbTgC8JrNpTgMtgIwu/Qt17UiVA=; b=PuiolBGConiVnjTGe/Cxv7XDOu20JXup5ynBsGSYdHCdTlDeX5CGByVY+Zycv0wEMd WmLXu1es6UmL5i51sESpPkUPEX95AuOX9w9h/i99/Gx4bgbENTAKEt8n934TbbAbXyrn 4yVroW9IFq9RT+j3i3ye196u+AZiqh5f/+0HhQNwhd8jOItg3FFSJlTUqRxuon5Ghiqe 6NExry9GFFcRM7y4MXy10Sh4CkAENXAZfRz3UeF17s6WBFsUNVlQU3gRzr80712rxqv6 2Y7l2wIxMg75U2TCDlgappA+Dav1nylMpQ/1VK5zvaQ7yJsDxKAx/WSSPOLLj6680pYO wZ+g== X-Gm-Message-State: APjAAAWs3dkwlzD/j59XL+heTgfbSGm4gnxqBKz9KH1lFAl/sqsct/L5 4Qk/7RC7RPg8mKyqg52YXU5e5HdUW1LHmtSv9Uo= X-Google-Smtp-Source: APXvYqy4L/EhX1UGswxoms78TPoTKiwSqstURAxWJskyhBpwYNdH9DbWQK1EWnFNRmZvI27uI5qWnNXZUDxhA4Bup/g= X-Received: by 2002:a9d:60d9:: with SMTP id b25mr3327170otk.133.1558586601832; Wed, 22 May 2019 21:43:21 -0700 (PDT) MIME-Version: 1.0 References: <21dcf273-929a-6fb1-7978-37145ea62301@bell.net> In-Reply-To: <21dcf273-929a-6fb1-7978-37145ea62301@bell.net> From: Grant Grundler Date: Wed, 22 May 2019 21:43:10 -0700 Message-ID: Subject: Re: HPPA problems with the PCI To: John David Anglin Cc: Carlo Pisani , linux-parisc Content-Type: text/plain; charset="UTF-8" Sender: linux-parisc-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-parisc@vger.kernel.org [resending as plain-text. :( gmail is such a PITA - it should just know by now that anything going to vger needs to be plain text. :/ ] On Wed, May 22, 2019 at 6:09 PM John David Anglin wrote: > > On 2019-05-22 8:38 p.m., Grant Grundler wrote: > > While PCI bus type is the same, most problems with device drivers are > > at DMA Coherency/Memory ordering level. C3600 has PA8600 processor > > and you might learn more about PA-8600 processor, Astro (IOMMU), and > > Elroy (PCI host controller) from > > https://www.openpa.net/systems/hp-visualize_b1000_c3000_c3600.html > > > > If you can try some experiments, start adding mb() calls after the > > driver adds or removes an IO request from any list or queue. > That's an interesting comment. I was also being very lazy and imprecise. :) More specifically, I was thinking the mb() should be placed AFTER adding any IOs to data structure in memory the device will read but BEFORE the driver tells the device more IO requests. I didn't look if such sequences even exist in the drivers mentioned. If the devices use "mail boxes", completely different issues around ordering can come up. > On a UP kernel, mb() is currently just a compiler > memory barrier. On a SMP kernel, mb() generates a "sync" instruction. We also > use "ldcw" as a barrier in spinlocks. Yeah, I'm not sure how strong the mb() needs to be and maybe I'm giving the wrong advice: use dma_wmb() for the case I've described above. Then use dma_rmb() before reading data structures updated by the device. See examples in the existing code: https://elixir.bootlin.com/linux/v4.20/ident/dma_wmb [It's interesting that mostly networking drivers are using these memory barriers and very few block devices do.] OTOH, limiting the compiler is often sufficient to get "expected behavior" since the caches and memory controllers usually aren't that busy/backlogged that memory accesses can get very much out of order - especially memory reads. Especially for drivers only tested on x86 where everything is pretty strongly ordered and compiler doesn't have many registers to work with. These older drivers are more likely to have issues with RISC compiler ordering accesses into sequences that are "unexpected" (even if perfectly correct from compilers PoV). > I'm thinking dma_rmb() and dma_wmb() may need to be stronger. Is "sync" or "syncdma" > a better choice for these defines? Hrm, maybe but seems like we should be using dma_*mb() calls since they are available. cheers, grant > > Cheers, > Dave > > -- > John David Anglin dave.anglin@bell.net >