From: Ding Tianhong <dingtianhong@huawei.com>
To: Casey Leedom <leedom@chelsio.com>,
Alexander Duyck <alexander.duyck@gmail.com>
Cc: Netdev <netdev@vger.kernel.org>,
Bjorn Helgaas <helgaas@kernel.org>,
"linux-arm-kernel@lists.infradead.org"
<linux-arm-kernel@lists.infradead.org>,
"David.Laight@aculab.com" <David.Laight@aculab.com>,
"ashok.raj@intel.com" <ashok.raj@intel.com>,
"Alex Williamson" <alex.williamson@redhat.com>,
"l.stach@pengutronix.de" <l.stach@pengutronix.de>,
"Suravee.Suthikulpanit@amd.com" <Suravee.Suthikulpanit@amd.com>,
"catalin.marinas@arm.com" <catalin.marinas@arm.com>,
"linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>,
"will.deacon@arm.com" <will.deacon@arm.com>,
Sinan Kaya <okaya@codeaurora.org>,
"robin.murphy@arm.com" <robin.murphy@arm.com>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"davem@davemloft.net" <davem@davemloft.net>,
Ganesh GR <ganeshgr@chelsio.com>,
"asit.k.mallick@intel.com" <asit.k.mallick@intel.com>,
"jeffrey.t.kirsher@intel.com" <jeffrey.t.kirsher@intel.com>,
"mark.rutland@arm.com" <mark.rutland@arm.com>,
"gabriele.paoloni@huawei.com" <gabriele.paoloni@huawei.com>,
Michael Werner <werner@chelsio.com>,
"bhelgaas@google.com" <bhelgaas@google.com>,
"patrick.j.cramer@intel.com" <patrick.j.cramer@intel.com>,
"linuxarm@huawei.com" <linuxarm@huawei.com>,
"amira@mellanox.com" <amira@mellanox.com>,
"Bob.Shaw@amd.com" <Bob.Shaw@amd.com>
Subject: Re: [PATCH v7 2/3] PCI: Enable PCIe Relaxed Ordering if supported
Date: Fri, 28 Jul 2017 10:48:59 +0800 [thread overview]
Message-ID: <52e7c477-2f26-16d2-79f7-5f6baba60685@huawei.com> (raw)
In-Reply-To: <MWHPR12MB1600556FFDCE3EC810D97591C8BE0@MWHPR12MB1600.namprd12.prod.outlook.com>
On 2017/7/28 1:44, Casey Leedom wrote:
> | From: Ding Tianhong <dingtianhong@huawei.com>
> | Sent: Wednesday, July 26, 2017 6:01 PM
> |
> | On 2017/7/27 3:05, Casey Leedom wrote:
> | >
> | > Ding, send me a note if you'd like me to work that [cxgb4vf patch] up
> | > for you.
> |
> | Ok, you could send the change log and I could put it in the v8 version
> | together, will you base on the patch 3/3 or build a independence patch?
>
> Which ever you'd prefer. It would basically mirror the same exact code that
> you've got for cxgb4. I.e. testing the setting of the VF's PCIe Capability
> Device Control[Relaxed Ordering Enable], setting a new flag in
> adpater->flags, testing that flag in cxgb4vf/sge.c:t4vf_sge_alloc_rxq().
> But since the VF's PF will already have disabled the PF's Relaxed Ordering
> Enable, the VF will also have it's Relaxed Ordering Enable disabled and any
> effort by the internal chip to send TLPs with the Relaxed Ordering Attribute
> will be gated by the PCIe logic. So it's not critical that this be in the
> first patch. Your call. Let me know if you'd like me to send that to you.
>
Good, please Send it to me, I will put it together and send the v8 this week,
I think Bjorn will be back next week .:)
>
> | From: Ding Tianhong <dingtianhong@huawei.com>
> | Sent: Wednesday, July 26, 2017 6:08 PM
> |
> | On 2017/7/27 2:26, Casey Leedom wrote:
> | >
> | > 1. Did we ever get any acknowledgement from either Intel or AMD
> | > on this patch? I know that we can't ensure that, but it sure would
> | > be nice since the PCI Quirks that we're putting in affect their
> | > products.
> |
> | Still no Intel and AMD guys has ack this, this is what I am worried about,
> | should I ping some man again ?
>
> By amusing coincidence, Patrik Cramer (now Cc'ed) from Intel sent me a note
> yesterday with a link to the official Intel performance tuning documentation
> which covers this issue:
>
> https://software.intel.com/sites/default/files/managed/9e/bc/64-ia-32-architectures-optimization-manual.pdf
>
> In section 3.9.1 we have:
>
> 3.9.1 Optimizing PCIe Performance for Accesses Toward Coherent Memory
> and Toward MMIO Regions (P2P)
>
> In order to maximize performance for PCIe devices in the processors
> listed in Table 3-6 below, the soft- ware should determine whether the
> accesses are toward coherent memory (system memory) or toward MMIO
> regions (P2P access to other devices). If the access is toward MMIO
> region, then software can command HW to set the RO bit in the TLP
> header, as this would allow hardware to achieve maximum throughput for
> these types of accesses. For accesses toward coherent memory, software
> can command HW to clear the RO bit in the TLP header (no RO), as this
> would allow hardware to achieve maximum throughput for these types of
> accesses.
>
> Table 3-6. Intel Processor CPU RP Device IDs for Processors Optimizing
> PCIe Performance
>
> Processor CPU RP Device IDs
>
> Intel Xeon processors based on 6F01H-6F0EH
> Broadwell microarchitecture
>
> Intel Xeon processors based on 2F01H-2F0EH
> Haswell microarchitecture
>
> Unfortunately that's a pretty thin section. But it does expand the set of
> Intel Root Complexes for which our Linux PCI Quirk will need to cover. So
> you should add those to the next (and hopefully final) spin of your patch.
> And, it also verifies the need to handle the use of Relaxed Ordering more
> subtlely than simply turning it off since the NVMe peer-to-peer example I
> keep bringing up would fall into the "need to use Relaxed Ordering" case ...
>
> It would have been nice to know why this is happening and if any future
> processor would fix this. After all, Relaxed Ordering, is just supposed to
> be a hint. At worst, a receiving device could just ignore the attribute
> entirely. Obviously someone made an effort to implement it but ... it
> didn't go the way they wanted.
>
> And, it also would have been nice to know if there was any hidden register
> in these Intel Root Complexes which can completely turn off the effort to
> pay attention to the Relaxed Ordering Attribute. We've spend an enormous
> amount of effort on this issue here on the Linux PCI email list struggling
> mightily to come up with a way to determine when it's
> safe/recommended/not-recommended/unsafe to use Relaxed Ordering when
> directing TLPs towards the Root Complex. And some architectures require RO
> for decent performance so we can't just "turn it off" unilatterally.
>
I am glad to hear that more person were focus on this problem, It would be great
if they could enter our discussion and give us more suggestion. :)
Thanks
Ding
> Casey
>
> .
>
next prev parent reply other threads:[~2017-07-28 2:49 UTC|newest]
Thread overview: 34+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-07-13 14:21 [PATCH v7 0/3] Add new PCI_DEV_FLAGS_NO_RELAXED_ORDERING flag Ding Tianhong
2017-07-13 14:21 ` [PATCH v7 1/3] PCI: Add new PCIe Fabric End Node flag, PCI_DEV_FLAGS_NO_RELAXED_ORDERING Ding Tianhong
2017-08-03 8:55 ` Raj, Ashok
2017-08-03 10:20 ` Ding Tianhong
2017-07-13 14:21 ` [PATCH v7 2/3] PCI: Enable PCIe Relaxed Ordering if supported Ding Tianhong
2017-07-13 21:09 ` Sinan Kaya
2017-07-14 1:26 ` Ding Tianhong
2017-07-14 13:54 ` Sinan Kaya
2017-07-22 4:19 ` Ding Tianhong
2017-07-24 15:05 ` Alex Williamson
2017-07-26 18:26 ` Casey Leedom
[not found] ` <CAKgT0UeAad6WArvrE71MFJywDs1wOnCF-iJRnbNLrL+knqhXeA@mail.gmail.com>
[not found] ` <CAKgT0Uf5hdXUXja_jUB6_kBg6pyX8zXuOMOGzCVNgeBFMUsWqQ@mail.gmail.com>
[not found] ` <CAKgT0Udn2vh6NaqZyiF69nXVnz2sT=e0ZgiDjWznhGZz-Gk+qQ@mail.gmail.com>
2017-07-26 19:05 ` Casey Leedom
2017-07-27 1:01 ` Ding Tianhong
2017-07-27 17:44 ` Casey Leedom
2017-07-27 18:42 ` Raj, Ashok
2017-07-28 2:57 ` Ding Tianhong
2017-07-28 2:48 ` Ding Tianhong [this message]
2017-07-27 1:08 ` Ding Tianhong
2017-07-27 17:49 ` Alexander Duyck
2017-07-28 3:00 ` Ding Tianhong
2017-08-02 17:53 ` Casey Leedom
2017-08-03 8:31 ` Raj, Ashok
2017-08-04 20:20 ` Casey Leedom
2017-08-04 20:21 ` Raj, Ashok
2017-08-04 20:48 ` Casey Leedom
2017-08-07 9:04 ` David Laight
2017-08-03 9:13 ` Raj, Ashok
2017-08-03 10:22 ` Ding Tianhong
2017-07-13 14:21 ` [PATCH v7 3/3] net/cxgb4: Use new PCI_DEV_FLAGS_NO_RELAXED_ORDERING flag Ding Tianhong
2017-07-13 18:14 ` Alexander Duyck
2017-07-13 18:17 ` Alexander Duyck
2017-07-14 0:00 ` Casey Leedom
[not found] ` <MWHPR12MB1600E5A4404EE9D97CD99F88C8AC0@MWHPR12MB1600.namprd12.prod.outlook.com>
2017-07-14 10:23 ` Ding Tianhong
2017-07-14 17:50 ` Casey Leedom
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=52e7c477-2f26-16d2-79f7-5f6baba60685@huawei.com \
--to=dingtianhong@huawei.com \
--cc=Bob.Shaw@amd.com \
--cc=David.Laight@aculab.com \
--cc=Suravee.Suthikulpanit@amd.com \
--cc=alex.williamson@redhat.com \
--cc=alexander.duyck@gmail.com \
--cc=amira@mellanox.com \
--cc=ashok.raj@intel.com \
--cc=asit.k.mallick@intel.com \
--cc=bhelgaas@google.com \
--cc=catalin.marinas@arm.com \
--cc=davem@davemloft.net \
--cc=gabriele.paoloni@huawei.com \
--cc=ganeshgr@chelsio.com \
--cc=helgaas@kernel.org \
--cc=jeffrey.t.kirsher@intel.com \
--cc=l.stach@pengutronix.de \
--cc=leedom@chelsio.com \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-pci@vger.kernel.org \
--cc=linuxarm@huawei.com \
--cc=mark.rutland@arm.com \
--cc=netdev@vger.kernel.org \
--cc=okaya@codeaurora.org \
--cc=patrick.j.cramer@intel.com \
--cc=robin.murphy@arm.com \
--cc=werner@chelsio.com \
--cc=will.deacon@arm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).