From: Myron Stowe <myron.stowe@gmail.com>
To: Ron Yuan <ron.yuan@memblaze.com>
Cc: Bjorn Helgaas <helgaas@kernel.org>,
Radjendirane Codandaramane <radjendirane.codanda@microsemi.com>,
Sinan Kaya <okaya@codeaurora.org>,
Bjorn Helgaas <bhelgaas@google.com>,
Bo Chen <bo.chen@memblaze.com>,
William Huang <william.huang@memblaze.com>,
Fengming Wu <fengming.wu@memblaze.com>,
Jason Jiang <jason.jiang@microsemi.com>,
Ramyakanth Edupuganti <Ramyakanth.Edupuganti@microsemi.com>,
William Cheng <william.cheng@microsemi.com>,
"Kim Helper (khelper)" <khelper@micron.com>,
Linux PCI <linux-pci@vger.kernel.org>
Subject: Re: One Question About PCIe BUS Config Type with pcie_bus_safe or pcie_bus_perf On NVMe Device
Date: Wed, 31 Jan 2018 17:01:47 -0700 [thread overview]
Message-ID: <CAL-B5D3rbY57UNANUcuguHy1X4CzhCSpGk0i9i-13RHEo9NQ6Q@mail.gmail.com> (raw)
In-Reply-To: <SHAPR01MB173F9910CA9B2651104E319FEFB0@SHAPR01MB173.CHNPR01.prod.partner.outlook.cn>
On Wed, Jan 31, 2018 at 1:40 AM, Ron Yuan <ron.yuan@memblaze.com> wrote:
> Hi, I would like to provide more information just for anyone who might be=
interested.
> We modify FW to simulate a MPS 128 capability SSD, and experiment with di=
fferent pcis_bus mode on Dell R730XD, hence we can have a better look at th=
e whole picture.
>
> First, cold boot with single SSD
> device: Slot (C 256) SSD (C 128B) Slot (C 256B) SSD (C 256B)
> MPS MRRS MPS MRRS MPS MRRS MPS MRRS
> Normal 128 128 128 4096
> Normal 256 128 256 4=
096
> Perf 256 128 128 128
> Perf 256 128 256 2=
56
> Safe 128 128 128 4096
> Safe 256 128 25=
6 4096
>
> Then cold boot with two devices, different MPS capability=EF=BC=8C both s=
lots are directly connect to CPU
> device: slot (C 256) SSD (C 128B) Slot (C 256B) SSD (C 256B)
> MPS MRRS MPS MRRS MPS MRRS MPS MRRS
> Normal 128 128 128 4096 256 128 256 4096
> Perf 256 128 128 128 256 128 256 2=
56
> Safe 128 128 128 4096 256 128 256 4096
>
> Finally, to match Sinan's example, we use a PCIe switch for two U.2 SSD:
> \-[0000:00]-+-00.0
> +-01.0-[03]----00.0
> +-02.0-[04]--
> +-03.0-[02]--+-00.0
> | \-00.1
> +-03.1-[01]--+-00.0
> | \-00.1
> +-03.2-[05-0a]----00.0-[06-0a]--+-04.0-[07]--
> | +-05.0-[08]----00.0 -> co=
nnect a 256B ssd
> | +-06.0-[09]----00.0 -> co=
nnect a 128B ssd
>
> 00.03.2 (C 256) 05:00.0 (C512) 06:05.0 (C512) 08:00.0=
(SSD C256=EF=BC=8906:06.0 (C512) 09:00.0 (SSD C128)
> MPS MRRS MPS MRRS MPS MRRS MPS M=
RRS MPS MRRS MPS MRRS
> Normal 128 128 128 128 128 128 128 =
4096 128 128 128 4096
> Perf 256 128 256 128 256 128 256 =
256 256 128 128 128
> Safe 128 128 128 128 128 128 =
128 4096 128 128 128 4096
>
> I think from above examples:
> 1. perf mode is moving devices to 256 MPS as it can.
> 2. safe mode is setting to 128 MPS
> 3. perf mode set MRRS=3DMPS is a CORRECT call for device with MPSC lower =
than its parents.
> 4. perf mode set MRRS=3DMPS is not necessary for a device with SAME MPSC =
as its parents?
> 5. it is an interested point to me that slot/switch/root MRRS are always =
set to 128B, I have not found out why.
In Sinan's original posting, a reference to
https://www.xilinx.com/support/documentation/white_papers/wp350.pdf
was provided. When I read that paper and got to the "Read Completion
Boundary" section I thought to myself: "If RCB can only be 64 or 128
bytes then what's the point of MPS (or MRRS) as all TLP completions
would be limited to 64 or 128 bytes? (see also the paper's 'Read
Completions with the RCB Set to 64 Bytes' figure)". I brought this up
to a colleague and they surmised that possibly only _lower end_
(a.k.a. lazy) chipset implementations would truly have RCB limited
sized completions; higher end chipsets would of course have to comply
with RCB when communicating with the memory controller but could then
aggregate data into larger MPS (or MRRS) sized TLP completion packets.
Perhaps this might explain why you always saw slot/switch/root values
set at 128B?
>
> Again, thanks for everyone's time on this subject. We have learnt a lot.
>
> Ron
next prev parent reply other threads:[~2018-02-01 0:01 UTC|newest]
Thread overview: 25+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <SH2PR01MB106A1E21DEB5FE3FFB3D61C83E90@SH2PR01MB106.CHNPR01.prod.partner.outlook.cn>
[not found] ` <ef16a3cc-b641-a30d-644a-7940e340eb3e@codeaurora.org>
[not found] ` <SHAPR01MB173A5EA1677C2138CB528F2FEE90@SHAPR01MB173.CHNPR01.prod.partner.outlook.cn>
[not found] ` <5727b0b1-f0d5-7c78-373e-fc9d1bd662d2@codeaurora.org>
[not found] ` <CABhMZUU0643U-qVc9ymA+1PMZToSLFm2dg8-cu=iQ2LGw2Pi8Q@mail.gmail.com>
[not found] ` <SHAPR01MB173A36104635A8BFF9A83E1FEE80@SHAPR01MB173.CHNPR01.prod.partner.outlook.cn>
2018-01-18 16:24 ` One Question About PCIe BUS Config Type with pcie_bus_safe or pcie_bus_perf On NVMe Device Sinan Kaya
2018-01-19 20:51 ` Bjorn Helgaas
2018-01-20 19:20 ` Sinan Kaya
2018-01-20 19:29 ` Sinan Kaya
2018-01-22 21:36 ` Bjorn Helgaas
2018-01-22 22:04 ` Sinan Kaya
2018-01-22 22:51 ` Bjorn Helgaas
2018-01-22 23:24 ` Sinan Kaya
2018-01-23 0:16 ` Bjorn Helgaas
2018-01-23 2:27 ` Sinan Kaya
2018-01-23 13:25 ` Ron Yuan
2018-01-23 14:01 ` Ron Yuan
2018-01-23 17:48 ` Bjorn Helgaas
2018-01-23 18:07 ` Bjorn Helgaas
2018-01-23 14:38 ` Bjorn Helgaas
2018-01-23 23:50 ` Radjendirane Codandaramane
2018-01-24 16:29 ` Myron Stowe
2018-01-24 17:59 ` Ron Yuan
2018-01-24 18:01 ` Bjorn Helgaas
2018-01-31 8:40 ` Ron Yuan
2018-02-01 0:01 ` Myron Stowe [this message]
2018-02-01 0:13 ` Sinan Kaya
2018-02-01 3:37 ` Bjorn Helgaas
2018-02-01 15:14 ` Sinan Kaya
2018-02-05 1:02 ` Sinan Kaya
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAL-B5D3rbY57UNANUcuguHy1X4CzhCSpGk0i9i-13RHEo9NQ6Q@mail.gmail.com \
--to=myron.stowe@gmail.com \
--cc=Ramyakanth.Edupuganti@microsemi.com \
--cc=bhelgaas@google.com \
--cc=bo.chen@memblaze.com \
--cc=fengming.wu@memblaze.com \
--cc=helgaas@kernel.org \
--cc=jason.jiang@microsemi.com \
--cc=khelper@micron.com \
--cc=linux-pci@vger.kernel.org \
--cc=okaya@codeaurora.org \
--cc=radjendirane.codanda@microsemi.com \
--cc=ron.yuan@memblaze.com \
--cc=william.cheng@microsemi.com \
--cc=william.huang@memblaze.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).