From: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
To: Kishon Vijay Abraham I <kishon@ti.com>
Cc: Bjorn Helgaas <helgaas@kernel.org>,
Bjorn Helgaas <bhelgaas@google.com>,
Jonathan Corbet <corbet@lwn.net>, Arnd Bergmann <arnd@arndb.de>,
Jon Mason <jdmason@kudzu.us>, Dave Jiang <dave.jiang@intel.com>,
Allen Hubbe <allenbh@gmail.com>, Tom Joseph <tjoseph@cadence.com>,
Rob Herring <robh@kernel.org>,
Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
linux-pci@vger.kernel.org, linux-doc@vger.kernel.org,
linux-kernel@vger.kernel.org, linux-ntb@googlegroups.com
Subject: Re: [PATCH v9 01/17] Documentation: PCI: Add specification for the *PCI NTB* function device
Date: Thu, 28 Jan 2021 12:11:47 +0000 [thread overview]
Message-ID: <20210128121147.GA23564@e121166-lin.cambridge.arm.com> (raw)
In-Reply-To: <797ec9f2-34c3-5dc4-cc0a-d4f7cdf4afb0@ti.com>
On Fri, Jan 22, 2021 at 07:48:52PM +0530, Kishon Vijay Abraham I wrote:
> Hi Bjorn,
>
> On 20/01/21 12:04 am, Bjorn Helgaas wrote:
> > On Mon, Jan 04, 2021 at 08:58:53PM +0530, Kishon Vijay Abraham I wrote:
> >> Add specification for the *PCI NTB* function device. The endpoint function
> >> driver and the host PCI driver should be created based on this
> >> specification.
> >>
> >> Signed-off-by: Kishon Vijay Abraham I <kishon@ti.com>
> >
> > A few typos below if there's opportunity for revisions.
>
> I'll fix them.
Hi Kishon,
if you have changes please send them along and I will re-merge the
whole series.
Thanks,
Lorenzo
> >> ---
> >> Documentation/PCI/endpoint/index.rst | 1 +
> >> .../PCI/endpoint/pci-ntb-function.rst | 351 ++++++++++++++++++
> >> 2 files changed, 352 insertions(+)
> >> create mode 100644 Documentation/PCI/endpoint/pci-ntb-function.rst
> >>
> >> diff --git a/Documentation/PCI/endpoint/index.rst b/Documentation/PCI/endpoint/index.rst
> >> index 4ca7439fbfc9..ef6861128506 100644
> >> --- a/Documentation/PCI/endpoint/index.rst
> >> +++ b/Documentation/PCI/endpoint/index.rst
> >> @@ -11,5 +11,6 @@ PCI Endpoint Framework
> >> pci-endpoint-cfs
> >> pci-test-function
> >> pci-test-howto
> >> + pci-ntb-function
> >>
> >> function/binding/pci-test
> >> diff --git a/Documentation/PCI/endpoint/pci-ntb-function.rst b/Documentation/PCI/endpoint/pci-ntb-function.rst
> >> new file mode 100644
> >> index 000000000000..a57908be4047
> >> --- /dev/null
> >> +++ b/Documentation/PCI/endpoint/pci-ntb-function.rst
> >> @@ -0,0 +1,351 @@
> >> +.. SPDX-License-Identifier: GPL-2.0
> >> +
> >> +=================
> >> +PCI NTB Function
> >> +=================
> >> +
> >> +:Author: Kishon Vijay Abraham I <kishon@ti.com>
> >> +
> >> +PCI Non Transparent Bridges (NTB) allow two host systems to communicate
> >> +with each other by exposing each host as a device to the other host.
> >> +NTBs typically support the ability to generate interrupts on the remote
> >> +machine, expose memory ranges as BARs and perform DMA. They also support
> >> +scratchpads which are areas of memory within the NTB that are accessible
> >> +from both machines.
> >> +
> >> +PCI NTB Function allows two different systems (or hosts) to communicate
> >> +with each other by configurig the endpoint instances in such a way that
> >> +transactions from one system is routed to the other system.
> >
> > s/is/are/
> >
> >> +In the below diagram, PCI NTB function configures the SoC with multiple
> >> +PCIe Endpoint (EP) instances in such a way that transaction from one EP
> >> +controller is routed to the other EP controller. Once PCI NTB function
> >
> > s/transaction ... is/transactions ... are/
> >
> >> +configures the SoC with multiple EP instances, HOST1 and HOST2 can
> >> +communicate with each other using SoC as a bridge.
> >> +
> >> +.. code-block:: text
> >> +
> >> + +-------------+ +-------------+
> >> + | | | |
> >> + | HOST1 | | HOST2 |
> >> + | | | |
> >> + +------^------+ +------^------+
> >> + | |
> >> + | |
> >> + +---------|-------------------------------------------------|---------+
> >> + | +------v------+ +------v------+ |
> >> + | | | | | |
> >> + | | EP | | EP | |
> >> + | | CONTROLLER1 | | CONTROLLER2 | |
> >> + | | <-----------------------------------> | |
> >> + | | | | | |
> >> + | | | | | |
> >> + | | | SoC With Multiple EP Instances | | |
> >> + | | | (Configured using NTB Function) | | |
> >> + | +-------------+ +-------------+ |
> >> + +---------------------------------------------------------------------+
> >> +
> >> +Constructs used for Implementing NTB
> >> +====================================
> >> +
> >> + 1) Config Region
> >> + 2) Self Scratchpad Registers
> >> + 3) Peer Scratchpad Registers
> >> + 4) Doorbell Registers
> >> + 5) Memory Window
> >> +
> >> +
> >> +Config Region:
> >> +--------------
> >> +
> >> +Config Region is a construct that is specific to NTB implemented using NTB
> >> +Endpoint Function Driver. The host and endpoint side NTB function driver will
> >> +exchange information with each other using this region. Config Region has
> >> +Control/Status Registers for configuring the Endpoint Controller. Host can
> >> +write into this region for configuring the outbound ATU and to indicate the
> >
> > Expand "ATU" since this is the first mention.
> >
> >> +link status. Endpoint can indicate the status of commands issued be host in
> >> +this region. Endpoint can also indicate the scratchpad offset, number of
> >> +memory windows to the host using this region.
> >
> > s/be host/by host/
> > s/offset, number/offset and number/
> >
> >> +The format of Config Region is given below. Each of the fields here are 32
> >> +bits.
> >
> > s/Each ... are/All ... are/
> >
> >> +
> >> +.. code-block:: text
> >> +
> >> + +------------------------+
> >> + | COMMAND |
> >> + +------------------------+
> >> + | ARGUMENT |
> >> + +------------------------+
> >> + | STATUS |
> >> + +------------------------+
> >> + | TOPOLOGY |
> >> + +------------------------+
> >> + | ADDRESS (LOWER 32) |
> >> + +------------------------+
> >> + | ADDRESS (UPPER 32) |
> >> + +------------------------+
> >> + | SIZE |
> >> + +------------------------+
> >> + | NO OF MEMORY WINDOW |
> >> + +------------------------+
> >> + | MEMORY WINDOW1 OFFSET |
> >> + +------------------------+
> >> + | SPAD OFFSET |
> >> + +------------------------+
> >> + | SPAD COUNT |
> >> + +------------------------+
> >> + | DB ENTRY SIZE |
> >> + +------------------------+
> >> + | DB DATA |
> >> + +------------------------+
> >> + | : |
> >> + +------------------------+
> >> + | : |
> >> + +------------------------+
> >> + | DB DATA |
> >> + +------------------------+
> >> +
> >> +
> >> + COMMAND:
> >> +
> >> + NTB function supports three commands:
> >> +
> >> + CMD_CONFIGURE_DOORBELL (0x1): Command to configure doorbell. Before
> >> + invoking this command, the host should allocate and initialize
> >> + MSI/MSI-X vectors (i.e initialize the MSI/MSI-X capability in the
> >
> > s/i.e/i.e.,/
> >
> >> + Endpoint). The endpoint on receiving this command will configure
> >> + the outbound ATU such that transaction to DB BAR will be routed
> >> + to the MSI/MSI-X address programmed by the host. The ARGUMENT
> >
> > s/transaction to/transactions to/
> >
> > Expand "DB BAR". I assume this refers to "Doorbell BAR" (which itself
> > is not defined). How do we know which is the Doorbell BAR?
>
> right doorbell. That part is explained in the "Modeling Constructs"
> section below.
> >
> > Also, "DB" itself needs to be expanded somehow for uses like below:
> >
> >> + register should be populated with number of DBs to configure (in the
> >> + lower 16 bits) and if MSI or MSI-X should be configured (BIT 16).
> >> + (TODO: Add support for MSI-X).
> >> +
> >> + CMD_CONFIGURE_MW (0x2): Command to configure memory window. The
> >> + host invokes this command after allocating a buffer that can be
> >> + accessed by remote host. The allocated address should be programmed
> >> + in the ADDRESS register (64 bit), the size should be programmed in
> >> + the SIZE register and the memory window index should be programmed
> >> + in the ARGUMENT register. The endpoint on receiving this command
> >> + will configure the outbound ATU such that trasaction to MW BAR
> >> + will be routed to the address provided by the host.
> >
> > How do we know which is the MW BAR? I assume "MW" refers to "Memory
> > Window".
>
> right memory window. That's again explained in the "Modeling Constructs"
> section below.
> >
> >> +
> >> + CMD_LINK_UP (0x3): Command to indicate an NTB application is
> >> + bound to the EP device on the host side. Once the endpoint
> >> + receives this command from both the hosts, the endpoint will
> >> + raise an LINK_UP event to both the hosts to indicate the hosts
> >> + can start communicating with each other.
> >
> > s/raise an/raise a/
> >
> > I guess this "LINK_UP event" is something other than the PCIe DL_Up
> > state, because each host has already been communicating with the
> > endpoint. Right? Is this LINK_UP a software construct?
>
> Yeah. This is when an NTB client application is bound to the NTB device.
> This is used for handshake between the applications running on the two
> hosts.
>
> Thanks
> Kishon
next prev parent reply other threads:[~2021-01-28 12:11 UTC|newest]
Thread overview: 34+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-01-04 15:28 [PATCH v9 00/17] Implement NTB Controller using multiple PCI EP Kishon Vijay Abraham I
2021-01-04 15:28 ` [PATCH v9 01/17] Documentation: PCI: Add specification for the *PCI NTB* function device Kishon Vijay Abraham I
2021-01-19 18:34 ` Bjorn Helgaas
2021-01-22 14:18 ` Kishon Vijay Abraham I
2021-01-28 12:11 ` Lorenzo Pieralisi [this message]
2021-01-28 13:34 ` Kishon Vijay Abraham I
2021-01-29 1:05 ` Randy Dunlap
2021-01-29 5:10 ` Kishon Vijay Abraham I
2021-01-04 15:28 ` [PATCH v9 02/17] PCI: endpoint: Make *_get_first_free_bar() take into account 64 bit BAR Kishon Vijay Abraham I
2021-01-04 15:28 ` [PATCH v9 03/17] PCI: endpoint: Add helper API to get the 'next' unreserved BAR Kishon Vijay Abraham I
2021-01-04 15:28 ` [PATCH v9 04/17] PCI: endpoint: Make *_free_bar() to return error codes on failure Kishon Vijay Abraham I
2021-01-04 15:28 ` [PATCH v9 05/17] PCI: endpoint: Remove unused pci_epf_match_device() Kishon Vijay Abraham I
2021-01-04 15:28 ` [PATCH v9 06/17] PCI: endpoint: Add support to associate secondary EPC with EPF Kishon Vijay Abraham I
2021-01-04 15:28 ` [PATCH v9 07/17] PCI: endpoint: Add support in configfs to associate two EPCs " Kishon Vijay Abraham I
2021-01-04 15:29 ` [PATCH v9 08/17] PCI: endpoint: Add pci_epc_ops to map MSI irq Kishon Vijay Abraham I
2021-01-04 15:29 ` [PATCH v9 09/17] PCI: endpoint: Add pci_epf_ops for epf drivers to expose function specific attrs Kishon Vijay Abraham I
2021-01-04 15:29 ` [PATCH v9 10/17] PCI: endpoint: Allow user to create sub-directory of 'EPF Device' directory Kishon Vijay Abraham I
2021-01-19 18:34 ` Bjorn Helgaas
2021-01-19 18:35 ` Bjorn Helgaas
2021-01-19 18:38 ` Greg Kroah-Hartman
2021-01-04 15:29 ` [PATCH v9 11/17] PCI: cadence: Implement ->msi_map_irq() ops Kishon Vijay Abraham I
2021-01-04 15:29 ` [PATCH v9 12/17] PCI: cadence: Configure LM_EP_FUNC_CFG based on epc->function_num_map Kishon Vijay Abraham I
2021-01-04 15:29 ` [PATCH v9 13/17] PCI: endpoint: Add EP function driver to provide NTB functionality Kishon Vijay Abraham I
2021-01-19 19:04 ` Bjorn Helgaas
2021-01-04 15:29 ` [PATCH v9 14/17] PCI: Add TI J721E device to pci ids Kishon Vijay Abraham I
2021-01-04 15:29 ` [PATCH v9 15/17] NTB: Add support for EPF PCI-Express Non-Transparent Bridge Kishon Vijay Abraham I
2021-01-18 17:12 ` Dave Jiang
2021-01-04 15:29 ` [PATCH v9 16/17] Documentation: PCI: Add configfs binding documentation for pci-ntb endpoint function Kishon Vijay Abraham I
2021-01-04 15:29 ` [PATCH v9 17/17] Documentation: PCI: Add userguide for PCI endpoint NTB function Kishon Vijay Abraham I
2021-01-19 18:34 ` Bjorn Helgaas
2021-01-22 14:08 ` Kishon Vijay Abraham I
2021-01-04 15:46 ` [PATCH v9 00/17] Implement NTB Controller using multiple PCI EP Kishon Vijay Abraham I
2021-01-18 5:14 ` Kishon Vijay Abraham I
2021-01-19 11:29 ` Lorenzo Pieralisi
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210128121147.GA23564@e121166-lin.cambridge.arm.com \
--to=lorenzo.pieralisi@arm.com \
--cc=allenbh@gmail.com \
--cc=arnd@arndb.de \
--cc=bhelgaas@google.com \
--cc=corbet@lwn.net \
--cc=dave.jiang@intel.com \
--cc=gregkh@linuxfoundation.org \
--cc=helgaas@kernel.org \
--cc=jdmason@kudzu.us \
--cc=kishon@ti.com \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-ntb@googlegroups.com \
--cc=linux-pci@vger.kernel.org \
--cc=robh@kernel.org \
--cc=tjoseph@cadence.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).