ntb.lists.linux.dev archive mirror
 help / color / mirror / Atom feed
From: Kishon Vijay Abraham I <kishon@ti.com>
To: Bjorn Helgaas <helgaas@kernel.org>
Cc: Bjorn Helgaas <bhelgaas@google.com>,
	Jonathan Corbet <corbet@lwn.net>,
	Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>,
	Arnd Bergmann <arnd@arndb.de>, Jon Mason <jdmason@kudzu.us>,
	Dave Jiang <dave.jiang@intel.com>,
	Allen Hubbe <allenbh@gmail.com>, Tom Joseph <tjoseph@cadence.com>,
	Rob Herring <robh@kernel.org>,
	Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	linux-pci@vger.kernel.org, linux-doc@vger.kernel.org,
	linux-kernel@vger.kernel.org, linux-ntb@googlegroups.com
Subject: Re: [PATCH v9 01/17] Documentation: PCI: Add specification for the *PCI NTB* function device
Date: Fri, 22 Jan 2021 19:48:52 +0530	[thread overview]
Message-ID: <797ec9f2-34c3-5dc4-cc0a-d4f7cdf4afb0@ti.com> (raw)
In-Reply-To: <20210119181106.GA2493893@bjorn-Precision-5520>

Hi Bjorn,

On 20/01/21 12:04 am, Bjorn Helgaas wrote:
> On Mon, Jan 04, 2021 at 08:58:53PM +0530, Kishon Vijay Abraham I wrote:
>> Add specification for the *PCI NTB* function device. The endpoint function
>> driver and the host PCI driver should be created based on this
>> specification.
>>
>> Signed-off-by: Kishon Vijay Abraham I <kishon@ti.com>
> 
> A few typos below if there's opportunity for revisions.

I'll fix them.
> 
>> ---
>>  Documentation/PCI/endpoint/index.rst          |   1 +
>>  .../PCI/endpoint/pci-ntb-function.rst         | 351 ++++++++++++++++++
>>  2 files changed, 352 insertions(+)
>>  create mode 100644 Documentation/PCI/endpoint/pci-ntb-function.rst
>>
>> diff --git a/Documentation/PCI/endpoint/index.rst b/Documentation/PCI/endpoint/index.rst
>> index 4ca7439fbfc9..ef6861128506 100644
>> --- a/Documentation/PCI/endpoint/index.rst
>> +++ b/Documentation/PCI/endpoint/index.rst
>> @@ -11,5 +11,6 @@ PCI Endpoint Framework
>>     pci-endpoint-cfs
>>     pci-test-function
>>     pci-test-howto
>> +   pci-ntb-function
>>  
>>     function/binding/pci-test
>> diff --git a/Documentation/PCI/endpoint/pci-ntb-function.rst b/Documentation/PCI/endpoint/pci-ntb-function.rst
>> new file mode 100644
>> index 000000000000..a57908be4047
>> --- /dev/null
>> +++ b/Documentation/PCI/endpoint/pci-ntb-function.rst
>> @@ -0,0 +1,351 @@
>> +.. SPDX-License-Identifier: GPL-2.0
>> +
>> +=================
>> +PCI NTB Function
>> +=================
>> +
>> +:Author: Kishon Vijay Abraham I <kishon@ti.com>
>> +
>> +PCI Non Transparent Bridges (NTB) allow two host systems to communicate
>> +with each other by exposing each host as a device to the other host.
>> +NTBs typically support the ability to generate interrupts on the remote
>> +machine, expose memory ranges as BARs and perform DMA.  They also support
>> +scratchpads which are areas of memory within the NTB that are accessible
>> +from both machines.
>> +
>> +PCI NTB Function allows two different systems (or hosts) to communicate
>> +with each other by configurig the endpoint instances in such a way that
>> +transactions from one system is routed to the other system.
> 
> s/is/are/
> 
>> +In the below diagram, PCI NTB function configures the SoC with multiple
>> +PCIe Endpoint (EP) instances in such a way that transaction from one EP
>> +controller is routed to the other EP controller. Once PCI NTB function
> 
> s/transaction ... is/transactions ... are/
> 
>> +configures the SoC with multiple EP instances, HOST1 and HOST2 can
>> +communicate with each other using SoC as a bridge.
>> +
>> +.. code-block:: text
>> +
>> +    +-------------+                                   +-------------+
>> +    |             |                                   |             |
>> +    |    HOST1    |                                   |    HOST2    |
>> +    |             |                                   |             |
>> +    +------^------+                                   +------^------+
>> +           |                                                 |
>> +           |                                                 |
>> + +---------|-------------------------------------------------|---------+
>> + |  +------v------+                                   +------v------+  |
>> + |  |             |                                   |             |  |
>> + |  |     EP      |                                   |     EP      |  |
>> + |  | CONTROLLER1 |                                   | CONTROLLER2 |  |
>> + |  |             <----------------------------------->             |  |
>> + |  |             |                                   |             |  |
>> + |  |             |                                   |             |  |
>> + |  |             |  SoC With Multiple EP Instances   |             |  |
>> + |  |             |  (Configured using NTB Function)  |             |  |
>> + |  +-------------+                                   +-------------+  |
>> + +---------------------------------------------------------------------+
>> +
>> +Constructs used for Implementing NTB
>> +====================================
>> +
>> +	1) Config Region
>> +	2) Self Scratchpad Registers
>> +	3) Peer Scratchpad Registers
>> +	4) Doorbell Registers
>> +	5) Memory Window
>> +
>> +
>> +Config Region:
>> +--------------
>> +
>> +Config Region is a construct that is specific to NTB implemented using NTB
>> +Endpoint Function Driver. The host and endpoint side NTB function driver will
>> +exchange information with each other using this region. Config Region has
>> +Control/Status Registers for configuring the Endpoint Controller. Host can
>> +write into this region for configuring the outbound ATU and to indicate the
> 
> Expand "ATU" since this is the first mention.
> 
>> +link status. Endpoint can indicate the status of commands issued be host in
>> +this region. Endpoint can also indicate the scratchpad offset, number of
>> +memory windows to the host using this region.
> 
> s/be host/by host/
> s/offset, number/offset and number/
> 
>> +The format of Config Region is given below. Each of the fields here are 32
>> +bits.
> 
> s/Each ... are/All ... are/
> 
>> +
>> +.. code-block:: text
>> +
>> +	+------------------------+
>> +	|         COMMAND        |
>> +	+------------------------+
>> +	|         ARGUMENT       |
>> +	+------------------------+
>> +	|         STATUS         |
>> +	+------------------------+
>> +	|         TOPOLOGY       |
>> +	+------------------------+
>> +	|    ADDRESS (LOWER 32)  |
>> +	+------------------------+
>> +	|    ADDRESS (UPPER 32)  |
>> +	+------------------------+
>> +	|           SIZE         |
>> +	+------------------------+
>> +	|   NO OF MEMORY WINDOW  |
>> +	+------------------------+
>> +	|  MEMORY WINDOW1 OFFSET |
>> +	+------------------------+
>> +	|       SPAD OFFSET      |
>> +	+------------------------+
>> +	|        SPAD COUNT      |
>> +	+------------------------+
>> +	|      DB ENTRY SIZE     |
>> +	+------------------------+
>> +	|         DB DATA        |
>> +	+------------------------+
>> +	|            :           |
>> +	+------------------------+
>> +	|            :           |
>> +	+------------------------+
>> +	|         DB DATA        |
>> +	+------------------------+
>> +
>> +
>> +  COMMAND:
>> +
>> +	NTB function supports three commands:
>> +
>> +	  CMD_CONFIGURE_DOORBELL (0x1): Command to configure doorbell. Before
>> +	invoking this command, the host should allocate and initialize
>> +	MSI/MSI-X vectors (i.e initialize the MSI/MSI-X capability in the
> 
> s/i.e/i.e.,/
> 
>> +	Endpoint). The endpoint on receiving this command will configure
>> +	the outbound ATU such that transaction to DB BAR will be routed
>> +	to the MSI/MSI-X address programmed by the host. The ARGUMENT
> 
> s/transaction to/transactions to/
> 
> Expand "DB BAR".  I assume this refers to "Doorbell BAR" (which itself
> is not defined).  How do we know which is the Doorbell BAR?

right doorbell. That part is explained in the "Modeling Constructs"
section below.
> 
> Also, "DB" itself needs to be expanded somehow for uses like below:
> 
>> +	register should be populated with number of DBs to configure (in the
>> +	lower 16 bits) and if MSI or MSI-X should be configured (BIT 16).
>> +	(TODO: Add support for MSI-X).
>> +
>> +	  CMD_CONFIGURE_MW (0x2): Command to configure memory window. The
>> +	host invokes this command after allocating a buffer that can be
>> +	accessed by remote host. The allocated address should be programmed
>> +	in the ADDRESS register (64 bit), the size should be programmed in
>> +	the SIZE register and the memory window index should be programmed
>> +	in the ARGUMENT register. The endpoint on receiving this command
>> +	will configure the outbound ATU such that trasaction to MW BAR
>> +	will be routed to the address provided by the host.
> 
> How do we know which is the MW BAR?  I assume "MW" refers to "Memory
> Window".

right memory window. That's again explained in the "Modeling Constructs"
section below.
> 
>> +
>> +	  CMD_LINK_UP (0x3): Command to indicate an NTB application is
>> +	bound to the EP device on the host side. Once the endpoint
>> +	receives this command from both the hosts, the endpoint will
>> +	raise an LINK_UP event to both the hosts to indicate the hosts
>> +	can start communicating with each other.
> 
> s/raise an/raise a/
> 
> I guess this "LINK_UP event" is something other than the PCIe DL_Up
> state, because each host has already been communicating with the
> endpoint.  Right?  Is this LINK_UP a software construct?

Yeah. This is when an NTB client application is bound to the NTB device.
This is used for handshake between the applications running on the two
hosts.

Thanks
Kishon

  reply	other threads:[~2021-01-22 14:19 UTC|newest]

Thread overview: 34+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-01-04 15:28 [PATCH v9 00/17] Implement NTB Controller using multiple PCI EP Kishon Vijay Abraham I
2021-01-04 15:28 ` [PATCH v9 01/17] Documentation: PCI: Add specification for the *PCI NTB* function device Kishon Vijay Abraham I
2021-01-19 18:34   ` Bjorn Helgaas
2021-01-22 14:18     ` Kishon Vijay Abraham I [this message]
2021-01-28 12:11       ` Lorenzo Pieralisi
2021-01-28 13:34         ` Kishon Vijay Abraham I
2021-01-29  1:05   ` Randy Dunlap
2021-01-29  5:10     ` Kishon Vijay Abraham I
2021-01-04 15:28 ` [PATCH v9 02/17] PCI: endpoint: Make *_get_first_free_bar() take into account 64 bit BAR Kishon Vijay Abraham I
2021-01-04 15:28 ` [PATCH v9 03/17] PCI: endpoint: Add helper API to get the 'next' unreserved BAR Kishon Vijay Abraham I
2021-01-04 15:28 ` [PATCH v9 04/17] PCI: endpoint: Make *_free_bar() to return error codes on failure Kishon Vijay Abraham I
2021-01-04 15:28 ` [PATCH v9 05/17] PCI: endpoint: Remove unused pci_epf_match_device() Kishon Vijay Abraham I
2021-01-04 15:28 ` [PATCH v9 06/17] PCI: endpoint: Add support to associate secondary EPC with EPF Kishon Vijay Abraham I
2021-01-04 15:28 ` [PATCH v9 07/17] PCI: endpoint: Add support in configfs to associate two EPCs " Kishon Vijay Abraham I
2021-01-04 15:29 ` [PATCH v9 08/17] PCI: endpoint: Add pci_epc_ops to map MSI irq Kishon Vijay Abraham I
2021-01-04 15:29 ` [PATCH v9 09/17] PCI: endpoint: Add pci_epf_ops for epf drivers to expose function specific attrs Kishon Vijay Abraham I
2021-01-04 15:29 ` [PATCH v9 10/17] PCI: endpoint: Allow user to create sub-directory of 'EPF Device' directory Kishon Vijay Abraham I
2021-01-19 18:34   ` Bjorn Helgaas
2021-01-19 18:35     ` Bjorn Helgaas
2021-01-19 18:38     ` Greg Kroah-Hartman
2021-01-04 15:29 ` [PATCH v9 11/17] PCI: cadence: Implement ->msi_map_irq() ops Kishon Vijay Abraham I
2021-01-04 15:29 ` [PATCH v9 12/17] PCI: cadence: Configure LM_EP_FUNC_CFG based on epc->function_num_map Kishon Vijay Abraham I
2021-01-04 15:29 ` [PATCH v9 13/17] PCI: endpoint: Add EP function driver to provide NTB functionality Kishon Vijay Abraham I
2021-01-19 19:04   ` Bjorn Helgaas
2021-01-04 15:29 ` [PATCH v9 14/17] PCI: Add TI J721E device to pci ids Kishon Vijay Abraham I
2021-01-04 15:29 ` [PATCH v9 15/17] NTB: Add support for EPF PCI-Express Non-Transparent Bridge Kishon Vijay Abraham I
2021-01-18 17:12   ` Dave Jiang
2021-01-04 15:29 ` [PATCH v9 16/17] Documentation: PCI: Add configfs binding documentation for pci-ntb endpoint function Kishon Vijay Abraham I
2021-01-04 15:29 ` [PATCH v9 17/17] Documentation: PCI: Add userguide for PCI endpoint NTB function Kishon Vijay Abraham I
2021-01-19 18:34   ` Bjorn Helgaas
2021-01-22 14:08     ` Kishon Vijay Abraham I
2021-01-04 15:46 ` [PATCH v9 00/17] Implement NTB Controller using multiple PCI EP Kishon Vijay Abraham I
2021-01-18  5:14   ` Kishon Vijay Abraham I
2021-01-19 11:29 ` Lorenzo Pieralisi

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=797ec9f2-34c3-5dc4-cc0a-d4f7cdf4afb0@ti.com \
    --to=kishon@ti.com \
    --cc=allenbh@gmail.com \
    --cc=arnd@arndb.de \
    --cc=bhelgaas@google.com \
    --cc=corbet@lwn.net \
    --cc=dave.jiang@intel.com \
    --cc=gregkh@linuxfoundation.org \
    --cc=helgaas@kernel.org \
    --cc=jdmason@kudzu.us \
    --cc=linux-doc@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-ntb@googlegroups.com \
    --cc=linux-pci@vger.kernel.org \
    --cc=lorenzo.pieralisi@arm.com \
    --cc=robh@kernel.org \
    --cc=tjoseph@cadence.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).