linux-pci.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Kishon Vijay Abraham I <kishon@ti.com>
To: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Cc: Bjorn Helgaas <helgaas@kernel.org>,
	Bjorn Helgaas <bhelgaas@google.com>,
	Jonathan Corbet <corbet@lwn.net>, Arnd Bergmann <arnd@arndb.de>,
	Jon Mason <jdmason@kudzu.us>, Dave Jiang <dave.jiang@intel.com>,
	Allen Hubbe <allenbh@gmail.com>, Tom Joseph <tjoseph@cadence.com>,
	Rob Herring <robh@kernel.org>,
	Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	<linux-pci@vger.kernel.org>, <linux-doc@vger.kernel.org>,
	<linux-kernel@vger.kernel.org>, <linux-ntb@googlegroups.com>
Subject: Re: [PATCH v9 01/17] Documentation: PCI: Add specification for the *PCI NTB* function device
Date: Thu, 28 Jan 2021 19:04:27 +0530	[thread overview]
Message-ID: <f29d75ec-504f-3def-ff64-f420fcd3d10e@ti.com> (raw)
In-Reply-To: <20210128121147.GA23564@e121166-lin.cambridge.arm.com>

Hi Lorenzo,

On 28/01/21 5:41 pm, Lorenzo Pieralisi wrote:
> On Fri, Jan 22, 2021 at 07:48:52PM +0530, Kishon Vijay Abraham I wrote:
>> Hi Bjorn,
>>
>> On 20/01/21 12:04 am, Bjorn Helgaas wrote:
>>> On Mon, Jan 04, 2021 at 08:58:53PM +0530, Kishon Vijay Abraham I wrote:
>>>> Add specification for the *PCI NTB* function device. The endpoint function
>>>> driver and the host PCI driver should be created based on this
>>>> specification.
>>>>
>>>> Signed-off-by: Kishon Vijay Abraham I <kishon@ti.com>
>>>
>>> A few typos below if there's opportunity for revisions.
>>
>> I'll fix them.
> 
> Hi Kishon,
> 
> if you have changes please send them along and I will re-merge the
> whole series.

I'll resend by tomorrow. Hope that's fine.

Thank You,
Kishon

> 
> Thanks,
> Lorenzo
> 
>>>> ---
>>>>  Documentation/PCI/endpoint/index.rst          |   1 +
>>>>  .../PCI/endpoint/pci-ntb-function.rst         | 351 ++++++++++++++++++
>>>>  2 files changed, 352 insertions(+)
>>>>  create mode 100644 Documentation/PCI/endpoint/pci-ntb-function.rst
>>>>
>>>> diff --git a/Documentation/PCI/endpoint/index.rst b/Documentation/PCI/endpoint/index.rst
>>>> index 4ca7439fbfc9..ef6861128506 100644
>>>> --- a/Documentation/PCI/endpoint/index.rst
>>>> +++ b/Documentation/PCI/endpoint/index.rst
>>>> @@ -11,5 +11,6 @@ PCI Endpoint Framework
>>>>     pci-endpoint-cfs
>>>>     pci-test-function
>>>>     pci-test-howto
>>>> +   pci-ntb-function
>>>>  
>>>>     function/binding/pci-test
>>>> diff --git a/Documentation/PCI/endpoint/pci-ntb-function.rst b/Documentation/PCI/endpoint/pci-ntb-function.rst
>>>> new file mode 100644
>>>> index 000000000000..a57908be4047
>>>> --- /dev/null
>>>> +++ b/Documentation/PCI/endpoint/pci-ntb-function.rst
>>>> @@ -0,0 +1,351 @@
>>>> +.. SPDX-License-Identifier: GPL-2.0
>>>> +
>>>> +=================
>>>> +PCI NTB Function
>>>> +=================
>>>> +
>>>> +:Author: Kishon Vijay Abraham I <kishon@ti.com>
>>>> +
>>>> +PCI Non Transparent Bridges (NTB) allow two host systems to communicate
>>>> +with each other by exposing each host as a device to the other host.
>>>> +NTBs typically support the ability to generate interrupts on the remote
>>>> +machine, expose memory ranges as BARs and perform DMA.  They also support
>>>> +scratchpads which are areas of memory within the NTB that are accessible
>>>> +from both machines.
>>>> +
>>>> +PCI NTB Function allows two different systems (or hosts) to communicate
>>>> +with each other by configurig the endpoint instances in such a way that
>>>> +transactions from one system is routed to the other system.
>>>
>>> s/is/are/
>>>
>>>> +In the below diagram, PCI NTB function configures the SoC with multiple
>>>> +PCIe Endpoint (EP) instances in such a way that transaction from one EP
>>>> +controller is routed to the other EP controller. Once PCI NTB function
>>>
>>> s/transaction ... is/transactions ... are/
>>>
>>>> +configures the SoC with multiple EP instances, HOST1 and HOST2 can
>>>> +communicate with each other using SoC as a bridge.
>>>> +
>>>> +.. code-block:: text
>>>> +
>>>> +    +-------------+                                   +-------------+
>>>> +    |             |                                   |             |
>>>> +    |    HOST1    |                                   |    HOST2    |
>>>> +    |             |                                   |             |
>>>> +    +------^------+                                   +------^------+
>>>> +           |                                                 |
>>>> +           |                                                 |
>>>> + +---------|-------------------------------------------------|---------+
>>>> + |  +------v------+                                   +------v------+  |
>>>> + |  |             |                                   |             |  |
>>>> + |  |     EP      |                                   |     EP      |  |
>>>> + |  | CONTROLLER1 |                                   | CONTROLLER2 |  |
>>>> + |  |             <----------------------------------->             |  |
>>>> + |  |             |                                   |             |  |
>>>> + |  |             |                                   |             |  |
>>>> + |  |             |  SoC With Multiple EP Instances   |             |  |
>>>> + |  |             |  (Configured using NTB Function)  |             |  |
>>>> + |  +-------------+                                   +-------------+  |
>>>> + +---------------------------------------------------------------------+
>>>> +
>>>> +Constructs used for Implementing NTB
>>>> +====================================
>>>> +
>>>> +	1) Config Region
>>>> +	2) Self Scratchpad Registers
>>>> +	3) Peer Scratchpad Registers
>>>> +	4) Doorbell Registers
>>>> +	5) Memory Window
>>>> +
>>>> +
>>>> +Config Region:
>>>> +--------------
>>>> +
>>>> +Config Region is a construct that is specific to NTB implemented using NTB
>>>> +Endpoint Function Driver. The host and endpoint side NTB function driver will
>>>> +exchange information with each other using this region. Config Region has
>>>> +Control/Status Registers for configuring the Endpoint Controller. Host can
>>>> +write into this region for configuring the outbound ATU and to indicate the
>>>
>>> Expand "ATU" since this is the first mention.
>>>
>>>> +link status. Endpoint can indicate the status of commands issued be host in
>>>> +this region. Endpoint can also indicate the scratchpad offset, number of
>>>> +memory windows to the host using this region.
>>>
>>> s/be host/by host/
>>> s/offset, number/offset and number/
>>>
>>>> +The format of Config Region is given below. Each of the fields here are 32
>>>> +bits.
>>>
>>> s/Each ... are/All ... are/
>>>
>>>> +
>>>> +.. code-block:: text
>>>> +
>>>> +	+------------------------+
>>>> +	|         COMMAND        |
>>>> +	+------------------------+
>>>> +	|         ARGUMENT       |
>>>> +	+------------------------+
>>>> +	|         STATUS         |
>>>> +	+------------------------+
>>>> +	|         TOPOLOGY       |
>>>> +	+------------------------+
>>>> +	|    ADDRESS (LOWER 32)  |
>>>> +	+------------------------+
>>>> +	|    ADDRESS (UPPER 32)  |
>>>> +	+------------------------+
>>>> +	|           SIZE         |
>>>> +	+------------------------+
>>>> +	|   NO OF MEMORY WINDOW  |
>>>> +	+------------------------+
>>>> +	|  MEMORY WINDOW1 OFFSET |
>>>> +	+------------------------+
>>>> +	|       SPAD OFFSET      |
>>>> +	+------------------------+
>>>> +	|        SPAD COUNT      |
>>>> +	+------------------------+
>>>> +	|      DB ENTRY SIZE     |
>>>> +	+------------------------+
>>>> +	|         DB DATA        |
>>>> +	+------------------------+
>>>> +	|            :           |
>>>> +	+------------------------+
>>>> +	|            :           |
>>>> +	+------------------------+
>>>> +	|         DB DATA        |
>>>> +	+------------------------+
>>>> +
>>>> +
>>>> +  COMMAND:
>>>> +
>>>> +	NTB function supports three commands:
>>>> +
>>>> +	  CMD_CONFIGURE_DOORBELL (0x1): Command to configure doorbell. Before
>>>> +	invoking this command, the host should allocate and initialize
>>>> +	MSI/MSI-X vectors (i.e initialize the MSI/MSI-X capability in the
>>>
>>> s/i.e/i.e.,/
>>>
>>>> +	Endpoint). The endpoint on receiving this command will configure
>>>> +	the outbound ATU such that transaction to DB BAR will be routed
>>>> +	to the MSI/MSI-X address programmed by the host. The ARGUMENT
>>>
>>> s/transaction to/transactions to/
>>>
>>> Expand "DB BAR".  I assume this refers to "Doorbell BAR" (which itself
>>> is not defined).  How do we know which is the Doorbell BAR?
>>
>> right doorbell. That part is explained in the "Modeling Constructs"
>> section below.
>>>
>>> Also, "DB" itself needs to be expanded somehow for uses like below:
>>>
>>>> +	register should be populated with number of DBs to configure (in the
>>>> +	lower 16 bits) and if MSI or MSI-X should be configured (BIT 16).
>>>> +	(TODO: Add support for MSI-X).
>>>> +
>>>> +	  CMD_CONFIGURE_MW (0x2): Command to configure memory window. The
>>>> +	host invokes this command after allocating a buffer that can be
>>>> +	accessed by remote host. The allocated address should be programmed
>>>> +	in the ADDRESS register (64 bit), the size should be programmed in
>>>> +	the SIZE register and the memory window index should be programmed
>>>> +	in the ARGUMENT register. The endpoint on receiving this command
>>>> +	will configure the outbound ATU such that trasaction to MW BAR
>>>> +	will be routed to the address provided by the host.
>>>
>>> How do we know which is the MW BAR?  I assume "MW" refers to "Memory
>>> Window".
>>
>> right memory window. That's again explained in the "Modeling Constructs"
>> section below.
>>>
>>>> +
>>>> +	  CMD_LINK_UP (0x3): Command to indicate an NTB application is
>>>> +	bound to the EP device on the host side. Once the endpoint
>>>> +	receives this command from both the hosts, the endpoint will
>>>> +	raise an LINK_UP event to both the hosts to indicate the hosts
>>>> +	can start communicating with each other.
>>>
>>> s/raise an/raise a/
>>>
>>> I guess this "LINK_UP event" is something other than the PCIe DL_Up
>>> state, because each host has already been communicating with the
>>> endpoint.  Right?  Is this LINK_UP a software construct?
>>
>> Yeah. This is when an NTB client application is bound to the NTB device.
>> This is used for handshake between the applications running on the two
>> hosts.
>>
>> Thanks
>> Kishon

  reply	other threads:[~2021-01-28 13:35 UTC|newest]

Thread overview: 34+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-01-04 15:28 [PATCH v9 00/17] Implement NTB Controller using multiple PCI EP Kishon Vijay Abraham I
2021-01-04 15:28 ` [PATCH v9 01/17] Documentation: PCI: Add specification for the *PCI NTB* function device Kishon Vijay Abraham I
2021-01-19 18:34   ` Bjorn Helgaas
2021-01-22 14:18     ` Kishon Vijay Abraham I
2021-01-28 12:11       ` Lorenzo Pieralisi
2021-01-28 13:34         ` Kishon Vijay Abraham I [this message]
2021-01-29  1:05   ` Randy Dunlap
2021-01-29  5:10     ` Kishon Vijay Abraham I
2021-01-04 15:28 ` [PATCH v9 02/17] PCI: endpoint: Make *_get_first_free_bar() take into account 64 bit BAR Kishon Vijay Abraham I
2021-01-04 15:28 ` [PATCH v9 03/17] PCI: endpoint: Add helper API to get the 'next' unreserved BAR Kishon Vijay Abraham I
2021-01-04 15:28 ` [PATCH v9 04/17] PCI: endpoint: Make *_free_bar() to return error codes on failure Kishon Vijay Abraham I
2021-01-04 15:28 ` [PATCH v9 05/17] PCI: endpoint: Remove unused pci_epf_match_device() Kishon Vijay Abraham I
2021-01-04 15:28 ` [PATCH v9 06/17] PCI: endpoint: Add support to associate secondary EPC with EPF Kishon Vijay Abraham I
2021-01-04 15:28 ` [PATCH v9 07/17] PCI: endpoint: Add support in configfs to associate two EPCs " Kishon Vijay Abraham I
2021-01-04 15:29 ` [PATCH v9 08/17] PCI: endpoint: Add pci_epc_ops to map MSI irq Kishon Vijay Abraham I
2021-01-04 15:29 ` [PATCH v9 09/17] PCI: endpoint: Add pci_epf_ops for epf drivers to expose function specific attrs Kishon Vijay Abraham I
2021-01-04 15:29 ` [PATCH v9 10/17] PCI: endpoint: Allow user to create sub-directory of 'EPF Device' directory Kishon Vijay Abraham I
2021-01-19 18:34   ` Bjorn Helgaas
2021-01-19 18:35     ` Bjorn Helgaas
2021-01-19 18:38     ` Greg Kroah-Hartman
2021-01-04 15:29 ` [PATCH v9 11/17] PCI: cadence: Implement ->msi_map_irq() ops Kishon Vijay Abraham I
2021-01-04 15:29 ` [PATCH v9 12/17] PCI: cadence: Configure LM_EP_FUNC_CFG based on epc->function_num_map Kishon Vijay Abraham I
2021-01-04 15:29 ` [PATCH v9 13/17] PCI: endpoint: Add EP function driver to provide NTB functionality Kishon Vijay Abraham I
2021-01-19 19:04   ` Bjorn Helgaas
2021-01-04 15:29 ` [PATCH v9 14/17] PCI: Add TI J721E device to pci ids Kishon Vijay Abraham I
2021-01-04 15:29 ` [PATCH v9 15/17] NTB: Add support for EPF PCI-Express Non-Transparent Bridge Kishon Vijay Abraham I
2021-01-18 17:12   ` Dave Jiang
2021-01-04 15:29 ` [PATCH v9 16/17] Documentation: PCI: Add configfs binding documentation for pci-ntb endpoint function Kishon Vijay Abraham I
2021-01-04 15:29 ` [PATCH v9 17/17] Documentation: PCI: Add userguide for PCI endpoint NTB function Kishon Vijay Abraham I
2021-01-19 18:34   ` Bjorn Helgaas
2021-01-22 14:08     ` Kishon Vijay Abraham I
2021-01-04 15:46 ` [PATCH v9 00/17] Implement NTB Controller using multiple PCI EP Kishon Vijay Abraham I
2021-01-18  5:14   ` Kishon Vijay Abraham I
2021-01-19 11:29 ` Lorenzo Pieralisi

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=f29d75ec-504f-3def-ff64-f420fcd3d10e@ti.com \
    --to=kishon@ti.com \
    --cc=allenbh@gmail.com \
    --cc=arnd@arndb.de \
    --cc=bhelgaas@google.com \
    --cc=corbet@lwn.net \
    --cc=dave.jiang@intel.com \
    --cc=gregkh@linuxfoundation.org \
    --cc=helgaas@kernel.org \
    --cc=jdmason@kudzu.us \
    --cc=linux-doc@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-ntb@googlegroups.com \
    --cc=linux-pci@vger.kernel.org \
    --cc=lorenzo.pieralisi@arm.com \
    --cc=robh@kernel.org \
    --cc=tjoseph@cadence.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).