From: Serge Semin <fancer.lancer@gmail.com>
To: jdmason@kudzu.us, dave.jiang@intel.com, Allen.Hubbe@emc.com,
Xiangliang.Yu@amd.com
Cc: Sergey.Semin@t-platforms.ru, linux-ntb@googlegroups.com,
linux-kernel@vger.kernel.org,
Serge Semin <fancer.lancer@gmail.com>
Subject: [PATCH v3 7/9] NTB: Add new Memory Windows API documentation
Date: Wed, 14 Dec 2016 02:49:19 +0300 [thread overview]
Message-ID: <1481672961-10753-8-git-send-email-fancer.lancer@gmail.com> (raw)
In-Reply-To: <1481672961-10753-1-git-send-email-fancer.lancer@gmail.com>
Since the new API slightly changes the way a typical NTB client driver
works, the documentation file needs to be appropriately updated.
Acked-by: Allen Hubbe <Allen.Hubbe@dell.com>
Signed-off-by: Serge Semin <fancer.lancer@gmail.com>
---
Documentation/ntb.txt | 99 ++++++++++++++++++++++++++++++++++++++++++++++-----
1 file changed, 91 insertions(+), 8 deletions(-)
diff --git a/Documentation/ntb.txt b/Documentation/ntb.txt
index 1d9bbab..a5af4f0 100644
--- a/Documentation/ntb.txt
+++ b/Documentation/ntb.txt
@@ -1,14 +1,16 @@
# NTB Drivers
NTB (Non-Transparent Bridge) is a type of PCI-Express bridge chip that connects
-the separate memory systems of two computers to the same PCI-Express fabric.
-Existing NTB hardware supports a common feature set, including scratchpad
-registers, doorbell registers, and memory translation windows. Scratchpad
-registers are read-and-writable registers that are accessible from either side
-of the device, so that peers can exchange a small amount of information at a
-fixed address. Doorbell registers provide a way for peers to send interrupt
-events. Memory windows allow translated read and write access to the peer
-memory.
+the separate memory systems of two or more computers to the same PCI-Express
+fabric. Existing NTB hardware supports a common feature set: doorbell
+registers and memory translation windows, as well as non common features like
+scratchpad and message registers. Scratchpad registers are read-and-writable
+registers that are accessible from either side of the device, so that peers can
+exchange a small amount of information at a fixed address. Message registers can
+be utilized for the same purpose. Additionally they are provided with with
+special status bits to make sure the information isn't rewritten by another
+peer. Doorbell registers provide a way for peers to send interrupt events.
+Memory windows allow translated read and write access to the peer memory.
## NTB Core Driver (ntb)
@@ -26,6 +28,87 @@ as ntb hardware, or hardware drivers, are inserted and removed. The
registration uses the Linux Device framework, so it should feel familiar to
anyone who has written a pci driver.
+### NTB Typical client driver implementation
+
+Primary purpose of NTB is to share some peace of memory between at least two
+systems. So the NTB device features like Scratchpad/Message registers are
+mainly used to perform the proper memory window initialization. Typically
+there are two types of memory window interfaces supported by the NTB API:
+inbound translation configured on the local ntb port and outbound translation
+configured by the peer, on the peer ntb port. The first type is
+depicted on the next figure
+
+Inbound translation:
+ Memory: Local NTB Port: Peer NTB Port: Peer MMIO:
+ ____________
+ | dma-mapped |-ntb_mw_set_trans(addr) |
+ | memory | _v____________ | ______________
+ | (addr) |<======| MW xlat addr |<====| MW base addr |<== memory-mapped IO
+ |------------| |--------------| | |--------------|
+
+So typical scenario of the first type memory window initialization looks:
+1) allocate a memory region, 2) put translated address to NTB config,
+3) somehow notify a peer device of performed initialization, 4) peer device
+maps corresponding outbound memory window so to have access to the shared
+memory region.
+
+The second type of interface, that implies the shared windows being
+initialized by a peer device, is depicted on the figure:
+
+Outbound translation:
+ Memory: Local NTB Port: Peer NTB Port: Peer MMIO:
+ ____________ ______________
+ | dma-mapped | | | MW base addr |<== memory-mapped IO
+ | memory | | |--------------|
+ | (addr) |<===================| MW xlat addr |<-ntb_peer_mw_set_trans(addr)
+ |------------| | |--------------|
+
+Typical scenario of the second type interface initialization would be:
+1) allocate a memory region, 2) somehow deliver a translated address to a peer
+device, 3) peer puts the translated address to NTB config, 4) peer device maps
+outbound memory window so to have access to the shared memory region.
+
+As one can see the described scenarios can be combined in one portable
+algorithm.
+ Local device:
+ 1) Allocate memory for a shared window
+ 2) Initialize memory window by translated address of the allocated region
+ (it may fail if local memory window initialization is unsupported)
+ 3) Send the translated address and memory window index to a peer device
+ Peer device:
+ 1) Initialize memory window with retrieved address of the allocated
+ by another device memory region (it may fail if peer memory window
+ initialization is unsupported)
+ 2) Map outbound memory window
+
+In accordance with this scenario, the NTB Memory Window API can be used as
+follows:
+ Local device:
+ 1) ntb_mw_count(pidx) - retrieve number of memory ranges, which can
+ be allocated for memory windows between local device and peer device
+ of port with specified index.
+ 2) ntb_get_align(pidx, midx) - retrieve parameters restricting the
+ shared memory region alignment and size. Then memory can be properly
+ allocated.
+ 3) Allocate physically contiguous memory region in compliance with
+ restrictions retrieved in 2).
+ 4) ntb_mw_set_trans(pidx, midx) - try to set translation address of
+ the memory window with specified index for the defined peer device
+ (it may fail if local translated address setting is not supported)
+ 5) Send translated base address (usually together with memory window
+ number) to the peer device using, for instance, scratchpad or message
+ registers.
+ Peer device:
+ 1) ntb_peer_mw_set_trans(pidx, midx) - try to set received from other
+ device (related to pidx) translated address for specified memory
+ window. It may fail if retrieved address, for instance, exceeds
+ maximum possible address or isn't properly aligned.
+ 2) ntb_peer_mw_get_addr(widx) - retrieve MMIO address to map the memory
+ window so to have an access to the shared memory.
+
+Also it is worth to note, that method ntb_mw_count(pidx) should return the
+same value as ntb_peer_mw_count() on the peer with port index - pidx.
+
### NTB Transport Client (ntb\_transport) and NTB Netdev (ntb\_netdev)
The primary client for NTB is the Transport client, used in tandem with NTB
--
2.6.6
next prev parent reply other threads:[~2016-12-13 23:50 UTC|newest]
Thread overview: 61+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-11-29 17:15 [PATCH 00/22] NTB: Alter kernel API to support multi-port devices Serge Semin
2016-11-29 17:15 ` [PATCH 01/22] NTB: Move link state API being first in sources Serge Semin
2016-11-29 17:15 ` [PATCH 02/22] NTB: Add peer indexed ports NTB API Serge Semin
2016-11-30 18:40 ` kbuild test robot
2016-11-30 19:12 ` kbuild test robot
2016-11-30 20:04 ` kbuild test robot
2016-11-29 17:15 ` [PATCH 03/22] NTB: Alter NTB API to support both inbound and outbound MW based interfaces Serge Semin
2016-11-30 18:54 ` kbuild test robot
2016-11-30 19:46 ` kbuild test robot
2016-11-29 17:15 ` [PATCH 04/22] NTB: Add messaging NTB API Serge Semin
2016-11-29 17:16 ` [PATCH 05/22] NTB: Alter Scratchpads NTB API to support multi-ports interface Serge Semin
2016-11-29 17:16 ` [PATCH 06/22] NTB: Slightly alter link state NTB API Serge Semin
2016-11-29 17:16 ` [PATCH 07/22] NTB: Fix a few ntb.h issues Serge Semin
2016-11-29 17:16 ` [PATCH 08/22] NTB: Add T-Platforms copyrights to NTB API Serge Semin
2016-11-29 17:16 ` [PATCH 09/22] NTB Intel: Move link-related methods being first in the driver Serge Semin
2016-11-29 17:16 ` [PATCH 10/22] NTB Intel: Add port-related NTB API callback methods Serge Semin
2016-12-07 22:56 ` Allen Hubbe
2016-11-29 17:16 ` [PATCH 11/22] NTB Intel: Alter MW interface to fit new NTB API Serge Semin
2016-11-29 17:16 ` [PATCH 12/22] NTB Intel: Alter Scratchpads " Serge Semin
2016-11-29 17:16 ` [PATCH 13/22] NTB Intel: Add T-Platforms copyrights to Intel NTB driver Serge Semin
2016-11-29 17:16 ` [PATCH 14/22] NTB AMD: Move link-related methods being first in the driver Serge Semin
2016-11-29 17:16 ` [PATCH 15/22] NTB AMD: Add port-related NTB API callback methods Serge Semin
2016-11-29 17:16 ` [PATCH 16/22] NTB AMD: Alter MW interface to fit new NTB API Serge Semin
2016-11-29 17:16 ` [PATCH 17/22] NTB AMD: Alter Scratchpads " Serge Semin
2016-11-29 17:16 ` [PATCH 18/22] NTB AMD: Add T-Platforms copyrights to AMD NTB driver Serge Semin
2016-11-29 17:16 ` [PATCH 19/22] NTB PingPong: Alter driver to work with two-ports NTB API Serge Semin
2016-11-29 17:16 ` [PATCH 20/22] NTB Tool: " Serge Semin
2016-11-29 17:16 ` [PATCH 21/22] NTB Perf: " Serge Semin
2016-11-29 17:16 ` [PATCH 22/22] NTB Transport: " Serge Semin
2016-12-12 21:08 ` [PATCH v2 0/9] NTB: Alter kernel API to support multi-port devices Serge Semin
2016-12-12 21:08 ` [PATCH v2 1/9] NTB: Make link-state API being declared first Serge Semin
2016-12-12 21:08 ` [PATCH v2 2/9] NTB: Add indexed ports NTB API Serge Semin
2016-12-12 21:08 ` [PATCH v2 3/9] NTB: Alter link-state API to support multi-port devices Serge Semin
2016-12-12 21:08 ` [PATCH v2 4/9] NTB: Alter MW API to support multi-ports devices Serge Semin
2016-12-12 21:08 ` [PATCH v2 5/9] NTB: Alter Scratchpads " Serge Semin
2016-12-12 21:08 ` [PATCH v2 6/9] NTB: Add Messaging NTB API Serge Semin
2016-12-12 21:08 ` [PATCH v2 7/9] NTB: Add new Memory Windows API documentation Serge Semin
2016-12-12 21:08 ` [PATCH v2 8/9] NTB: Add PCIe Gen4 link speed Serge Semin
2016-12-12 21:08 ` [PATCH v2 9/9] NTB: Add ntb.h comments Serge Semin
2016-12-13 23:49 ` [PATCH v3 0/9] NTB: Alter kernel API to support multi-port devices Serge Semin
2016-12-13 23:49 ` [PATCH v3 1/9] NTB: Make link-state API being declared first Serge Semin
2016-12-14 7:07 ` Allen Hubbe
2016-12-13 23:49 ` [PATCH v3 2/9] NTB: Add indexed ports NTB API Serge Semin
2016-12-14 7:07 ` Allen Hubbe
2016-12-13 23:49 ` [PATCH v3 3/9] NTB: Alter link-state API to support multi-port devices Serge Semin
2016-12-13 23:49 ` [PATCH v3 4/9] NTB: Alter MW API to support multi-ports devices Serge Semin
2016-12-14 7:08 ` Allen Hubbe
2017-01-11 0:11 ` Serge Semin
2016-12-13 23:49 ` [PATCH v3 5/9] NTB: Alter Scratchpads " Serge Semin
2016-12-14 7:08 ` Allen Hubbe
2017-01-11 0:13 ` Serge Semin
2017-02-01 20:01 ` Jon Mason
2016-12-13 23:49 ` [PATCH v3 6/9] NTB: Add Messaging NTB API Serge Semin
2016-12-20 9:48 ` Serge Semin
2016-12-13 23:49 ` Serge Semin [this message]
2016-12-13 23:49 ` [PATCH v3 8/9] NTB: Add PCIe Gen4 link speed Serge Semin
2016-12-13 23:49 ` [PATCH v3 9/9] NTB: Add ntb.h comments Serge Semin
2016-12-14 7:09 ` Allen Hubbe
2016-12-14 7:16 ` Serge Semin
2016-12-14 7:19 ` Allen Hubbe
2016-12-20 9:50 ` Serge Semin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1481672961-10753-8-git-send-email-fancer.lancer@gmail.com \
--to=fancer.lancer@gmail.com \
--cc=Allen.Hubbe@emc.com \
--cc=Sergey.Semin@t-platforms.ru \
--cc=Xiangliang.Yu@amd.com \
--cc=dave.jiang@intel.com \
--cc=jdmason@kudzu.us \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-ntb@googlegroups.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).