Re: [dpdk-dev] [PATCH v12 0/6] ethdev: introduce Rx buffer split

[Slava Ovsiienko <viacheslavo@nvidia.com>]

> -----Original Message-----
> From: Ferruh Yigit <ferruh.yigit@intel.com>
> Sent: Friday, October 16, 2020 20:05
> To: Slava Ovsiienko <viacheslavo@nvidia.com>; dev@dpdk.org
> Cc: NBU-Contact-Thomas Monjalon <thomas@monjalon.net>;
> stephen@networkplumber.org; olivier.matz@6wind.com;
> jerinjacobk@gmail.com; maxime.coquelin@redhat.com;
> david.marchand@redhat.com; arybchenko@solarflare.com
> Subject: Re: [dpdk-dev] [PATCH v12 0/6] ethdev: introduce Rx buffer split
> 
> On 10/16/2020 5:44 PM, Viacheslav Ovsiienko wrote:
> > The DPDK datapath in the transmit direction is very flexible.
> > An application can build the multi-segment packet and manages almost
> > all data aspects - the memory pools where segments are allocated from,
> > the segment lengths, the memory attributes like external buffers,
> > registered for DMA, etc.
> >
> > In the receiving direction, the datapath is much less flexible, an
> > application can only specify the memory pool to configure the
> > receiving queue and nothing more. In order to extend receiving
> > datapath capabilities it is proposed to add the way to provide
> > extended information how to split the packets being received.
> >
> > The new offload flag RTE_ETH_RX_OFFLOAD_BUFFER_SPLIT in device
> > capabilities is introduced to present the way for PMD to report to
> > application about supporting Rx packet split to configurable segments.
> > Prior invoking the rte_eth_rx_queue_setup() routine application should
> > check RTE_ETH_RX_OFFLOAD_BUFFER_SPLIT flag.
> >
> > The following structure is introduced to specify the Rx packet segment
> > for RTE_ETH_RX_OFFLOAD_BUFFER_SPLIT offload:
> >
> > struct rte_eth_rxseg_split {
> >
> >      struct rte_mempool *mp; /* memory pools to allocate segment from */
> >      uint16_t length; /* segment maximal data length,
> > 		       	configures "split point" */
> >      uint16_t offset; /* data offset from beginning
> > 		       	of mbuf data buffer */
> >      uint32_t reserved; /* reserved field */ };
> >
> > The segment descriptions are added to the rte_eth_rxconf structure:
> >     rx_seg - pointer the array of segment descriptions, each element
> >               describes the memory pool, maximal data length, initial
> >               data offset from the beginning of data buffer in mbuf.
> > 	     This array allows to specify the different settings for
> > 	     each segment in individual fashion.
> >     rx_nseg - number of elements in the array
> >
> > If the extended segment descriptions is provided with these new fields
> > the mp parameter of the rte_eth_rx_queue_setup must be specified as
> > NULL to avoid ambiguity.
> >
> > There are two options to specify Rx buffer configuration:
> > - mp is not NULL, rx_conf.rx_seg is NULL, rx_conf.rx_nseg is zero,
> >    it is compatible configuration, follows existing implementation,
> >    provides single pool and no description for segment sizes
> >    and offsets.
> > - mp is NULL, rx_conf.rx_seg is not NULL, rx_conf.rx_nseg is not
> >    zero, it provides the extended configuration, individually for
> >    each segment.
> >
> > f the Rx queue is configured with new settings the packets being
> > received will be split into multiple segments pushed to the mbufs with
> > specified attributes. The PMD will split the received packets into
> > multiple segments according to the specification in the description
> > array.
> >
> > For example, let's suppose we configured the Rx queue with the
> > following segments:
> >      seg0 - pool0, len0=14B, off0=2
> >      seg1 - pool1, len1=20B, off1=128B
> >      seg2 - pool2, len2=20B, off2=0B
> >      seg3 - pool3, len3=512B, off3=0B
> >
> > The packet 46 bytes long will look like the following:
> >      seg0 - 14B long @ RTE_PKTMBUF_HEADROOM + 2 in mbuf from pool0
> >      seg1 - 20B long @ 128 in mbuf from pool1
> >      seg2 - 12B long @ 0 in mbuf from pool2
> >
> > The packet 1500 bytes long will look like the following:
> >      seg0 - 14B @ RTE_PKTMBUF_HEADROOM + 2 in mbuf from pool0
> >      seg1 - 20B @ 128 in mbuf from pool1
> >      seg2 - 20B @ 0 in mbuf from pool2
> >      seg3 - 512B @ 0 in mbuf from pool3
> >      seg4 - 512B @ 0 in mbuf from pool3
> >      seg5 - 422B @ 0 in mbuf from pool3
> >
> > The offload RTE_ETH_RX_OFFLOAD_SCATTER must be present and
> configured
> > to support new buffer split feature (if rx_nseg is greater than one).
> >
> > The split limitations imposed by underlying PMD is reported in the new
> > introduced rte_eth_dev_info->rx_seg_capa field.
> >
> > The new approach would allow splitting the ingress packets into
> > multiple parts pushed to the memory with different attributes.
> > For example, the packet headers can be pushed to the embedded data
> > buffers within mbufs and the application data into the external
> > buffers attached to mbufs allocated from the different memory pools.
> > The memory attributes for the split parts may differ either - for
> > example the application data may be pushed into the external memory
> > located on the dedicated physical device, say GPU or NVMe. This would
> > improve the DPDK receiving datapath flexibility with preserving
> > compatibility with existing API.
> >
> > Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
> > ---
> >
> > v1:
> > https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpatch
> >
> es.dpdk.org%2Fpatch%2F79594%2F&amp;data=04%7C01%7Cviacheslavo%40n
> vidia
> >
> .com%7C2acfee0f2bf14ffcf9c008d871f5b3fa%7C43083d15727340c1b7db39efd
> 9cc
> >
> c17a%7C0%7C0%7C637384647418293191%7CUnknown%7CTWFpbGZsb3d8eyJ
> WIjoiMC4w
> >
> LjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;
> sdat
> >
> a=78ja3UHYl%2Blyg%2Fjv1hDLqCCy2Gf9WRFyidgrKTnwfbk%3D&amp;reserved
> =0
> > v2:
> https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpatches
> .dpdk.org%2Fpatch%2F79893%2F&amp;data=04%7C01%7Cviacheslavo%40nvi
> dia.com%7C2acfee0f2bf14ffcf9c008d871f5b3fa%7C43083d15727340c1b7db39
> efd9ccc17a%7C0%7C0%7C637384647418293191%7CUnknown%7CTWFpbGZsb
> 3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%
> 3D%7C1000&amp;sdata=rLC8kS6CxFR7IprwFjAL16Po6Exqw%2BtqVn1soNZ3v6
> A%3D&amp;reserved=0
> >      - add feature support to mlx5 PMD
> >
> > v3:
> https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpatches
> .dpdk.org%2Fpatch%2F80389%2F&amp;data=04%7C01%7Cviacheslavo%40nvi
> dia.com%7C2acfee0f2bf14ffcf9c008d871f5b3fa%7C43083d15727340c1b7db39
> efd9ccc17a%7C0%7C0%7C637384647418293191%7CUnknown%7CTWFpbGZsb
> 3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%
> 3D%7C1000&amp;sdata=bLZ4mnluEMojl%2FP%2BmFDBgO8jTHpDoc7Rj362Ws
> mdbJk%3D&amp;reserved=0
> >      - rte_eth_rx_queue_setup_ex is renamed to rte_eth_rxseg_queue_setup
> >      - DEV_RX_OFFLOAD_BUFFER_SPLIT is renamed to
> RTE_ETH_RX_OFFLOAD_BUFFER_SPLIT
> >      - commit message update
> >      - documentaion provided
> >      - release notes update
> >      - minor bug fixes in testpmd related part
> >
> > v4:
> https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpatches
> .dpdk.org%2Fpatch%2F80401%2F&amp;data=04%7C01%7Cviacheslavo%40nvi
> dia.com%7C2acfee0f2bf14ffcf9c008d871f5b3fa%7C43083d15727340c1b7db39
> efd9ccc17a%7C0%7C0%7C637384647418293191%7CUnknown%7CTWFpbGZsb
> 3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%
> 3D%7C1000&amp;sdata=5lbRRML9If9i5mupJ57eX%2BUhfidisNrRHHkhUibjwSs
> %3D&amp;reserved=0
> >      - common part of rx_queue_setup/rxseg_queue_setup
> >
> > v5:
> https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpatches
> .dpdk.org%2Fpatch%2F80609%2F&amp;data=04%7C01%7Cviacheslavo%40nvi
> dia.com%7C2acfee0f2bf14ffcf9c008d871f5b3fa%7C43083d15727340c1b7db39
> efd9ccc17a%7C0%7C0%7C637384647418293191%7CUnknown%7CTWFpbGZsb
> 3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%
> 3D%7C1000&amp;sdata=QqfwQd6DMkrfsS68ENksClFDbRu3VWqoVOdow7AqO
> XU%3D&amp;reserved=0
> >      - refactored to approach of providing split configuration
> >        in the rte_eth_rxconf structure instead of introducing
> >        the new API routine
> >      - added support for rxoffs command to testpmd to
> >        provide segment offsets for complete testing of split
> >        configurations
> >      - patchset is split into two parts - PMD part will
> >        be presented as separate series
> >
> > v6:
> https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpatches
> .dpdk.org%2Fpatch%2F80785%2F&amp;data=04%7C01%7Cviacheslavo%40nvi
> dia.com%7C2acfee0f2bf14ffcf9c008d871f5b3fa%7C43083d15727340c1b7db39
> efd9ccc17a%7C0%7C0%7C637384647418293191%7CUnknown%7CTWFpbGZsb
> 3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%
> 3D%7C1000&amp;sdata=winuZo94BCsyX2tKbeQJKZnZgUA75r5%2BptD9Y1b6n
> Yo%3D&amp;reserved=0
> >      - wordy comments rephrased
> >      - typos fixed
> >      - rte_eth_rx_queue_setup configuration check isolated
> >        for two main options
> >      - the rest of comments addressed
> >
> > v7:
> https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpatches
> .dpdk.org%2Fpatch%2F80978%2F&amp;data=04%7C01%7Cviacheslavo%40nvi
> dia.com%7C2acfee0f2bf14ffcf9c008d871f5b3fa%7C43083d15727340c1b7db39
> efd9ccc17a%7C0%7C0%7C637384647418293191%7CUnknown%7CTWFpbGZsb
> 3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%
> 3D%7C1000&amp;sdata=o6vyym10K%2BpEpBtJQYYMnWEEoQDC2pcgXQSku8
> %2FJ2PE%3D&amp;reserved=0
> >      - description as union of feature dedicated segment split description
> >      - the split limitations reported in the rte_eth_dev_info->rx_seg_capa
> >      - comments made less wordy
> >
> > v8: - bitfield instead of uint8;
> >      - named union to avoid warinings;
> >      - typos fixed
> >
> >
> > v9:
> https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpatches
> .dpdk.org%2Fpatch%2F81080%2F&amp;data=04%7C01%7Cviacheslavo%40nvi
> dia.com%7C2acfee0f2bf14ffcf9c008d871f5b3fa%7C43083d15727340c1b7db39
> efd9ccc17a%7C0%7C0%7C637384647418293191%7CUnknown%7CTWFpbGZsb
> 3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%
> 3D%7C1000&amp;sdata=H3cmcZO7fWt2A3quAPFNlJKVSv97c51vZWmzM86hE
> Ms%3D&amp;reserved=0
> >      - simplified rte_eth_rxseg union
> >      - moved out wordy comment from rte_eth_rxconf
> >      - rte_eth_rxseg_capa uint32 bitfield and naming
> >      - added experimental tags
> >
> > v10: - missed in mailing list, see v11 for changes
> >
> > v11:
> https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpatches
> .dpdk.org%2Fpatch%2F81116%2F&amp;data=04%7C01%7Cviacheslavo%40nvi
> dia.com%7C2acfee0f2bf14ffcf9c008d871f5b3fa%7C43083d15727340c1b7db39
> efd9ccc17a%7C0%7C0%7C637384647418293191%7CUnknown%7CTWFpbGZsb
> 3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%
> 3D%7C1000&amp;sdata=CR5SFYRE1odbQAzWAhOxER0KcgIXadTj1UDEq%2BRr
> 4Ek%3D&amp;reserved=0
> >       - documentation cleanup (removed rxq_info_get mention)
> >       - max_nseg moved from the recored and converted to uint16_t;
> >       - reserved uint16_t is added to rte_eth_rxseg_capa;
> >
> > v12: - __rte_experimental at structure declarations
> >         converted to the tag in the comment
> >       - testpmd pool names shortened, prefix provided
> >       - comments reworded
> >
> > ---
> > Viacheslav Ovsiienko (6):
> >    ethdev: introduce Rx buffer split
> >    app/testpmd: add multiple pools per core creation
> >    app/testpmd: add buffer split offload configuration
> >    app/testpmd: add rxpkts commands and parameters
> >    app/testpmd: add rxoffs commands and parameters
> >    app/testpmd: add extended Rx queue setup
> >
> 
> Series applied to dpdk-next-net/main, thanks.

Thanks to all whoever was involved, for discussion and help in series polishing!