DPDK-dev Archive on lore.kernel.org
 help / color / Atom feed
* [dpdk-dev] Mempool handler ops index allocation issue
@ 2019-05-09 22:19 Eads, Gage
  2019-05-13 12:14 ` Olivier Matz
  0 siblings, 1 reply; 4+ messages in thread
From: Eads, Gage @ 2019-05-09 22:19 UTC (permalink / raw)
  To: dev; +Cc: olivier.matz, arybchenko

Hi all,

I ran into a problem with a multi-process application, in which two processes assigned the same mempool handler ops index to *different* handlers. This happened because the two processes supplied the -d EAL arguments in different order, e.g.:

sudo ./appA -dlibrte_mempool_bucket.so -dlibrte_mempool_ring.so --proc-type primary &
sudo ./appB -dlibrte_mempool_ring.so -dlibrte_mempool_bucket.so --proc-type secondary &

The dynamic load order matters because the ops indexes are assigned in the order the library ctors are run. This can result in the different processes trying to use different handlers for the same mempool.

I'm not aware of any requirement that the EAL argument order should match across processes, so I don't think this is a user error. This could also happen in static libraries if they linked the libraries in a different order - but that shouldn't occur if both applications are following the rules for building an external application (https://doc.dpdk.org/guides/prog_guide/dev_kit_build_system.html#building-external-applications).

If you agree that this is an issue, I see a couple possible resolutions:


1.       Add a note/warning to the mempool handlers section of the user guide (https://doc.dpdk.org/guides/prog_guide/mempool_lib.html#mempool-handlers)

2.       Modify rte_mempool_register_ops() so that built-in handlers (ring, stack, etc.) have fixed IDs. E.g. ring is always 0, stack is always 1, etc. These handlers could be identified by their name string. User-registered mempools would still be susceptible to this problem, though.

Thoughts? Alternatives?

Thanks,
Gage

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [dpdk-dev] Mempool handler ops index allocation issue
  2019-05-09 22:19 [dpdk-dev] Mempool handler ops index allocation issue Eads, Gage
@ 2019-05-13 12:14 ` Olivier Matz
  2019-05-13 12:22   ` [dpdk-dev] ***Spam*** " Andrew Rybchenko
  0 siblings, 1 reply; 4+ messages in thread
From: Olivier Matz @ 2019-05-13 12:14 UTC (permalink / raw)
  To: Eads, Gage; +Cc: dev, arybchenko

Hi Gage,

On Thu, May 09, 2019 at 10:19:55PM +0000, Eads, Gage wrote:
> Hi all,
> 
> I ran into a problem with a multi-process application, in which two processes assigned the same mempool handler ops index to *different* handlers. This happened because the two processes supplied the -d EAL arguments in different order, e.g.:
> 
> sudo ./appA -dlibrte_mempool_bucket.so -dlibrte_mempool_ring.so --proc-type primary &
> sudo ./appB -dlibrte_mempool_ring.so -dlibrte_mempool_bucket.so --proc-type secondary &
> 
> The dynamic load order matters because the ops indexes are assigned in the order the library ctors are run. This can result in the different processes trying to use different handlers for the same mempool.
> 
> I'm not aware of any requirement that the EAL argument order should match across processes, so I don't think this is a user error. This could also happen in static libraries if they linked the libraries in a different order - but that shouldn't occur if both applications are following the rules for building an external application (https://doc.dpdk.org/guides/prog_guide/dev_kit_build_system.html#building-external-applications).
> 
> If you agree that this is an issue, I see a couple possible resolutions:
> 
> 
> 1.       Add a note/warning to the mempool handlers section of the user guide (https://doc.dpdk.org/guides/prog_guide/mempool_lib.html#mempool-handlers)
> 
> 2.       Modify rte_mempool_register_ops() so that built-in handlers (ring, stack, etc.) have fixed IDs. E.g. ring is always 0, stack is always 1, etc. These handlers could be identified by their name string. User-registered mempools would still be susceptible to this problem, though.
> 
> Thoughts? Alternatives?

What about this:

- add a new table in a named memory zone that stores the association
  between mempool_ops id and name (but not the ops pointers) of the
  primary process.

- change rte_mempool_register_ops() to have a specific behavior in case
  of a secondary process: lookup in that table to get the id associated
  to the name (fail if not found).


On the other hand, using secondary processes always looked a bit dangerous
to me (for several reasons), so adding a note in the programmer's guide
(your proposal 1) is also fine to me.

Thanks,
Olivier

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [dpdk-dev] ***Spam*** Re: Mempool handler ops index allocation issue
  2019-05-13 12:14 ` Olivier Matz
@ 2019-05-13 12:22   ` " Andrew Rybchenko
  2019-06-18 18:14     ` Eads, Gage
  0 siblings, 1 reply; 4+ messages in thread
From: Andrew Rybchenko @ 2019-05-13 12:22 UTC (permalink / raw)
  To: Olivier Matz, Eads, Gage; +Cc: dev

Hi Olivier,

On 5/13/19 3:14 PM, Olivier Matz wrote:
> Hi Gage,
>
> On Thu, May 09, 2019 at 10:19:55PM +0000, Eads, Gage wrote:
>> Hi all,
>>
>> I ran into a problem with a multi-process application, in which two processes assigned the same mempool handler ops index to *different* handlers. This happened because the two processes supplied the -d EAL arguments in different order, e.g.:
>>
>> sudo ./appA -dlibrte_mempool_bucket.so -dlibrte_mempool_ring.so --proc-type primary &
>> sudo ./appB -dlibrte_mempool_ring.so -dlibrte_mempool_bucket.so --proc-type secondary &
>>
>> The dynamic load order matters because the ops indexes are assigned in the order the library ctors are run. This can result in the different processes trying to use different handlers for the same mempool.
>>
>> I'm not aware of any requirement that the EAL argument order should match across processes, so I don't think this is a user error. This could also happen in static libraries if they linked the libraries in a different order - but that shouldn't occur if both applications are following the rules for building an external application (https://doc.dpdk.org/guides/prog_guide/dev_kit_build_system.html#building-external-applications).
>>
>> If you agree that this is an issue, I see a couple possible resolutions:
>>
>>
>> 1.       Add a note/warning to the mempool handlers section of the user guide (https://doc.dpdk.org/guides/prog_guide/mempool_lib.html#mempool-handlers)
>>
>> 2.       Modify rte_mempool_register_ops() so that built-in handlers (ring, stack, etc.) have fixed IDs. E.g. ring is always 0, stack is always 1, etc. These handlers could be identified by their name string. User-registered mempools would still be susceptible to this problem, though.
>>
>> Thoughts? Alternatives?
> What about this:
>
> - add a new table in a named memory zone that stores the association
>    between mempool_ops id and name (but not the ops pointers) of the
>    primary process.
>
> - change rte_mempool_register_ops() to have a specific behavior in case
>    of a secondary process: lookup in that table to get the id associated
>    to the name (fail if not found).
>
>
> On the other hand, using secondary processes always looked a bit dangerous
> to me (for several reasons), so adding a note in the programmer's guide
> (your proposal 1) is also fine to me.

Suggested solution looks good to me.

Thanks,
Andrew.

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [dpdk-dev] ***Spam*** Re: Mempool handler ops index allocation issue
  2019-05-13 12:22   ` [dpdk-dev] ***Spam*** " Andrew Rybchenko
@ 2019-06-18 18:14     ` Eads, Gage
  0 siblings, 0 replies; 4+ messages in thread
From: Eads, Gage @ 2019-06-18 18:14 UTC (permalink / raw)
  To: Andrew Rybchenko, Olivier Matz; +Cc: dev



> -----Original Message-----
> From: Andrew Rybchenko [mailto:arybchenko@solarflare.com]
> Sent: Monday, May 13, 2019 7:22 AM
> To: Olivier Matz <olivier.matz@6wind.com>; Eads, Gage
> <gage.eads@intel.com>
> Cc: dev@dpdk.org
> Subject: Re: ***Spam*** Re: Mempool handler ops index allocation issue
> 
> Hi Olivier,
> 
> On 5/13/19 3:14 PM, Olivier Matz wrote:
> > Hi Gage,
> >
> > On Thu, May 09, 2019 at 10:19:55PM +0000, Eads, Gage wrote:
> >> Hi all,
> >>
> >> I ran into a problem with a multi-process application, in which two
> processes assigned the same mempool handler ops index to *different*
> handlers. This happened because the two processes supplied the -d EAL
> arguments in different order, e.g.:
> >>
> >> sudo ./appA -dlibrte_mempool_bucket.so -dlibrte_mempool_ring.so
> >> --proc-type primary & sudo ./appB -dlibrte_mempool_ring.so
> >> -dlibrte_mempool_bucket.so --proc-type secondary &
> >>
> >> The dynamic load order matters because the ops indexes are assigned in
> the order the library ctors are run. This can result in the different processes
> trying to use different handlers for the same mempool.
> >>
> >> I'm not aware of any requirement that the EAL argument order should
> match across processes, so I don't think this is a user error. This could also
> happen in static libraries if they linked the libraries in a different order - but
> that shouldn't occur if both applications are following the rules for building an
> external application
> (https://doc.dpdk.org/guides/prog_guide/dev_kit_build_system.html#build
> ing-external-applications).
> >>
> >> If you agree that this is an issue, I see a couple possible resolutions:
> >>
> >>
> >> 1.       Add a note/warning to the mempool handlers section of the user
> guide
> (https://doc.dpdk.org/guides/prog_guide/mempool_lib.html#mempool-
> handlers)
> >>
> >> 2.       Modify rte_mempool_register_ops() so that built-in handlers (ring,
> stack, etc.) have fixed IDs. E.g. ring is always 0, stack is always 1, etc. These
> handlers could be identified by their name string. User-registered mempools
> would still be susceptible to this problem, though.
> >>
> >> Thoughts? Alternatives?
> > What about this:
> >
> > - add a new table in a named memory zone that stores the association
> >    between mempool_ops id and name (but not the ops pointers) of the
> >    primary process.
> >
> > - change rte_mempool_register_ops() to have a specific behavior in case
> >    of a secondary process: lookup in that table to get the id associated
> >    to the name (fail if not found).
> >

Sorry for the delay, just revisiting this now.

Memory zones won't be available in rte_mempool_register_ops when using static libraries (or shared libraries loaded at program startup), since the ctors execute before rte_eal_init() is called.

> >
> > On the other hand, using secondary processes always looked a bit
> > dangerous to me (for several reasons), so adding a note in the
> > programmer's guide (your proposal 1) is also fine to me.

Considering the memzone issue, I'll go this route.

Thanks,
Gage

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, back to index

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-05-09 22:19 [dpdk-dev] Mempool handler ops index allocation issue Eads, Gage
2019-05-13 12:14 ` Olivier Matz
2019-05-13 12:22   ` [dpdk-dev] ***Spam*** " Andrew Rybchenko
2019-06-18 18:14     ` Eads, Gage

DPDK-dev Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/dpdk-dev/0 dpdk-dev/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 dpdk-dev dpdk-dev/ https://lore.kernel.org/dpdk-dev \
		dev@dpdk.org dpdk-dev@archiver.kernel.org
	public-inbox-index dpdk-dev


Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.dpdk.dev


AGPL code for this site: git clone https://public-inbox.org/ public-inbox