[RFC] kernel paramters like DPDK CLI options

[RFC] kernel paramters like DPDK CLI options

[Yuanhan Liu]

Hi all,

I guess we (maybe just me :) have stated few times something like
"hey, this kind of stuff is good to have, but you are trying to
add an EAL CLI option for a specific subsystem/driver, which is
wrong".

One recent example that is still fresh in my mind is the one from
Christian [0], that he made a proposal to introduce two new EAL
options, --vhost-owner and --vhost-perm, to configure the vhost
user socket file permission.

    [0]: http://dpdk.org/ml/archives/dev/2016-April/037948.html

Another example is the one I met while enabling virtio 1.0 support.
QEMU has the ability to support both virtio 0.95 (legacy) and 1.0
(modern) at the same time for one virtio device, therefore, we
could either use legacy driver or modern driver to operate the
device. However, the current logic is we try with modern driver
first, and then legacy driver if it failed. In above case, we will
never hit the legacy driver. But sometimes, it's nice to let it
force back to the legacy driver, say, for debug or compare purpose.

Apparently, adding a new EAL option like "--force-legacy" looks
wrong.

The generic yet elegant solution I just thought of while having
lunch is to add a new EAL option, say, --extra-options, where we
could specify driver/subsystem specific options. As you see, it's
nothing big deal, it just looks like Linux kernel parameters.

Take above two cases as example, it could be:

    --extra-options "vhost-owner=kvm:kvm force-legacy"

Note that those options could also be delimited by comma.

DPDK EAL then will provide some generic helper functions to get
and parse those options, and let the specific driver/subsystem
to invoke them to do the actual parse and do the proper action
when some option is specified, say, virtio PMD driver will force
back to legacy driver when "force-legacy" is given.

Comments? Makes sense to you guys, or something nice to have?

	--yliu

Re: [RFC] kernel paramters like DPDK CLI options

[Thomas Monjalon]

Hi,

2016-06-01 14:04, Yuanhan Liu:
> Hi all,
> 
> I guess we (maybe just me :) have stated few times something like
> "hey, this kind of stuff is good to have, but you are trying to
> add an EAL CLI option for a specific subsystem/driver, which is
> wrong".

Yes

> One recent example that is still fresh in my mind is the one from
> Christian [0], that he made a proposal to introduce two new EAL
> options, --vhost-owner and --vhost-perm, to configure the vhost
> user socket file permission.
> 
>     [0]: http://dpdk.org/ml/archives/dev/2016-April/037948.html
> 
> Another example is the one I met while enabling virtio 1.0 support.
> QEMU has the ability to support both virtio 0.95 (legacy) and 1.0
> (modern) at the same time for one virtio device, therefore, we
> could either use legacy driver or modern driver to operate the
> device. However, the current logic is we try with modern driver
> first, and then legacy driver if it failed. In above case, we will
> never hit the legacy driver. But sometimes, it's nice to let it
> force back to the legacy driver, say, for debug or compare purpose.
> 
> Apparently, adding a new EAL option like "--force-legacy" looks
> wrong.
> 
> The generic yet elegant solution I just thought of while having
> lunch is to add a new EAL option, say, --extra-options, where we
> could specify driver/subsystem specific options. As you see, it's
> nothing big deal, it just looks like Linux kernel parameters.
> 
> Take above two cases as example, it could be:
> 
>     --extra-options "vhost-owner=kvm:kvm force-legacy"

I think it's better to have CLI options per device.
Currently we can pass devargs
	- to PCI device via --pci-whitelist
	- to virtual device via --vdev
I think we just need to refactor these options to have a generic
--device or keep the options in --vdev and add a new --pciopt
or something like that.

And more importantly, these devargs must be set via a new EAL API
to allow applications do these configurations without building/faking
some command line arguments.

To make it clear, applications use API and users use CLI (which call API).

> Note that those options could also be delimited by comma.
> 
> DPDK EAL then will provide some generic helper functions to get
> and parse those options, and let the specific driver/subsystem
> to invoke them to do the actual parse and do the proper action
> when some option is specified, say, virtio PMD driver will force
> back to legacy driver when "force-legacy" is given.
> 
> Comments? Makes sense to you guys, or something nice to have?

Thanks for starting the discussion.

Re: [RFC] kernel paramters like DPDK CLI options

[Yerden Zhumabekov]

I recently felt tired enough of specifying various options for EAL, so I 
came up to use ini-based configuration. EAL parameters from dedicated 
section of ini file are parsed to argv array which is subsequently fed 
to rte_eal_init(). Quite handy, but maybe a little overkill.


On 01.06.2016 12:04, Yuanhan Liu wrote:
> Hi all,
>
> I guess we (maybe just me :) have stated few times something like
> "hey, this kind of stuff is good to have, but you are trying to
> add an EAL CLI option for a specific subsystem/driver, which is
> wrong".
>
> One recent example that is still fresh in my mind is the one from
> Christian [0], that he made a proposal to introduce two new EAL
> options, --vhost-owner and --vhost-perm, to configure the vhost
> user socket file permission.
>
>      [0]: http://dpdk.org/ml/archives/dev/2016-April/037948.html
>
> Another example is the one I met while enabling virtio 1.0 support.
> QEMU has the ability to support both virtio 0.95 (legacy) and 1.0
> (modern) at the same time for one virtio device, therefore, we
> could either use legacy driver or modern driver to operate the
> device. However, the current logic is we try with modern driver
> first, and then legacy driver if it failed. In above case, we will
> never hit the legacy driver. But sometimes, it's nice to let it
> force back to the legacy driver, say, for debug or compare purpose.
>
> Apparently, adding a new EAL option like "--force-legacy" looks
> wrong.
>
> The generic yet elegant solution I just thought of while having
> lunch is to add a new EAL option, say, --extra-options, where we
> could specify driver/subsystem specific options. As you see, it's
> nothing big deal, it just looks like Linux kernel parameters.
>
> Take above two cases as example, it could be:
>
>      --extra-options "vhost-owner=kvm:kvm force-legacy"
>
> Note that those options could also be delimited by comma.
>
> DPDK EAL then will provide some generic helper functions to get
> and parse those options, and let the specific driver/subsystem
> to invoke them to do the actual parse and do the proper action
> when some option is specified, say, virtio PMD driver will force
> back to legacy driver when "force-legacy" is given.
>
> Comments? Makes sense to you guys, or something nice to have?
>
> 	--yliu

Re: [RFC] kernel paramters like DPDK CLI options

[Yuanhan Liu]

On Wed, Jun 01, 2016 at 12:17:50PM +0200, Thomas Monjalon wrote:
> Hi,
> 
> 2016-06-01 14:04, Yuanhan Liu:
> > Hi all,
> > 
> > I guess we (maybe just me :) have stated few times something like
> > "hey, this kind of stuff is good to have, but you are trying to
> > add an EAL CLI option for a specific subsystem/driver, which is
> > wrong".
> 
> Yes
> 
> > One recent example that is still fresh in my mind is the one from
> > Christian [0], that he made a proposal to introduce two new EAL
> > options, --vhost-owner and --vhost-perm, to configure the vhost
> > user socket file permission.
> > 
> >     [0]: http://dpdk.org/ml/archives/dev/2016-April/037948.html
> > 
> > Another example is the one I met while enabling virtio 1.0 support.
> > QEMU has the ability to support both virtio 0.95 (legacy) and 1.0
> > (modern) at the same time for one virtio device, therefore, we
> > could either use legacy driver or modern driver to operate the
> > device. However, the current logic is we try with modern driver
> > first, and then legacy driver if it failed. In above case, we will
> > never hit the legacy driver. But sometimes, it's nice to let it
> > force back to the legacy driver, say, for debug or compare purpose.
> > 
> > Apparently, adding a new EAL option like "--force-legacy" looks
> > wrong.
> > 
> > The generic yet elegant solution I just thought of while having
> > lunch is to add a new EAL option, say, --extra-options, where we
> > could specify driver/subsystem specific options. As you see, it's
> > nothing big deal, it just looks like Linux kernel parameters.
> > 
> > Take above two cases as example, it could be:
> > 
> >     --extra-options "vhost-owner=kvm:kvm force-legacy"
> 
> I think it's better to have CLI options per device.
> Currently we can pass devargs
> 	- to PCI device via --pci-whitelist

Isn't it just for whitelisting a specific PCI device?

> 	- to virtual device via --vdev

Yes, --vdev works great here. However, as its name states, it's
just for virtual devices. Say, it will not work for virtio PMD,
the force-legacy option mentioned above.

> I think we just need to refactor these options to have a generic
> --device or keep the options in --vdev and add a new --pciopt
> or something like that.

--pciopt should be able to allow us pass more options to a specific
driver. But what about a library, say vhost?

> And more importantly, these devargs must be set via a new EAL API
> to allow applications do these configurations without building/faking
> some command line arguments.
> 
> To make it clear, applications use API and users use CLI (which call API).

I would agree with that. But that basically works for library; it does
not quite make sense to me to introduce a new API for some a driver
option, such as the force-legacy option for virtio PMD.

So, let me make a summary from reading your email, to make sure I get
you right: for drivers (virtual or physical), we could use --vdev or
--pciopt for passing args, respectively. For the libraries, we should
add a new API, and let the application to introduce some options to
invoke it, to pass the options.

I'd say, that would work, but I see inflexibility and some drawbacks:

- I would assume "--pciopt" has the input style of

      "domain:bus:devid:func,option1,option2,..."

  It then looks hard to me to use it: I need figure out the
  pci id first.

- For the libraries, we have to write code to add new options for
  each applictions. With the generic option, user just need use it;
  don't need write a single line of code, which could save user's
  effort. It also gives user an united interface.

  And to make it clear, as stated, I don't object to having an API.
  I mean, the generic option gives us a chance to do the right
  configuration at startup time: it would still invoke the right
  API to do the right thing in the end.

> > Note that those options could also be delimited by comma.
> > 
> > DPDK EAL then will provide some generic helper functions to get
> > and parse those options, and let the specific driver/subsystem
> > to invoke them to do the actual parse and do the proper action
> > when some option is specified, say, virtio PMD driver will force
> > back to legacy driver when "force-legacy" is given.
> > 
> > Comments? Makes sense to you guys, or something nice to have?
> 
> Thanks for starting the discussion.

Thanks for making comments :)

	--yliu

Re: [RFC] kernel paramters like DPDK CLI options

[Thomas Monjalon]

2016-06-01 19:40, Yuanhan Liu:
> On Wed, Jun 01, 2016 at 12:17:50PM +0200, Thomas Monjalon wrote:
> > 2016-06-01 14:04, Yuanhan Liu:
> > > Apparently, adding a new EAL option like "--force-legacy" looks
> > > wrong.
> > > 
> > > The generic yet elegant solution I just thought of while having
> > > lunch is to add a new EAL option, say, --extra-options, where we
> > > could specify driver/subsystem specific options. As you see, it's
> > > nothing big deal, it just looks like Linux kernel parameters.
> > > 
> > > Take above two cases as example, it could be:
> > > 
> > >     --extra-options "vhost-owner=kvm:kvm force-legacy"
> > 
> > I think it's better to have CLI options per device.

Sorry I was just talking about per-device options, not libraries.
We may also talk about driver options. So we end up with 4 kind of options:
	- device options
	- driver options
	- other EAL options
	- library options
It is not clear what the scope of the --extra-options would be.
We already have EAL options (without specific name or prefix)
and vdev options.
I suggest to add
	- PCI options
	- driver options (if it makes sense)
	- library options

> > Currently we can pass devargs
> > 	- to PCI device via --pci-whitelist
> 
> Isn't it just for whitelisting a specific PCI device?

Yes it is. As a side effect, it allows to pass some devargs.

> > 	- to virtual device via --vdev
> 
> Yes, --vdev works great here. However, as its name states, it's
> just for virtual devices. Say, it will not work for virtio PMD,
> the force-legacy option mentioned above.
> 
> > I think we just need to refactor these options to have a generic
> > --device or keep the options in --vdev and add a new --pciopt
> > or something like that.
> 
> --pciopt should be able to allow us pass more options to a specific
> driver. But what about a library, say vhost?

For libraries we can have some specific options.
Should it be prefixed like --mempool:handler=fifo or without prefix
(like EAL options): --mempool-handler=fifo.
This first choice makes easier to parse every mempool options inside
the library.

> > And more importantly, these devargs must be set via a new EAL API
> > to allow applications do these configurations without building/faking
> > some command line arguments.
> > 
> > To make it clear, applications use API and users use CLI (which call API).
> 
> I would agree with that. But that basically works for library; it does
> not quite make sense to me to introduce a new API for some a driver
> option, such as the force-legacy option for virtio PMD.

For drivers, the API must be something like:
	devargs_set(port_id, char*)
or
	devargs_add(port_id, char*)
So it is generic for every driver-specific options.
There is already rte_eal_devargs_add() but it probably needs to be reworked.

> So, let me make a summary from reading your email, to make sure I get
> you right: for drivers (virtual or physical), we could use --vdev or
> --pciopt for passing args, respectively. For the libraries, we should
> add a new API, and let the application to introduce some options to
> invoke it, to pass the options.

I was thinking to implement the library options parsing in DPDK.
But if the application implements its own options parsing without using
the DPDK one, yes the option parsing is obviously done in the application.

> I'd say, that would work, but I see inflexibility and some drawbacks:
> 
> - I would assume "--pciopt" has the input style of
> 
>       "domain:bus:devid:func,option1,option2,..."
> 
>   It then looks hard to me to use it: I need figure out the
>   pci id first.

What do you suggest instead of PCI id?

> - For the libraries, we have to write code to add new options for
>   each applictions. With the generic option, user just need use it;
>   don't need write a single line of code, which could save user's
>   effort. It also gives user an united interface.

Applications can leverage the DPDK options parsing (without writing
any new line of code).

>   And to make it clear, as stated, I don't object to having an API.
>   I mean, the generic option gives us a chance to do the right
>   configuration at startup time: it would still invoke the right
>   API to do the right thing in the end.

I agree. I just want to split your --extra-option proposal in few options
with a bit more context.

Re: [RFC] kernel paramters like DPDK CLI options

[Yuanhan Liu]

On Wed, Jun 01, 2016 at 02:39:28PM +0200, Thomas Monjalon wrote:
> I was thinking to implement the library options parsing in DPDK.
> But if the application implements its own options parsing without using
> the DPDK one, yes the option parsing is obviously done in the application.
> 
> > I'd say, that would work, but I see inflexibility and some drawbacks:
> > 
> > - I would assume "--pciopt" has the input style of
> > 
> >       "domain:bus:devid:func,option1,option2,..."
> > 
> >   It then looks hard to me to use it: I need figure out the
> >   pci id first.
> 
> What do you suggest instead of PCI id?

That might depend on what you need: if you want to configure a specific
device, yes, PCI is needed (or even, a must). In such case, the --pciopt
or the side effect of --pci-whitelist could help. I confess this would
be helpful in some cases.

I guess there is another need is that we might want to pass an option
to a driver, say ixgbe, that would work for all devices operated by it.
In such case, we could use driver name (see the example below).

> > - For the libraries, we have to write code to add new options for
> >   each applictions. With the generic option, user just need use it;
> >   don't need write a single line of code, which could save user's
> >   effort. It also gives user an united interface.
> 
> Applications can leverage the DPDK options parsing (without writing
> any new line of code).
> 
> >   And to make it clear, as stated, I don't object to having an API.
> >   I mean, the generic option gives us a chance to do the right
> >   configuration at startup time: it would still invoke the right
> >   API to do the right thing in the end.
> 
> I agree. I just want to split your --extra-option proposal in few options
> with a bit more context.

Firstly, the name "--extra-option" might not be well taken :(
I just want to show the idea first.

Secondly, splitting it to more options would result to quite many
options introduced; it's also hard to list all of them. User intend
to get lost if so many options provided. It also introduces more
chaos, especially when we are going to add one option for each
library.

For the context issue, I guess we could address it by adding a prefix.
Such as,

    --extra-options (or --whatever) "vhost.owner=kvm:kvm virtio.force_legacy
                                     mempool.handler=xxx"

Based on that, we could introduce other sematics, to let it be:

    driver.opt | driver.pci_id.opt

Where,

- driver.opt
  applies to all devices operated by this driver

- driver.pci_id.opt
  applies only to a specific device with the same pci ID.

This could save us changing the --pci-whitelist sematic, yet it saves
us introducing a new option (--pciopt).

What do you think of it?

	--yliu

Re: [RFC] kernel paramters like DPDK CLI options

[Thomas Monjalon]

2016-06-01 21:19, Yuanhan Liu:
> On Wed, Jun 01, 2016 at 02:39:28PM +0200, Thomas Monjalon wrote:
> > I was thinking to implement the library options parsing in DPDK.
> > But if the application implements its own options parsing without using
> > the DPDK one, yes the option parsing is obviously done in the application.
> > 
> > > I'd say, that would work, but I see inflexibility and some drawbacks:
> > > 
> > > - I would assume "--pciopt" has the input style of
> > > 
> > >       "domain:bus:devid:func,option1,option2,..."
> > > 
> > >   It then looks hard to me to use it: I need figure out the
> > >   pci id first.
> > 
> > What do you suggest instead of PCI id?
> 
> That might depend on what you need: if you want to configure a specific
> device, yes, PCI is needed (or even, a must). In such case, the --pciopt
> or the side effect of --pci-whitelist could help. I confess this would
> be helpful in some cases.
> 
> I guess there is another need is that we might want to pass an option
> to a driver, say ixgbe, that would work for all devices operated by it.
> In such case, we could use driver name (see the example below).
> 
> > > - For the libraries, we have to write code to add new options for
> > >   each applictions. With the generic option, user just need use it;
> > >   don't need write a single line of code, which could save user's
> > >   effort. It also gives user an united interface.
> > 
> > Applications can leverage the DPDK options parsing (without writing
> > any new line of code).
> > 
> > >   And to make it clear, as stated, I don't object to having an API.
> > >   I mean, the generic option gives us a chance to do the right
> > >   configuration at startup time: it would still invoke the right
> > >   API to do the right thing in the end.
> > 
> > I agree. I just want to split your --extra-option proposal in few options
> > with a bit more context.
> 
> Firstly, the name "--extra-option" might not be well taken :(
> I just want to show the idea first.
> 
> Secondly, splitting it to more options would result to quite many
> options introduced; it's also hard to list all of them. User intend
> to get lost if so many options provided. It also introduces more
> chaos, especially when we are going to add one option for each
> library.
> 
> For the context issue, I guess we could address it by adding a prefix.
> Such as,
> 
>     --extra-options (or --whatever) "vhost.owner=kvm:kvm virtio.force_legacy
>                                      mempool.handler=xxx"
> 
> Based on that, we could introduce other sematics, to let it be:
> 
>     driver.opt | driver.pci_id.opt
> 
> Where,
> 
> - driver.opt
>   applies to all devices operated by this driver
> 
> - driver.pci_id.opt
>   applies only to a specific device with the same pci ID.
> 
> This could save us changing the --pci-whitelist sematic, yet it saves
> us introducing a new option (--pciopt).
> 
> What do you think of it?

I like the idea :)
One important benefit of having only one option is to make easier to rename
in applications to e.g. --dpdk-options and pass the string to the DPDK
parsing function.
I think we must allow several occurences of this new option on the CLI.

At the end, the main issue is to find a shiny name for this option ;)

Re: [RFC] kernel paramters like DPDK CLI options

[Wiles, Keith]

Not to highjack this thread I created another one ☺ please have a look, thanks.
http://dpdk.org/ml/archives/dev/2016-June/040079.html

Regards,
Keith

-----Original Message-----
From: dev <dev-bounces@dpdk.org> on behalf of Thomas Monjalon <thomas.monjalon@6wind.com>
Date: Wednesday, June 1, 2016 at 9:03 AM
To: Yuanhan Liu <yuanhan.liu@linux.intel.com>
Cc: "dev@dpdk.org" <dev@dpdk.org>, Bruce Richardson <bruce.richardson@intel.com>, "Tan, Jianfeng" <jianfeng.tan@intel.com>, Stephen Hemminger <stephen@networkplumber.org>, Christian Ehrhardt <christian.ehrhardt@canonical.com>, Panu Matilainen <pmatilai@redhat.com>, Olivier Matz <olivier.matz@6wind.com>
Subject: Re: [dpdk-dev] [RFC] kernel paramters like DPDK CLI options

>2016-06-01 21:19, Yuanhan Liu:
>> On Wed, Jun 01, 2016 at 02:39:28PM +0200, Thomas Monjalon wrote:
>> > I was thinking to implement the library options parsing in DPDK.
>> > But if the application implements its own options parsing without using
>> > the DPDK one, yes the option parsing is obviously done in the application.
>> > 
>> > > I'd say, that would work, but I see inflexibility and some drawbacks:
>> > > 
>> > > - I would assume "--pciopt" has the input style of
>> > > 
>> > >       "domain:bus:devid:func,option1,option2,..."
>> > > 
>> > >   It then looks hard to me to use it: I need figure out the
>> > >   pci id first.
>> > 
>> > What do you suggest instead of PCI id?
>> 

Re: [RFC] kernel paramters like DPDK CLI options

[Yuanhan Liu]

On Wed, Jun 01, 2016 at 04:03:07PM +0200, Thomas Monjalon wrote:
> 2016-06-01 21:19, Yuanhan Liu:
> > On Wed, Jun 01, 2016 at 02:39:28PM +0200, Thomas Monjalon wrote:
> > > I was thinking to implement the library options parsing in DPDK.
> > > But if the application implements its own options parsing without using
> > > the DPDK one, yes the option parsing is obviously done in the application.
> > > 
> > > > I'd say, that would work, but I see inflexibility and some drawbacks:
> > > > 
> > > > - I would assume "--pciopt" has the input style of
> > > > 
> > > >       "domain:bus:devid:func,option1,option2,..."
> > > > 
> > > >   It then looks hard to me to use it: I need figure out the
> > > >   pci id first.
> > > 
> > > What do you suggest instead of PCI id?
> > 
> > That might depend on what you need: if you want to configure a specific
> > device, yes, PCI is needed (or even, a must). In such case, the --pciopt
> > or the side effect of --pci-whitelist could help. I confess this would
> > be helpful in some cases.
> > 
> > I guess there is another need is that we might want to pass an option
> > to a driver, say ixgbe, that would work for all devices operated by it.
> > In such case, we could use driver name (see the example below).
> > 
> > > > - For the libraries, we have to write code to add new options for
> > > >   each applictions. With the generic option, user just need use it;
> > > >   don't need write a single line of code, which could save user's
> > > >   effort. It also gives user an united interface.
> > > 
> > > Applications can leverage the DPDK options parsing (without writing
> > > any new line of code).
> > > 
> > > >   And to make it clear, as stated, I don't object to having an API.
> > > >   I mean, the generic option gives us a chance to do the right
> > > >   configuration at startup time: it would still invoke the right
> > > >   API to do the right thing in the end.
> > > 
> > > I agree. I just want to split your --extra-option proposal in few options
> > > with a bit more context.
> > 
> > Firstly, the name "--extra-option" might not be well taken :(
> > I just want to show the idea first.
> > 
> > Secondly, splitting it to more options would result to quite many
> > options introduced; it's also hard to list all of them. User intend
> > to get lost if so many options provided. It also introduces more
> > chaos, especially when we are going to add one option for each
> > library.
> > 
> > For the context issue, I guess we could address it by adding a prefix.
> > Such as,
> > 
> >     --extra-options (or --whatever) "vhost.owner=kvm:kvm virtio.force_legacy
> >                                      mempool.handler=xxx"
> > 
> > Based on that, we could introduce other sematics, to let it be:
> > 
> >     driver.opt | driver.pci_id.opt
> > 
> > Where,
> > 
> > - driver.opt
> >   applies to all devices operated by this driver
> > 
> > - driver.pci_id.opt
> >   applies only to a specific device with the same pci ID.
> > 
> > This could save us changing the --pci-whitelist sematic, yet it saves
> > us introducing a new option (--pciopt).
> > 
> > What do you think of it?
> 
> I like the idea :)

Superb!

> One important benefit of having only one option is to make easier to rename
> in applications to e.g. --dpdk-options and pass the string to the DPDK
> parsing function.
> I think we must allow several occurences of this new option on the CLI.

No idea so far; I'm thinking one should be enough. But I also see no
issue when allowing several occurences. Let's recheck it later.

> 
> At the end, the main issue is to find a shiny name for this option ;)

You know what, I'm really not good at naming, so might need your
help :-)

	--yliu