[PATCH] dev-manual: correct note about virtual providers

[PATCH] dev-manual: correct note about virtual providers

[Michael Opdenacker]

Fixes [YOCTO #14530]

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Reported-by: Robert P. J. Day <rpjday@crashcourse.ca>
---
 documentation/dev-manual/common-tasks.rst | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/documentation/dev-manual/common-tasks.rst b/documentation/dev-manual/common-tasks.rst
index f3b5084436..dd015a761e 100644
--- a/documentation/dev-manual/common-tasks.rst
+++ b/documentation/dev-manual/common-tasks.rst
@@ -2114,9 +2114,10 @@ The following lists specific examples of virtual providers:
 
 .. note::
 
-   Virtual providers only apply to build time dependencies specified with
-   :term:`PROVIDES` and :term:`DEPENDS`. They do not apply to runtime
-   dependencies specified with :term:`RPROVIDES` and :term:`RDEPENDS`.
+   Virtual providers can be applied not only to build time dependencies
+   specified with :term:`PROVIDES` and :term:`DEPENDS`, as show in the
+   example above, but also to runtime dependencies specified with
+   :term:`RPROVIDES` and :term:`RDEPENDS`.
 
 Properly Versioning Pre-Release Recipes
 ---------------------------------------
-- 
2.25.1

Re: [docs] [PATCH] dev-manual: correct note about virtual providers

[Quentin Schulz]

Hi Michael,

On Mon, Aug 30, 2021 at 05:37:20PM +0200, Michael Opdenacker wrote:
> Fixes [YOCTO #14530]
> 
> Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
> Reported-by: Robert P. J. Day <rpjday@crashcourse.ca>
> ---
>  documentation/dev-manual/common-tasks.rst | 7 ++++---
>  1 file changed, 4 insertions(+), 3 deletions(-)
> 
> diff --git a/documentation/dev-manual/common-tasks.rst b/documentation/dev-manual/common-tasks.rst
> index f3b5084436..dd015a761e 100644
> --- a/documentation/dev-manual/common-tasks.rst
> +++ b/documentation/dev-manual/common-tasks.rst
> @@ -2114,9 +2114,10 @@ The following lists specific examples of virtual providers:
>  
>  .. note::
>  
> -   Virtual providers only apply to build time dependencies specified with
> -   :term:`PROVIDES` and :term:`DEPENDS`. They do not apply to runtime
> -   dependencies specified with :term:`RPROVIDES` and :term:`RDEPENDS`.
> +   Virtual providers can be applied not only to build time dependencies
> +   specified with :term:`PROVIDES` and :term:`DEPENDS`, as show in the
> +   example above, but also to runtime dependencies specified with
> +   :term:`RPROVIDES` and :term:`RDEPENDS`.
>  

I'm not entirely sure what we wanted to tell initially but I feel like the
following could be a bit clearer?

"
Virtual providers can be used not only for build time dependencies
(:term:`DEPENDS`) when specified in :term:`PROVIDES`, as shown in the
example above, but also for runtime dependencies (:term:`RDEPENDS`) when
specified in :term:`RPROVIDES`.
"

Also wondering if we shouldn't use em dashes instead of commas here.

Cheers,
Quentin

>  Properly Versioning Pre-Release Recipes
>  ---------------------------------------
> -- 
> 2.25.1
> 

> 
> 
> 

Re: [docs] [PATCH] dev-manual: correct note about virtual providers

[Richard Purdie]

On Mon, 2021-08-30 at 17:37 +0200, Michael Opdenacker wrote:
> Fixes [YOCTO #14530]
> 
> Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
> Reported-by: Robert P. J. Day <rpjday@crashcourse.ca>
> ---
>  documentation/dev-manual/common-tasks.rst | 7 ++++---
>  1 file changed, 4 insertions(+), 3 deletions(-)
> 
> diff --git a/documentation/dev-manual/common-tasks.rst b/documentation/dev-manual/common-tasks.rst
> index f3b5084436..dd015a761e 100644
> --- a/documentation/dev-manual/common-tasks.rst
> +++ b/documentation/dev-manual/common-tasks.rst
> @@ -2114,9 +2114,10 @@ The following lists specific examples of virtual providers:
>  
>  .. note::
>  
> -   Virtual providers only apply to build time dependencies specified with
> -   :term:`PROVIDES` and :term:`DEPENDS`. They do not apply to runtime
> -   dependencies specified with :term:`RPROVIDES` and :term:`RDEPENDS`.
> +   Virtual providers can be applied not only to build time dependencies
> +   specified with :term:`PROVIDES` and :term:`DEPENDS`, as show in the
> +   example above, but also to runtime dependencies specified with
> +   :term:`RPROVIDES` and :term:`RDEPENDS`.
>  

This isn't correct. virtual providers of the form virtual/ *only* apply to the
build namespace, not the runtime one.

Cheers,

Richard

Re: [docs] [PATCH] dev-manual: correct note about virtual providers

[Quentin Schulz]

On Tue, Aug 31, 2021 at 10:16:49AM +0100, Richard Purdie wrote:
> On Mon, 2021-08-30 at 17:37 +0200, Michael Opdenacker wrote:
> > Fixes [YOCTO #14530]
> > 
> > Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
> > Reported-by: Robert P. J. Day <rpjday@crashcourse.ca>
> > ---
> >  documentation/dev-manual/common-tasks.rst | 7 ++++---
> >  1 file changed, 4 insertions(+), 3 deletions(-)
> > 
> > diff --git a/documentation/dev-manual/common-tasks.rst b/documentation/dev-manual/common-tasks.rst
> > index f3b5084436..dd015a761e 100644
> > --- a/documentation/dev-manual/common-tasks.rst
> > +++ b/documentation/dev-manual/common-tasks.rst
> > @@ -2114,9 +2114,10 @@ The following lists specific examples of virtual providers:
> >  
> >  .. note::
> >  
> > -   Virtual providers only apply to build time dependencies specified with
> > -   :term:`PROVIDES` and :term:`DEPENDS`. They do not apply to runtime
> > -   dependencies specified with :term:`RPROVIDES` and :term:`RDEPENDS`.
> > +   Virtual providers can be applied not only to build time dependencies
> > +   specified with :term:`PROVIDES` and :term:`DEPENDS`, as show in the
> > +   example above, but also to runtime dependencies specified with
> > +   :term:`RPROVIDES` and :term:`RDEPENDS`.
> >  
> 
> This isn't correct. virtual providers of the form virtual/ *only* apply to the
> build namespace, not the runtime one.
> 

What is the mechanism behind RPROVIDES:${PN} then? You could technically
set it to "virtual/something" right? How Bitbake chooses between two
recipes creating a package RPROVIDES'ing the same "virtual/something" is
something I have no knowledge in but I feel like it's something that
should be explained here, or at least explicit why it does not apply to
runtime dependencies or point to where the providers for runtime
dependencies are explained in the docs.

Cheers,
Quentin

Re: [docs] [PATCH] dev-manual: correct note about virtual providers

[Richard Purdie]

On Tue, 2021-08-31 at 11:49 +0200, Quentin Schulz wrote:
> On Tue, Aug 31, 2021 at 10:16:49AM +0100, Richard Purdie wrote:
> > On Mon, 2021-08-30 at 17:37 +0200, Michael Opdenacker wrote:
> > > Fixes [YOCTO #14530]
> > > 
> > > Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
> > > Reported-by: Robert P. J. Day <rpjday@crashcourse.ca>
> > > ---
> > >  documentation/dev-manual/common-tasks.rst | 7 ++++---
> > >  1 file changed, 4 insertions(+), 3 deletions(-)
> > > 
> > > diff --git a/documentation/dev-manual/common-tasks.rst b/documentation/dev-manual/common-tasks.rst
> > > index f3b5084436..dd015a761e 100644
> > > --- a/documentation/dev-manual/common-tasks.rst
> > > +++ b/documentation/dev-manual/common-tasks.rst
> > > @@ -2114,9 +2114,10 @@ The following lists specific examples of virtual providers:
> > >  
> > >  .. note::
> > >  
> > > -   Virtual providers only apply to build time dependencies specified with
> > > -   :term:`PROVIDES` and :term:`DEPENDS`. They do not apply to runtime
> > > -   dependencies specified with :term:`RPROVIDES` and :term:`RDEPENDS`.
> > > +   Virtual providers can be applied not only to build time dependencies
> > > +   specified with :term:`PROVIDES` and :term:`DEPENDS`, as show in the
> > > +   example above, but also to runtime dependencies specified with
> > > +   :term:`RPROVIDES` and :term:`RDEPENDS`.
> > >  
> > 
> > This isn't correct. virtual providers of the form virtual/ *only* apply to the
> > build namespace, not the runtime one.
> > 
> 
> What is the mechanism behind RPROVIDES:${PN} then? You could technically
> set it to "virtual/something" right?

There is no mechanism and whilst you could set it to that, it wouldn't do
anything useful.

> How Bitbake chooses between two recipes creating a package RPROVIDES'ing the 
> same "virtual/something" is something I have no knowledge in 

You don't set that. It doesn't work. It isn't supported. This is what that
manual section is trying to say. Making the manual pretend it does doesn't
change this.

> but I feel like it's something that
> should be explained here, or at least explicit why it does not apply to
> runtime dependencies or point to where the providers for runtime
> dependencies are explained in the docs.

I've explained this many times before but let me try once again.

Bitbake is the interpreter of DEPENDS and PROVIDES and it can see multiple
providers and choose between them. We have full control over that piece of the
process.

Bitbake picks the combination it thinks the user wants and then bakes it into
the task graph. For any given configuration, one task graph should result.

The runtime namespace is quite different. RPROVIDES are written into the
packages and they are interpreted by the package managers. We have no way to
tell the package manager that when it sees "virtual/shell", it should be bash.
It would be something different for each package manager and at best it is badly
supported by the distros and packaging backends (e.g. dpkg-reconfigure). We've
therefore not gone there.

It is understandable the user still needs to make some choices in OE. These are
the VIRTUAL-RUNTIME_xxxx variables. These are the virtual/ equivalent from
DEPENDS/PROVIDES. The distro hardcodes values of things for the VIRTUAL-
RUNTIME_xxx variables. This is to try and make it clear these are not like
virtual/ and do not respect PREFERRED_PROVIDER. The selections are hardcoded
into packages as they are built.

VIRTUAL-RUNTIME is ugly and it is an OE thing, not a bitbake one, there is
nothing related to it in bitbake. This does make it harder to document.

I'd love something nicer but we've never found a way to make it work reliably
with the package managers, deterministic builds and all our other constraints.

Cheers,

Richard

Re: [docs] [PATCH] dev-manual: correct note about virtual providers

[Michael Opdenacker]

Hi Richard,

On 8/31/21 12:01 PM, Richard Purdie wrote:
>
>> How Bitbake chooses between two recipes creating a package RPROVIDES'ing the 
>> same "virtual/something" is something I have no knowledge in 
> You don't set that. It doesn't work. It isn't supported. This is what that
> manual section is trying to say. Making the manual pretend it does doesn't
> change this.
>
>> but I feel like it's something that
>> should be explained here, or at least explicit why it does not apply to
>> runtime dependencies or point to where the providers for runtime
>> dependencies are explained in the docs.
> I've explained this many times before but let me try once again.
>
> Bitbake is the interpreter of DEPENDS and PROVIDES and it can see multiple
> providers and choose between them. We have full control over that piece of the
> process.
>
> Bitbake picks the combination it thinks the user wants and then bakes it into
> the task graph. For any given configuration, one task graph should result.
>
> The runtime namespace is quite different. RPROVIDES are written into the
> packages and they are interpreted by the package managers. We have no way to
> tell the package manager that when it sees "virtual/shell", it should be bash.
> It would be something different for each package manager and at best it is badly
> supported by the distros and packaging backends (e.g. dpkg-reconfigure). We've
> therefore not gone there.

Thank you for the clarification. I got confused by the below definitions
in recipes:

poky$ git grep virtual | grep RDEPENDS
meta/recipes-bsp/grub/grub-efi_2.06.bb:RDEPENDS:${PN} = "grub-common
virtual/grub-bootconf"
meta/recipes-core/systemd/systemd-boot_249.3.bb:RDEPENDS:${PN} +=
"virtual/systemd-bootconf"

poky$ git grep virtual | grep RPROVIDES
meta/recipes-bsp/grub/grub-bootconf_1.00.bb:RPROVIDES:${PN} +=
"virtual/grub-bootconf"
meta/recipes-core/glibc/glibc-package.inc:RPROVIDES:${PN}-dev =
"eglibc-dev libc6-dev virtual-libc-dev"
meta/recipes-core/musl/musl_git.bb:RPROVIDES:${PN}-dev += "libc-dev
virtual-libc-dev"
meta/recipes-core/systemd/systemd-bootconf_1.00.bb:RPROVIDES:${PN} +=
"virtual/systemd-bootconf"

In these cases, if I understand correctly, "virtual/xxx" are just
regular strings with no special meaning that the package managers will
use, right? Should such packages avoid the use of "virtual/" RPROVIDES?

Thanks again,

Michael.

-- 
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [docs] [PATCH] dev-manual: correct note about virtual providers

[Richard Purdie]

On Tue, 2021-08-31 at 18:56 +0200, Michael Opdenacker wrote:
> Hi Richard,
> 
> On 8/31/21 12:01 PM, Richard Purdie wrote:
> > 
> > > How Bitbake chooses between two recipes creating a package RPROVIDES'ing the 
> > > same "virtual/something" is something I have no knowledge in 
> > You don't set that. It doesn't work. It isn't supported. This is what that
> > manual section is trying to say. Making the manual pretend it does doesn't
> > change this.
> > 
> > > but I feel like it's something that
> > > should be explained here, or at least explicit why it does not apply to
> > > runtime dependencies or point to where the providers for runtime
> > > dependencies are explained in the docs.
> > I've explained this many times before but let me try once again.
> > 
> > Bitbake is the interpreter of DEPENDS and PROVIDES and it can see multiple
> > providers and choose between them. We have full control over that piece of the
> > process.
> > 
> > Bitbake picks the combination it thinks the user wants and then bakes it into
> > the task graph. For any given configuration, one task graph should result.
> > 
> > The runtime namespace is quite different. RPROVIDES are written into the
> > packages and they are interpreted by the package managers. We have no way to
> > tell the package manager that when it sees "virtual/shell", it should be bash.
> > It would be something different for each package manager and at best it is badly
> > supported by the distros and packaging backends (e.g. dpkg-reconfigure). We've
> > therefore not gone there.
> 
> Thank you for the clarification. I got confused by the below definitions
> in recipes:
> 
> poky$ git grep virtual | grep RDEPENDS
> meta/recipes-bsp/grub/grub-efi_2.06.bb:RDEPENDS:${PN} = "grub-common
> virtual/grub-bootconf"
> meta/recipes-core/systemd/systemd-boot_249.3.bb:RDEPENDS:${PN} +=
> "virtual/systemd-bootconf"
> 
> poky$ git grep virtual | grep RPROVIDES
> meta/recipes-bsp/grub/grub-bootconf_1.00.bb:RPROVIDES:${PN} +=
> "virtual/grub-bootconf"
> meta/recipes-core/glibc/glibc-package.inc:RPROVIDES:${PN}-dev =
> "eglibc-dev libc6-dev virtual-libc-dev"
> meta/recipes-core/musl/musl_git.bb:RPROVIDES:${PN}-dev += "libc-dev
> virtual-libc-dev"
> meta/recipes-core/systemd/systemd-bootconf_1.00.bb:RPROVIDES:${PN} +=
> "virtual/systemd-bootconf"
> 
> In these cases, if I understand correctly, "virtual/xxx" are just
> regular strings with no special meaning that the package managers will
> use, right? Should such packages avoid the use of "virtual/" RPROVIDES?

The virtual/xxx should not be there and yes, are just strings.

Cheers,

Richard

Re: [docs] [PATCH] dev-manual: correct note about virtual providers

[Michael Opdenacker]

On 8/31/21 6:58 PM, Richard Purdie wrote:
>
>> Thank you for the clarification. I got confused by the below definitions
>> in recipes:
>>
>> poky$ git grep virtual | grep RDEPENDS
>> meta/recipes-bsp/grub/grub-efi_2.06.bb:RDEPENDS:${PN} = "grub-common
>> virtual/grub-bootconf"
>> meta/recipes-core/systemd/systemd-boot_249.3.bb:RDEPENDS:${PN} +=
>> "virtual/systemd-bootconf"
>>
>> poky$ git grep virtual | grep RPROVIDES
>> meta/recipes-bsp/grub/grub-bootconf_1.00.bb:RPROVIDES:${PN} +=
>> "virtual/grub-bootconf"
>> meta/recipes-core/glibc/glibc-package.inc:RPROVIDES:${PN}-dev =
>> "eglibc-dev libc6-dev virtual-libc-dev"
>> meta/recipes-core/musl/musl_git.bb:RPROVIDES:${PN}-dev += "libc-dev
>> virtual-libc-dev"
>> meta/recipes-core/systemd/systemd-bootconf_1.00.bb:RPROVIDES:${PN} +=
>> "virtual/systemd-bootconf"
>>
>> In these cases, if I understand correctly, "virtual/xxx" are just
>> regular strings with no special meaning that the package managers will
>> use, right? Should such packages avoid the use of "virtual/" RPROVIDES?
> The virtual/xxx should not be there and yes, are just strings.


Thanks for confirming this. What about replacing then "virtual/" by
"virtual-" as already done in "virtual-libc-dev" above?

Cheers

Michael.

-- 
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [docs] [PATCH] dev-manual: correct note about virtual providers

[Quentin Schulz]

On Tue, Aug 31, 2021 at 11:01:33AM +0100, Richard Purdie wrote:
> On Tue, 2021-08-31 at 11:49 +0200, Quentin Schulz wrote:
> > On Tue, Aug 31, 2021 at 10:16:49AM +0100, Richard Purdie wrote:
> > > On Mon, 2021-08-30 at 17:37 +0200, Michael Opdenacker wrote:
> > > > Fixes [YOCTO #14530]
> > > > 
> > > > Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
> > > > Reported-by: Robert P. J. Day <rpjday@crashcourse.ca>
> > > > ---
> > > >  documentation/dev-manual/common-tasks.rst | 7 ++++---
> > > >  1 file changed, 4 insertions(+), 3 deletions(-)
> > > > 
> > > > diff --git a/documentation/dev-manual/common-tasks.rst b/documentation/dev-manual/common-tasks.rst
> > > > index f3b5084436..dd015a761e 100644
> > > > --- a/documentation/dev-manual/common-tasks.rst
> > > > +++ b/documentation/dev-manual/common-tasks.rst
> > > > @@ -2114,9 +2114,10 @@ The following lists specific examples of virtual providers:
> > > >  
> > > >  .. note::
> > > >  
> > > > -   Virtual providers only apply to build time dependencies specified with
> > > > -   :term:`PROVIDES` and :term:`DEPENDS`. They do not apply to runtime
> > > > -   dependencies specified with :term:`RPROVIDES` and :term:`RDEPENDS`.
> > > > +   Virtual providers can be applied not only to build time dependencies
> > > > +   specified with :term:`PROVIDES` and :term:`DEPENDS`, as show in the
> > > > +   example above, but also to runtime dependencies specified with
> > > > +   :term:`RPROVIDES` and :term:`RDEPENDS`.
> > > >  
> > > 
> > > This isn't correct. virtual providers of the form virtual/ *only* apply to the
> > > build namespace, not the runtime one.
> > > 
> > 
> > What is the mechanism behind RPROVIDES:${PN} then? You could technically
> > set it to "virtual/something" right?
> 
> There is no mechanism and whilst you could set it to that, it wouldn't do
> anything useful.
> 
> > How Bitbake chooses between two recipes creating a package RPROVIDES'ing the 
> > same "virtual/something" is something I have no knowledge in 
> 
> You don't set that. It doesn't work. It isn't supported. This is what that
> manual section is trying to say. Making the manual pretend it does doesn't
> change this.
> 
> > but I feel like it's something that
> > should be explained here, or at least explicit why it does not apply to
> > runtime dependencies or point to where the providers for runtime
> > dependencies are explained in the docs.
> 
> I've explained this many times before but let me try once again.
> 
> Bitbake is the interpreter of DEPENDS and PROVIDES and it can see multiple
> providers and choose between them. We have full control over that piece of the
> process.
> 
> Bitbake picks the combination it thinks the user wants and then bakes it into
> the task graph. For any given configuration, one task graph should result.
> 
> The runtime namespace is quite different. RPROVIDES are written into the
> packages and they are interpreted by the package managers. We have no way to
> tell the package manager that when it sees "virtual/shell", it should be bash.
> It would be something different for each package manager and at best it is badly
> supported by the distros and packaging backends (e.g. dpkg-reconfigure). We've
> therefore not gone there.
> 
> It is understandable the user still needs to make some choices in OE. These are
> the VIRTUAL-RUNTIME_xxxx variables. These are the virtual/ equivalent from
> DEPENDS/PROVIDES. The distro hardcodes values of things for the VIRTUAL-
> RUNTIME_xxx variables. This is to try and make it clear these are not like
> virtual/ and do not respect PREFERRED_PROVIDER. The selections are hardcoded
> into packages as they are built.
> 
> VIRTUAL-RUNTIME is ugly and it is an OE thing, not a bitbake one, there is
> nothing related to it in bitbake. This does make it harder to document.
> 
> I'd love something nicer but we've never found a way to make it work reliably
> with the package managers, deterministic builds and all our other constraints.
> 

Thanks so much for having taken the time to answer :)

I want to link to the IRC log here because I continued this discussion on IRC
with Richard, please see:
https://www.yoctoproject.org/irc/%23yocto.2021-09-01.log.html#t2021-09-01T11:55:01

Cheers,
Quentin