[meta-virtualization] [PATCH 1/2] xen: run do_deploy after do_compile and do_populate_sysroot

[meta-virtualization] [PATCH 1/2] xen: run do_deploy after do_compile and do_populate_sysroot

[Diego Sueiro]

When having rm_work and bitbake needs to re-run do_deploy, we need to ensure
that the ${B} is re-generated, otherwise the deploy-xen will be empty.

Signed-off-by: Diego Sueiro <diego.sueiro@arm.com>
---
 recipes-extended/xen/xen-hypervisor.inc | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/recipes-extended/xen/xen-hypervisor.inc b/recipes-extended/xen/xen-hypervisor.inc
index 3bfee8e..6a75d16 100644
--- a/recipes-extended/xen/xen-hypervisor.inc
+++ b/recipes-extended/xen/xen-hypervisor.inc
@@ -76,11 +76,13 @@ do_deploy() {
 #   at least run afer that task has completed
 # - the hypervisor binaries may be included in the image filesystem, so we
 #   must ensure that the binaries deployed match what is staged in the sysroot:
-#   so do deploy must run after do_populate_sysroot -- which is always after
-#   do_compile, so that handles 'after do_compile' too
+#   so do_deploy must run after do_populate_sysroot and after do_compile is
+#   also needed for when having rm_work and bitbake needs to re-run do_deploy,
+#   we ensure that the ${B} is re-generated, otherwise the deploy-xen will be
+#   empty
 # - add the task before do_build to ensure that deployment has completed when
 #   the recipe build done stamp is written
-addtask deploy after do_populate_sysroot before do_build
+addtask deploy after do_compile do_populate_sysroot before do_build
 # To ensure that a deployed hypervisor has matching tools, add a dependency to
 # make sure that the tools have built and been staged:
 do_deploy[depends] += "xen-tools:do_populate_sysroot"
-- 
2.17.1

[meta-virtualization] [PATCH 2/2] xen: extend SYSROOT_DIRS

[Diego Sueiro]

The do_install task also ships files to /boot and /usr/lib64. Hence, extend
SYSROOT_DIRS to include these paths.

Signed-off-by: Diego Sueiro <diego.sueiro@arm.com>
---
 recipes-extended/xen/xen-hypervisor.inc | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/recipes-extended/xen/xen-hypervisor.inc b/recipes-extended/xen/xen-hypervisor.inc
index 6a75d16..916bd3c 100644
--- a/recipes-extended/xen/xen-hypervisor.inc
+++ b/recipes-extended/xen/xen-hypervisor.inc
@@ -55,6 +55,8 @@ do_compile() {
 do_install() {
     oe_runmake DESTDIR="${D}" install-xen
 }
+# The do_install also ships files in /boot and /usr/lib64
+SYSROOT_DIRS += "/boot ${exec_prefix}/lib64"
 
 do_deploy() {
     install -d ${DEPLOYDIR}
-- 
2.17.1

Re: [meta-virtualization] [PATCH 1/2] xen: run do_deploy after do_compile and do_populate_sysroot

[Bertrand Marquis]

Hi,

> On 27 Oct 2020, at 15:55, Diego Sueiro via lists.yoctoproject.org <diego.sueiro=arm.com@lists.yoctoproject.org> wrote:
> 
> When having rm_work and bitbake needs to re-run do_deploy, we need to ensure
> that the ${B} is re-generated, otherwise the deploy-xen will be empty.

I can confirm this was encountered by me to and this is solving it.

> 
> Signed-off-by: Diego Sueiro <diego.sueiro@arm.com>
Reviewed-by: Bertrand Marquis <bertrand.marquis@arm.com>

Cheers
Bertrand

> ---
> recipes-extended/xen/xen-hypervisor.inc | 8 +++++---
> 1 file changed, 5 insertions(+), 3 deletions(-)
> 
> diff --git a/recipes-extended/xen/xen-hypervisor.inc b/recipes-extended/xen/xen-hypervisor.inc
> index 3bfee8e..6a75d16 100644
> --- a/recipes-extended/xen/xen-hypervisor.inc
> +++ b/recipes-extended/xen/xen-hypervisor.inc
> @@ -76,11 +76,13 @@ do_deploy() {
> #   at least run afer that task has completed
> # - the hypervisor binaries may be included in the image filesystem, so we
> #   must ensure that the binaries deployed match what is staged in the sysroot:
> -#   so do deploy must run after do_populate_sysroot -- which is always after
> -#   do_compile, so that handles 'after do_compile' too
> +#   so do_deploy must run after do_populate_sysroot and after do_compile is
> +#   also needed for when having rm_work and bitbake needs to re-run do_deploy,
> +#   we ensure that the ${B} is re-generated, otherwise the deploy-xen will be
> +#   empty
> # - add the task before do_build to ensure that deployment has completed when
> #   the recipe build done stamp is written
> -addtask deploy after do_populate_sysroot before do_build
> +addtask deploy after do_compile do_populate_sysroot before do_build
> # To ensure that a deployed hypervisor has matching tools, add a dependency to
> # make sure that the tools have built and been staged:
> do_deploy[depends] += "xen-tools:do_populate_sysroot"
> -- 
> 2.17.1
> 
> 
> 
> 

Re: [meta-virtualization] [PATCH 2/2] xen: extend SYSROOT_DIRS

[Bertrand Marquis]

Hi,

> On 27 Oct 2020, at 15:55, Diego Sueiro via lists.yoctoproject.org <diego.sueiro=arm.com@lists.yoctoproject.org> wrote:
> 
> The do_install task also ships files to /boot and /usr/lib64. Hence, extend
> SYSROOT_DIRS to include these paths.
> 
> Signed-off-by: Diego Sueiro <diego.sueiro@arm.com>
Reviewed-by: Bertrand Marquis <bertrand.marquis@arm.com>

Cheers
Bertrand

> ---
> recipes-extended/xen/xen-hypervisor.inc | 2 ++
> 1 file changed, 2 insertions(+)
> 
> diff --git a/recipes-extended/xen/xen-hypervisor.inc b/recipes-extended/xen/xen-hypervisor.inc
> index 6a75d16..916bd3c 100644
> --- a/recipes-extended/xen/xen-hypervisor.inc
> +++ b/recipes-extended/xen/xen-hypervisor.inc
> @@ -55,6 +55,8 @@ do_compile() {
> do_install() {
>     oe_runmake DESTDIR="${D}" install-xen
> }
> +# The do_install also ships files in /boot and /usr/lib64
> +SYSROOT_DIRS += "/boot ${exec_prefix}/lib64"
> 
> do_deploy() {
>     install -d ${DEPLOYDIR}
> -- 
> 2.17.1
> 
> 
> 
> 

Re: [meta-virtualization] [PATCH 1/2] xen: run do_deploy after do_compile and do_populate_sysroot

[Christopher Clark]

On Wed, Oct 28, 2020 at 1:51 AM Bertrand Marquis
<Bertrand.Marquis@arm.com> wrote:
>
> Hi,
>
> > On 27 Oct 2020, at 15:55, Diego Sueiro via lists.yoctoproject.org <diego.sueiro=arm.com@lists.yoctoproject.org> wrote:
> >
> > When having rm_work and bitbake needs to re-run do_deploy, we need to ensure
> > that the ${B} is re-generated, otherwise the deploy-xen will be empty.
>
> I can confirm this was encountered by me to and this is solving it.

This change makes sense and thanks for including the update to comment
in the commit.

> > Signed-off-by: Diego Sueiro <diego.sueiro@arm.com>
> Reviewed-by: Bertrand Marquis <bertrand.marquis@arm.com>

Reviewed-by: Christopher Clark <christopher.w.clark@gmail.com>

Christopher


>
> Cheers
> Bertrand
>
> > ---
> > recipes-extended/xen/xen-hypervisor.inc | 8 +++++---
> > 1 file changed, 5 insertions(+), 3 deletions(-)
> >
> > diff --git a/recipes-extended/xen/xen-hypervisor.inc b/recipes-extended/xen/xen-hypervisor.inc
> > index 3bfee8e..6a75d16 100644
> > --- a/recipes-extended/xen/xen-hypervisor.inc
> > +++ b/recipes-extended/xen/xen-hypervisor.inc
> > @@ -76,11 +76,13 @@ do_deploy() {
> > #   at least run afer that task has completed
> > # - the hypervisor binaries may be included in the image filesystem, so we
> > #   must ensure that the binaries deployed match what is staged in the sysroot:
> > -#   so do deploy must run after do_populate_sysroot -- which is always after
> > -#   do_compile, so that handles 'after do_compile' too
> > +#   so do_deploy must run after do_populate_sysroot and after do_compile is
> > +#   also needed for when having rm_work and bitbake needs to re-run do_deploy,
> > +#   we ensure that the ${B} is re-generated, otherwise the deploy-xen will be
> > +#   empty
> > # - add the task before do_build to ensure that deployment has completed when
> > #   the recipe build done stamp is written
> > -addtask deploy after do_populate_sysroot before do_build
> > +addtask deploy after do_compile do_populate_sysroot before do_build
> > # To ensure that a deployed hypervisor has matching tools, add a dependency to
> > # make sure that the tools have built and been staged:
> > do_deploy[depends] += "xen-tools:do_populate_sysroot"
> > --
> > 2.17.1
> >
> >
> > 
> >
>

Re: [meta-virtualization] [PATCH 2/2] xen: extend SYSROOT_DIRS

[Diego Sueiro]

>-----Original Message-----
>From: meta-virtualization@lists.yoctoproject.org <meta-
>virtualization@lists.yoctoproject.org> On Behalf Of Diego Sueiro via
>lists.yoctoproject.org
>Sent: 27 October 2020 15:56
>To: meta-virtualization@lists.yoctoproject.org
>Cc: nd <nd@arm.com>; Diego Sueiro <Diego.Sueiro@arm.com>;
>christopher.w.clark@gmail.com; cardoe@gentoo.org
>Subject: [meta-virtualization] [PATCH 2/2] xen: extend SYSROOT_DIRS
>
>The do_install task also ships files to /boot and /usr/lib64. Hence, extend
>SYSROOT_DIRS to include these paths.
>
>Signed-off-by: Diego Sueiro <diego.sueiro@arm.com>

@Christopher,
Can you please also review this patch?

Thanks,

--
Diego Sueiro

>---
> recipes-extended/xen/xen-hypervisor.inc | 2 ++
> 1 file changed, 2 insertions(+)
>
>diff --git a/recipes-extended/xen/xen-hypervisor.inc b/recipes-
>extended/xen/xen-hypervisor.inc
>index 6a75d16..916bd3c 100644
>--- a/recipes-extended/xen/xen-hypervisor.inc
>+++ b/recipes-extended/xen/xen-hypervisor.inc
>@@ -55,6 +55,8 @@ do_compile() {
> do_install() {
>     oe_runmake DESTDIR="${D}" install-xen  }
>+# The do_install also ships files in /boot and /usr/lib64 SYSROOT_DIRS
>++= "/boot ${exec_prefix}/lib64"
>
> do_deploy() {
>     install -d ${DEPLOYDIR}
>--
>2.17.1

Re: [meta-virtualization] [PATCH 2/2] xen: extend SYSROOT_DIRS

[Christopher Clark]

> On Oct 30, 2020, at 1:43 AM, Diego Sueiro <diego.sueiro@arm.com> wrote:
> 
> 
>> 
>> -----Original Message-----
>> From: meta-virtualization@lists.yoctoproject.org <meta-
>> virtualization@lists.yoctoproject.org> On Behalf Of Diego Sueiro via
>> lists.yoctoproject.org
>> Sent: 27 October 2020 15:56
>> To: meta-virtualization@lists.yoctoproject.org
>> Cc: nd <nd@arm.com>; Diego Sueiro <Diego.Sueiro@arm.com>;
>> christopher.w.clark@gmail.com; cardoe@gentoo.org
>> Subject: [meta-virtualization] [PATCH 2/2] xen: extend SYSROOT_DIRS
>> 
>> The do_install task also ships files to /boot and /usr/lib64. Hence, extend
>> SYSROOT_DIRS to include these paths.
>> 
>> Signed-off-by: Diego Sueiro <diego.sueiro@arm.com>
> 
> @Christopher,
> Can you please also review this patch?

Hi Diego,

I started on reviewing this but it took a little longer than I'd anticipated as it requires it digging into what the SYSROOT_DIRS variable does.

The variables you use for the directories being added look ok, given that the recipe does install to there, but I don't quite understand why they need to be added to SYSROOT_DIRS. Does every directory that the recipe installs to need to be listed in that variable? I saw that ${bindir} is not listed as included in it by default in the Yocto mega-manual docs.
What behaviour is this change intended to fix? Could it be described in the commit message?

Christopher 

> 
> Thanks,
> 
> --
> Diego Sueiro
> 
>> ---
>> recipes-extended/xen/xen-hypervisor.inc | 2 ++
>> 1 file changed, 2 insertions(+)
>> 
>> diff --git a/recipes-extended/xen/xen-hypervisor.inc b/recipes-
>> extended/xen/xen-hypervisor.inc
>> index 6a75d16..916bd3c 100644
>> --- a/recipes-extended/xen/xen-hypervisor.inc
>> +++ b/recipes-extended/xen/xen-hypervisor.inc
>> @@ -55,6 +55,8 @@ do_compile() {
>> do_install() {
>>    oe_runmake DESTDIR="${D}" install-xen  }
>> +# The do_install also ships files in /boot and /usr/lib64 SYSROOT_DIRS
>> ++= "/boot ${exec_prefix}/lib64"
>> 
>> do_deploy() {
>>    install -d ${DEPLOYDIR}
>> --
>> 2.17.1
> 

Re: [meta-virtualization] [PATCH 2/2] xen: extend SYSROOT_DIRS

[Diego Sueiro]

>-----Original Message-----
>From: meta-virtualization@lists.yoctoproject.org <meta-
>virtualization@lists.yoctoproject.org> On Behalf Of Christopher Clark via
>lists.yoctoproject.org
>Sent: 30 October 2020 18:09
>To: Diego Sueiro <Diego.Sueiro@arm.com>
>Cc: meta-virtualization@lists.yoctoproject.org; nd <nd@arm.com>;
>cardoe@gentoo.org
>Subject: Re: [meta-virtualization] [PATCH 2/2] xen: extend SYSROOT_DIRS
>
>
>
>> On Oct 30, 2020, at 1:43 AM, Diego Sueiro <diego.sueiro@arm.com> wrote:
>>
>> 
>>>
>>> -----Original Message-----
>>> From: meta-virtualization@lists.yoctoproject.org <meta-
>>> virtualization@lists.yoctoproject.org> On Behalf Of Diego Sueiro via
>>> lists.yoctoproject.org
>>> Sent: 27 October 2020 15:56
>>> To: meta-virtualization@lists.yoctoproject.org
>>> Cc: nd <nd@arm.com>; Diego Sueiro <Diego.Sueiro@arm.com>;
>>> christopher.w.clark@gmail.com; cardoe@gentoo.org
>>> Subject: [meta-virtualization] [PATCH 2/2] xen: extend SYSROOT_DIRS
>>>
>>> The do_install task also ships files to /boot and /usr/lib64. Hence,
>>> extend SYSROOT_DIRS to include these paths.
>>>
>>> Signed-off-by: Diego Sueiro <diego.sueiro@arm.com>
>>
>> @Christopher,
>> Can you please also review this patch?
>
>Hi Diego,
>
>I started on reviewing this but it took a little longer than I'd anticipated as it
>requires it digging into what the SYSROOT_DIRS variable does.
>
>The variables you use for the directories being added look ok, given that the
>recipe does install to there, but I don't quite understand why they need to be
>added to SYSROOT_DIRS. Does every directory that the recipe installs to need
>to be listed in that variable? I saw that ${bindir} is not listed as included in it by
>default in the Yocto mega-manual docs.
>What behaviour is this change intended to fix? Could it be described in the
>commit message?

Quoting the manual (https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#new-sharing-files-between-recipes):
" A subset of the files installed by the do_install task are used by the do_populate_sysroot task as defined by the the SYSROOT_DIRS variable to automatically populate the sysroot. It is possible to modify the list of directories that populate the sysroot. The following example shows how you could add the /opt directory to the list of directories within a recipe:

     SYSROOT_DIRS += "/opt"
"

If a recipe DEPENDS_${PN} = "xen", it expects all the installed files to be in the RECIPE_SYSROOT. In the case of xen, specially the shipped binaries, as of today
only the sysroot is only including the  /usr/lib/debug/xen-syms-4.14.0[.map] files.

I'll add more words in the commit message.

--
Diego Sueiro

>
>Christopher
>
>>
>> Thanks,
>>
>> --
>> Diego Sueiro
>>
>>> ---
>>> recipes-extended/xen/xen-hypervisor.inc | 2 ++
>>> 1 file changed, 2 insertions(+)
>>>
>>> diff --git a/recipes-extended/xen/xen-hypervisor.inc b/recipes-
>>> extended/xen/xen-hypervisor.inc index 6a75d16..916bd3c 100644
>>> --- a/recipes-extended/xen/xen-hypervisor.inc
>>> +++ b/recipes-extended/xen/xen-hypervisor.inc
>>> @@ -55,6 +55,8 @@ do_compile() {
>>> do_install() {
>>>    oe_runmake DESTDIR="${D}" install-xen  }
>>> +# The do_install also ships files in /boot and /usr/lib64
>>> +SYSROOT_DIRS
>>> ++= "/boot ${exec_prefix}/lib64"
>>>
>>> do_deploy() {
>>>    install -d ${DEPLOYDIR}
>>> --
>>> 2.17.1
>>

[meta-virtualization] [PATCH v2 2/2] xen: extend SYSROOT_DIRS to include installed binaries

[Diego Sueiro]

If a recipe DEPENDS on xen we should include its binaries in the sysroot.
The do_install task also ships files to /boot and /usr/lib64. Hence, extend
SYSROOT_DIRS to include these paths.

Signed-off-by: Diego Sueiro <diego.sueiro@arm.com>
---
Changes in v2:
   - Updated commit subject and message

 recipes-extended/xen/xen-hypervisor.inc | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/recipes-extended/xen/xen-hypervisor.inc b/recipes-extended/xen/xen-hypervisor.inc
index 6a75d16..916bd3c 100644
--- a/recipes-extended/xen/xen-hypervisor.inc
+++ b/recipes-extended/xen/xen-hypervisor.inc
@@ -55,6 +55,8 @@ do_compile() {
 do_install() {
     oe_runmake DESTDIR="${D}" install-xen
 }
+# The do_install also ships files in /boot and /usr/lib64
+SYSROOT_DIRS += "/boot ${exec_prefix}/lib64"
 
 do_deploy() {
     install -d ${DEPLOYDIR}
-- 
2.17.1

Re: [meta-virtualization] [PATCH v2 2/2] xen: extend SYSROOT_DIRS to include installed binaries

[Christopher Clark]

On Mon, Nov 2, 2020 at 12:14 AM Diego Sueiro <diego.sueiro@arm.com> wrote:
>
> If a recipe DEPENDS on xen we should include its binaries in the sysroot.
> The do_install task also ships files to /boot and /usr/lib64. Hence, extend
> SYSROOT_DIRS to include these paths.
>
> Signed-off-by: Diego Sueiro <diego.sueiro@arm.com>

Reviewed-by: Christopher Clark <christopher.w.clark@gmail.com>

> ---
> Changes in v2:
>    - Updated commit subject and message
>
>  recipes-extended/xen/xen-hypervisor.inc | 2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/recipes-extended/xen/xen-hypervisor.inc b/recipes-extended/xen/xen-hypervisor.inc
> index 6a75d16..916bd3c 100644
> --- a/recipes-extended/xen/xen-hypervisor.inc
> +++ b/recipes-extended/xen/xen-hypervisor.inc
> @@ -55,6 +55,8 @@ do_compile() {
>  do_install() {
>      oe_runmake DESTDIR="${D}" install-xen
>  }
> +# The do_install also ships files in /boot and /usr/lib64
> +SYSROOT_DIRS += "/boot ${exec_prefix}/lib64"
>
>  do_deploy() {
>      install -d ${DEPLOYDIR}
> --
> 2.17.1
>

Re: [meta-virtualization] [PATCH v2 2/2] xen: extend SYSROOT_DIRS to include installed binaries

[Bruce Ashfield]

v2 looks good to me as well (FWIW: I had to do the same thing
when implementing initramfs and qemu to ensure that extra parts
of the qemu build were availble to recipes).

This is now merged.

Bruce


In message: [meta-virtualization] [PATCH v2 2/2] xen: extend SYSROOT_DIRS to include installed binaries
on 02/11/2020 Diego Sueiro wrote:

> If a recipe DEPENDS on xen we should include its binaries in the sysroot.
> The do_install task also ships files to /boot and /usr/lib64. Hence, extend
> SYSROOT_DIRS to include these paths.
> 
> Signed-off-by: Diego Sueiro <diego.sueiro@arm.com>
> ---
> Changes in v2:
>    - Updated commit subject and message
> 
>  recipes-extended/xen/xen-hypervisor.inc | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/recipes-extended/xen/xen-hypervisor.inc b/recipes-extended/xen/xen-hypervisor.inc
> index 6a75d16..916bd3c 100644
> --- a/recipes-extended/xen/xen-hypervisor.inc
> +++ b/recipes-extended/xen/xen-hypervisor.inc
> @@ -55,6 +55,8 @@ do_compile() {
>  do_install() {
>      oe_runmake DESTDIR="${D}" install-xen
>  }
> +# The do_install also ships files in /boot and /usr/lib64
> +SYSROOT_DIRS += "/boot ${exec_prefix}/lib64"
>  
>  do_deploy() {
>      install -d ${DEPLOYDIR}
> -- 
> 2.17.1
> 

> 
> 
> 

Re: [meta-virtualization] [PATCH v2 2/2] xen: extend SYSROOT_DIRS to include installed binaries

[Bertrand Marquis]

> On 2 Nov 2020, at 08:13, Diego Sueiro via lists.yoctoproject.org <diego.sueiro=arm.com@lists.yoctoproject.org> wrote:
> 
> If a recipe DEPENDS on xen we should include its binaries in the sysroot.
> The do_install task also ships files to /boot and /usr/lib64. Hence, extend
> SYSROOT_DIRS to include these paths.
> 
> Signed-off-by: Diego Sueiro <diego.sueiro@arm.com>
Reviewed-by: Bertrand Marquis <bertrand.marquis@arm.com>

> ---
> Changes in v2:
>   - Updated commit subject and message
> 
> recipes-extended/xen/xen-hypervisor.inc | 2 ++
> 1 file changed, 2 insertions(+)
> 
> diff --git a/recipes-extended/xen/xen-hypervisor.inc b/recipes-extended/xen/xen-hypervisor.inc
> index 6a75d16..916bd3c 100644
> --- a/recipes-extended/xen/xen-hypervisor.inc
> +++ b/recipes-extended/xen/xen-hypervisor.inc
> @@ -55,6 +55,8 @@ do_compile() {
> do_install() {
>     oe_runmake DESTDIR="${D}" install-xen
> }
> +# The do_install also ships files in /boot and /usr/lib64
> +SYSROOT_DIRS += "/boot ${exec_prefix}/lib64"
> 
> do_deploy() {
>     install -d ${DEPLOYDIR}
> -- 
> 2.17.1
> 
> 
> 
>