Significance of "SRCREV_FORMAT" variable for multiple sources fetch

Significance of "SRCREV_FORMAT" variable for multiple sources fetch

[Abhishek Shah]

[-- Attachment #1: Type: text/plain, Size: 1147 bytes --]

Hi,

We have a recipe, which has multiple SRC_URI. For each SRC_URI, we have specified SRCREV using SRCREV_name.

I am facing issue when trying to add one more SRC_URI:

While adding new source, I am updating SRCREV_FORMAT variable.
This causea lengthening of PATH where all the fetched source are downloaded.
For example, if earlier location was
tmp/yocto_build/poky/build/tmp/work/aarch64-poky-linux/xxx/gitAUTOINC+a20a30cd86_dbbf7c9b30_f3635d87f4_90d93a11ba_ae871b61ae_5ba575618e-r0/git/
new location would be elongated something like:
tmp/yocto_build/poky/build/tmp/work/aarch64-poky-linux/xxx/gitAUTOINC+a20a30cd86_ bfa48a337e_ dbbf7c9b30_f3635d87f4_90d93a11ba_ae871b61ae_5ba575618e-r0/git/

One of the source is using a Python script, which is using absolute PATH to create a symlink.
Because of elongated PATH, I get below error:
os.symlink(directory, symlink)
OSError: [Errno 36] File name too long

If I don't update SRCREV_FORMAT variable even after adding new source, I do not see above issue.

So, the question is, is that okay if we do not update SRCREV_FORMAT  when adding new SRC_URI in the bitbake recipe?

[-- Attachment #2: Type: text/html, Size: 2117 bytes --]

Re: [bitbake-devel] Significance of "SRCREV_FORMAT" variable for multiple sources fetch

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 1624 bytes --]

Which Yocto version are you on? Zeus and later is probably ok, earlier
versions is not, as do_fetch task hash in the older versions does not
include all the SRCREV variations, only the combined SRCPV. So if you
change one of the SRCREV, it will erroneously not rebuild.

Alex

On Fri, 17 Apr 2020 at 20:45, Abhishek Shah via lists.openembedded.org
<abhishek.shah=broadcom.com@lists.openembedded.org> wrote:

> Hi,
>
> We have a recipe, which has multiple SRC_URI. For each SRC_URI, we have
> specified SRCREV using SRCREV_name.
>
> I am facing issue when trying to add one more SRC_URI:
>
> While adding new source, I am updating SRCREV_FORMAT variable.
> This causea lengthening of PATH where all the fetched source are
> downloaded.
> For example, if earlier location was
>
> tmp/yocto_build/poky/build/tmp/work/aarch64-poky-linux/xxx/gitAUTOINC+a20a30cd86_dbbf7c9b30_f3635d87f4_90d93a11ba_ae871b61ae_5ba575618e-r0/git/
> new location would be elongated something like:
>
> tmp/yocto_build/poky/build/tmp/work/aarch64-poky-linux/xxx/gitAUTOINC+a20a30cd86_
> bfa48a337e_dbbf7c9b30_f3635d87f4_90d93a11ba_ae871b61ae_5ba575618e-r0/git/
>
> One of the source is using a Python script, which is using absolute PATH
> to create a symlink.
> Because of elongated PATH, I get below error:
> os.symlink(directory, symlink)
> OSError: [Errno 36] File name too long
>
> If I don't update SRCREV_FORMAT variable even after adding new source, I
> do not see above issue.
>
> So, the question is, is that okay if we do not update SRCREV_FORMAT  when
> adding new SRC_URI in the bitbake recipe? 
>

[-- Attachment #2: Type: text/html, Size: 2882 bytes --]

Re: [bitbake-devel] Significance of "SRCREV_FORMAT" variable for multiple sources fetch

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 1866 bytes --]

I'd also suggest that you fix the source to create a relative symlink
rather than absolute one.

Alex

On Fri, 17 Apr 2020 at 20:53, Alexander Kanavin <alex.kanavin@gmail.com>
wrote:

> Which Yocto version are you on? Zeus and later is probably ok, earlier
> versions is not, as do_fetch task hash in the older versions does not
> include all the SRCREV variations, only the combined SRCPV. So if you
> change one of the SRCREV, it will erroneously not rebuild.
>
> Alex
>
> On Fri, 17 Apr 2020 at 20:45, Abhishek Shah via lists.openembedded.org
> <abhishek.shah=broadcom.com@lists.openembedded.org> wrote:
>
>> Hi,
>>
>> We have a recipe, which has multiple SRC_URI. For each SRC_URI, we have
>> specified SRCREV using SRCREV_name.
>>
>> I am facing issue when trying to add one more SRC_URI:
>>
>> While adding new source, I am updating SRCREV_FORMAT variable.
>> This causea lengthening of PATH where all the fetched source are
>> downloaded.
>> For example, if earlier location was
>>
>> tmp/yocto_build/poky/build/tmp/work/aarch64-poky-linux/xxx/gitAUTOINC+a20a30cd86_dbbf7c9b30_f3635d87f4_90d93a11ba_ae871b61ae_5ba575618e-r0/git/
>> new location would be elongated something like:
>>
>> tmp/yocto_build/poky/build/tmp/work/aarch64-poky-linux/xxx/gitAUTOINC+a20a30cd86_
>> bfa48a337e_dbbf7c9b30_f3635d87f4_90d93a11ba_ae871b61ae_5ba575618e-r0/git/
>>
>> One of the source is using a Python script, which is using absolute PATH
>> to create a symlink.
>> Because of elongated PATH, I get below error:
>> os.symlink(directory, symlink)
>> OSError: [Errno 36] File name too long
>>
>> If I don't update SRCREV_FORMAT variable even after adding new source, I
>> do not see above issue.
>>
>> So, the question is, is that okay if we do not update SRCREV_FORMAT  when
>> adding new SRC_URI in the bitbake recipe? 
>>
>

[-- Attachment #2: Type: text/html, Size: 3423 bytes --]

Re: Significance of "SRCREV_FORMAT" variable for multiple sources fetch

[Abhishek Shah]

[-- Attachment #1: Type: text/plain, Size: 186 bytes --]

We are using dunfell (3.1) yocto.

How does SRCREV_FORMAT get used to check rebuild is needed or not?
Shouldn't SRCREV_name be enough to check if "name" needs to be fectched again?

[-- Attachment #2: Type: text/html, Size: 202 bytes --]

Re: [bitbake-devel] Significance of "SRCREV_FORMAT" variable for multiple sources fetch

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 583 bytes --]

Nowadays it's enough, but there used to be a bug where none of the SRCREV
named overrides were factored into the task hash:
http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?h=master-next&id=74331a426ff263cd7b74a7578d6f49eb72d7f3c2

Alex

On Fri, 17 Apr 2020 at 21:51, Abhishek Shah via lists.openembedded.org
<abhishek.shah=broadcom.com@lists.openembedded.org> wrote:

> We are using dunfell (3.1) yocto.
>
> How does SRCREV_FORMAT get used to check rebuild is needed or not?
> Shouldn't SRCREV_name be enough to check if "name" needs to be fectched
> again? 
>

[-- Attachment #2: Type: text/html, Size: 1114 bytes --]

Re: Significance of "SRCREV_FORMAT" variable for multiple sources fetch

[Abhishek Shah]

[-- Attachment #1: Type: text/plain, Size: 295 bytes --]

I just checked that we have mentioned commit in yocto we are using.

Could you please confirm, with this in place, is there no need to keep on updating SRCREV_FORMAT
when new SRC_URI is added in the recipe?
Rather, can we just skip setting this variable for multiple source bitbake recipe?

[-- Attachment #2: Type: text/html, Size: 311 bytes --]

Re: [bitbake-devel] Significance of "SRCREV_FORMAT" variable for multiple sources fetch

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 655 bytes --]

I think you should fix the code you're building so that it creates a
relative symlink.

Dropping SRCREV_FORMAT is a workaround to make the path shorter, not a real
fix for the actual problem.

Alex

On Fri, 17 Apr 2020 at 22:11, Abhishek Shah via lists.openembedded.org
<abhishek.shah=broadcom.com@lists.openembedded.org> wrote:

> I just checked that we have mentioned commit in yocto we are using.
>
> Could you please confirm, with this in place, is there no need to keep on
> updating SRCREV_FORMAT
> when new SRC_URI is added in the recipe?
> Rather, can we just skip setting this variable for multiple source bitbake
> recipe? 
>

[-- Attachment #2: Type: text/html, Size: 1054 bytes --]

Re: Significance of "SRCREV_FORMAT" variable for multiple sources fetch

[Abhishek Shah]

[-- Attachment #1: Type: text/plain, Size: 1047 bytes --]

Thanks.

FYI, here is the example directory and its symlink:
directory path:
'/tmp/yocto_builds/ 20133855_ genericx86-64/poky/build/tmp/work/
aarch64-poky-linux/brcm-zephyr/gitAUTOINC+a20a30cd86_bfa48a337e_dbbf7c9b30_f3635d87f4_90d93a11ba_ae871b61ae_5ba575618e-r0/git/
zephyr/include/arch/arm/aarch32/cortex_m/mpu'

It's symlink created by subfolder_list.py:
'/tmp/yocto_builds/ 20133855_ genericx86-64/poky/build/tmp/work/
aarch64-poky-linux/brcm-zephyr/gitAUTOINC+a20a30cd86_bfa48a337e_dbbf7c9b30_f3635d87f4_90d93a11ba_ae871b61ae_5ba575618e-r0/git/
zephyr/samples/hello_world/build_soc/zephyr/misc/generated/syscalls_links/
_tmp_yocto_builds_20133855_genericx86-64_poky_build_tmp_work_aarch64-poky-linux_brcm-zephyr_gitAUTOINC+a20a30cd86_bfa48a337e_dbbf7c9b30_f3635d87f4_90d93a11ba_ae871b61ae_5ba575618e-r0_git_zephyr_include_arch_arm_aarch32_cortex_m_mpu'

I have dropped an mail to community to take it further in direction of reducing symlink length:
https://lists.zephyrproject.org/g/devel/message/6944

Regards,
Abhishek

[-- Attachment #2: Type: text/html, Size: 1248 bytes --]