* Overriding SDE_FILE
@ 2020-02-24 7:25 Jacob Kroon
2020-02-25 15:53 ` Jacob Kroon
0 siblings, 1 reply; 14+ messages in thread
From: Jacob Kroon @ 2020-02-24 7:25 UTC (permalink / raw)
To: douglas.royds; +Cc: openembedded-core
Hi Douglas,
You updated a comment in reproducible_build.bbclass, commit
e7b891b76954c784f5a93bd0a1c91315673ce40d:
> -# Once the value of SOURCE_DATE_EPOCH is determined, it is stored in the recipe's ${SDE_FILE}.
> +# Once the value of SOURCE_DATE_EPOCH is determined, it is stored in the recipe's SDE_FILE.
> +# If none of these mechanisms are suitable, replace the do_deploy_source_date_epoch task
> +# with recipe-specific functionality to write the appropriate SOURCE_DATE_EPOCH into the SDE_FILE.
> +#
But I can't really get this to work. What did work for me was to replace
"do_create_source_date_epoch_stamp()" in my recipe:
do_create_source_date_epoch_stamp() {
mkdir -p ${SDE_DIR}
date -d "1981-03-03" "+%s" > ${SDE_FILE}
}
What is the intended way to achieve the thing I'm trying to do here ?
Cheers,
Jacob
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: Overriding SDE_FILE 2020-02-24 7:25 Overriding SDE_FILE Jacob Kroon @ 2020-02-25 15:53 ` Jacob Kroon 2020-02-27 5:46 ` Douglas Royds 0 siblings, 1 reply; 14+ messages in thread From: Jacob Kroon @ 2020-02-25 15:53 UTC (permalink / raw) To: douglas.royds; +Cc: openembedded-core On 2/24/20 8:25 AM, Jacob Kroon wrote: > Hi Douglas, > > You updated a comment in reproducible_build.bbclass, commit > e7b891b76954c784f5a93bd0a1c91315673ce40d: > >> -# Once the value of SOURCE_DATE_EPOCH is determined, it is stored in >> the recipe's ${SDE_FILE}. >> +# Once the value of SOURCE_DATE_EPOCH is determined, it is stored in >> the recipe's SDE_FILE. >> +# If none of these mechanisms are suitable, replace the >> do_deploy_source_date_epoch task >> +# with recipe-specific functionality to write the appropriate >> SOURCE_DATE_EPOCH into the SDE_FILE. >> +# > > But I can't really get this to work. What did work for me was to replace > "do_create_source_date_epoch_stamp()" in my recipe: > > do_create_source_date_epoch_stamp() { > mkdir -p ${SDE_DIR} > date -d "1981-03-03" "+%s" > ${SDE_FILE} > } > > What is the intended way to achieve the thing I'm trying to do here ? > FYI, JPEW has a proposed patch here http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=d091d2aa53ea417f70c10f5ce89151820c3db9ce for allowing a recipe to just set SOURCE_DATE_EPOCH directly. But maybe that currently is at odds with SOURCE_DATE_EPOCH being in BB_HASHBASE_WHITELIST ? /Jacob ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Overriding SDE_FILE 2020-02-25 15:53 ` Jacob Kroon @ 2020-02-27 5:46 ` Douglas Royds 2020-02-27 15:01 ` Joshua Watt 0 siblings, 1 reply; 14+ messages in thread From: Douglas Royds @ 2020-02-27 5:46 UTC (permalink / raw) To: Jacob Kroon; +Cc: openembedded-core [-- Attachment #1: Type: text/plain, Size: 1958 bytes --] On 26/02/20 4:53 am, Jacob Kroon wrote: > On 2/24/20 8:25 AM, Jacob Kroon wrote: >> Hi Douglas, >> >> You updated a comment in reproducible_build.bbclass, commit >> e7b891b76954c784f5a93bd0a1c91315673ce40d: >> >>> -# Once the value of SOURCE_DATE_EPOCH is determined, it is stored >>> in the recipe's ${SDE_FILE}. >>> +# Once the value of SOURCE_DATE_EPOCH is determined, it is stored >>> in the recipe's SDE_FILE. >>> +# If none of these mechanisms are suitable, replace the >>> do_deploy_source_date_epoch task >>> +# with recipe-specific functionality to write the appropriate >>> SOURCE_DATE_EPOCH into the SDE_FILE. >>> +# >> >> But I can't really get this to work. What did work for me was to >> replace "do_create_source_date_epoch_stamp()" in my recipe: >> >> do_create_source_date_epoch_stamp() { >> mkdir -p ${SDE_DIR} >> date -d "1981-03-03" "+%s" > ${SDE_FILE} >> } >> >> What is the intended way to achieve the thing I'm trying to do here ? >> > > FYI, JPEW has a proposed patch here > > http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=d091d2aa53ea417f70c10f5ce89151820c3db9ce > > > for allowing a recipe to just set SOURCE_DATE_EPOCH directly. > > But maybe that currently is at odds with SOURCE_DATE_EPOCH being in > BB_HASHBASE_WHITELIST ? > > /Jacob On the surface of it, my comment appears to be just wrong: It does make sense to replace do_create_source_date_epoch_stamp() as you suggest. Joshua's proposed patch looks promising: * Should the new function not be called first, so that it takes priority over the git, known files, and youngest file functions? If someone has explicitly set SOURCE_DATE_EPOCH, then they want it to take priority. * As you observe, SOURCE_DATE_EPOCH would need to be removed from BB_HASHBASE_WHITELIST. I'm not sure why it was in the whitelist in the first place. [-- Attachment #2: Type: text/html, Size: 3156 bytes --] ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Overriding SDE_FILE 2020-02-27 5:46 ` Douglas Royds @ 2020-02-27 15:01 ` Joshua Watt 2020-02-27 16:45 ` Joshua Watt 0 siblings, 1 reply; 14+ messages in thread From: Joshua Watt @ 2020-02-27 15:01 UTC (permalink / raw) To: Douglas Royds, Jacob Kroon; +Cc: openembedded-core [-- Attachment #1: Type: text/plain, Size: 2566 bytes --] On 2/26/20 11:46 PM, Douglas Royds wrote: > > On 26/02/20 4:53 am, Jacob Kroon wrote: > >> On 2/24/20 8:25 AM, Jacob Kroon wrote: >>> Hi Douglas, >>> >>> You updated a comment in reproducible_build.bbclass, commit >>> e7b891b76954c784f5a93bd0a1c91315673ce40d: >>> >>>> -# Once the value of SOURCE_DATE_EPOCH is determined, it is stored >>>> in the recipe's ${SDE_FILE}. >>>> +# Once the value of SOURCE_DATE_EPOCH is determined, it is stored >>>> in the recipe's SDE_FILE. >>>> +# If none of these mechanisms are suitable, replace the >>>> do_deploy_source_date_epoch task >>>> +# with recipe-specific functionality to write the appropriate >>>> SOURCE_DATE_EPOCH into the SDE_FILE. >>>> +# >>> >>> But I can't really get this to work. What did work for me was to >>> replace "do_create_source_date_epoch_stamp()" in my recipe: >>> >>> do_create_source_date_epoch_stamp() { >>> mkdir -p ${SDE_DIR} >>> date -d "1981-03-03" "+%s" > ${SDE_FILE} >>> } >>> >>> What is the intended way to achieve the thing I'm trying to do here ? >>> >> >> FYI, JPEW has a proposed patch here >> >> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=d091d2aa53ea417f70c10f5ce89151820c3db9ce >> >> >> for allowing a recipe to just set SOURCE_DATE_EPOCH directly. >> >> But maybe that currently is at odds with SOURCE_DATE_EPOCH being in >> BB_HASHBASE_WHITELIST ? >> >> /Jacob > > > On the surface of it, my comment appears to be just wrong: It does > make sense to replace do_create_source_date_epoch_stamp() as you suggest. > > Joshua's proposed patch looks promising: > > * Should the new function not be called first, so that it takes > priority over the git, known files, and youngest file functions? > If someone has explicitly set SOURCE_DATE_EPOCH, then they want it > to take priority. > Having that be the first option makes sense. The only case in which that might not work, is if a recipe does something like: SOURCE_DATE_EPOCH = "${@my_awesome_sde_calculation(d)}" e.g. uses a function to get the SDE instead of setting to a fixed value, but that's probably going to be extremely rare. > * > > > > * As you observe, SOURCE_DATE_EPOCH would need to be removed from > BB_HASHBASE_WHITELIST. I'm not sure why it was in the whitelist in > the first place. > I'm not sure why exactly it is whitelisted; I didn't write the original code that whitelisted it, but I've CC'd Juro in case he happens to remember. [-- Attachment #2: Type: text/html, Size: 4130 bytes --] ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Overriding SDE_FILE 2020-02-27 15:01 ` Joshua Watt @ 2020-02-27 16:45 ` Joshua Watt 2020-02-27 21:22 ` Douglas Royds 0 siblings, 1 reply; 14+ messages in thread From: Joshua Watt @ 2020-02-27 16:45 UTC (permalink / raw) To: Douglas Royds, Jacob Kroon; +Cc: openembedded-core [-- Attachment #1: Type: text/plain, Size: 3134 bytes --] On 2/27/20 9:01 AM, Joshua Watt wrote: > > > On 2/26/20 11:46 PM, Douglas Royds wrote: >> >> On 26/02/20 4:53 am, Jacob Kroon wrote: >> >>> On 2/24/20 8:25 AM, Jacob Kroon wrote: >>>> Hi Douglas, >>>> >>>> You updated a comment in reproducible_build.bbclass, commit >>>> e7b891b76954c784f5a93bd0a1c91315673ce40d: >>>> >>>>> -# Once the value of SOURCE_DATE_EPOCH is determined, it is stored >>>>> in the recipe's ${SDE_FILE}. >>>>> +# Once the value of SOURCE_DATE_EPOCH is determined, it is stored >>>>> in the recipe's SDE_FILE. >>>>> +# If none of these mechanisms are suitable, replace the >>>>> do_deploy_source_date_epoch task >>>>> +# with recipe-specific functionality to write the appropriate >>>>> SOURCE_DATE_EPOCH into the SDE_FILE. >>>>> +# >>>> >>>> But I can't really get this to work. What did work for me was to >>>> replace "do_create_source_date_epoch_stamp()" in my recipe: >>>> >>>> do_create_source_date_epoch_stamp() { >>>> mkdir -p ${SDE_DIR} >>>> date -d "1981-03-03" "+%s" > ${SDE_FILE} >>>> } >>>> >>>> What is the intended way to achieve the thing I'm trying to do here ? >>>> >>> >>> FYI, JPEW has a proposed patch here >>> >>> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=d091d2aa53ea417f70c10f5ce89151820c3db9ce >>> >>> >>> for allowing a recipe to just set SOURCE_DATE_EPOCH directly. >>> >>> But maybe that currently is at odds with SOURCE_DATE_EPOCH being in >>> BB_HASHBASE_WHITELIST ? >>> >>> /Jacob >> >> >> On the surface of it, my comment appears to be just wrong: It does >> make sense to replace do_create_source_date_epoch_stamp() as you suggest. >> >> Joshua's proposed patch looks promising: >> >> * Should the new function not be called first, so that it takes >> priority over the git, known files, and youngest file functions? >> If someone has explicitly set SOURCE_DATE_EPOCH, then they want >> it to take priority. >> > Having that be the first option makes sense. The only case in which > that might not work, is if a recipe does something like: > > SOURCE_DATE_EPOCH = "${@my_awesome_sde_calculation(d)}" > > e.g. uses a function to get the SDE instead of setting to a fixed > value, but that's probably going to be extremely rare. > > >> * >> >> >> >> * As you observe, SOURCE_DATE_EPOCH would need to be removed from >> BB_HASHBASE_WHITELIST. I'm not sure why it was in the whitelist >> in the first place. >> > I'm not sure why exactly it is whitelisted; I didn't write the > original code that whitelisted it, but I've CC'd Juro in case he > happens to remember. > After a discussion with Richard, we figured out why SOURCE_DATE_EPOCH has to be whitelisted. The value of the variable *must* be calculable at parse time before any task is ran, but in practice it's value is only available once the __source_date_epoch.txt file is present, which is after parsing. This causes the taskhash to be calculated differently during parsing and task execution which causes taskhash mismatch errors. [-- Attachment #2: Type: text/html, Size: 5126 bytes --] ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Overriding SDE_FILE 2020-02-27 16:45 ` Joshua Watt @ 2020-02-27 21:22 ` Douglas Royds 2020-02-27 21:49 ` Joshua Watt 0 siblings, 1 reply; 14+ messages in thread From: Douglas Royds @ 2020-02-27 21:22 UTC (permalink / raw) To: Joshua Watt; +Cc: openembedded-core [-- Attachment #1: Type: text/plain, Size: 3397 bytes --] On 28/02/20 5:45 am, Joshua Watt wrote: > On 2/27/20 9:01 AM, Joshua Watt wrote: >> On 2/26/20 11:46 PM, Douglas Royds wrote: >>> >>> On 26/02/20 4:53 am, Jacob Kroon wrote: >>> >>>> On 2/24/20 8:25 AM, Jacob Kroon wrote: >>>>> Hi Douglas, >>>>> >>>>> You updated a comment in reproducible_build.bbclass, commit >>>>> e7b891b76954c784f5a93bd0a1c91315673ce40d: >>>>> >>>>>> -# Once the value of SOURCE_DATE_EPOCH is determined, it is >>>>>> stored in the recipe's ${SDE_FILE}. >>>>>> +# Once the value of SOURCE_DATE_EPOCH is determined, it is >>>>>> stored in the recipe's SDE_FILE. >>>>>> +# If none of these mechanisms are suitable, replace the >>>>>> do_deploy_source_date_epoch task >>>>>> +# with recipe-specific functionality to write the appropriate >>>>>> SOURCE_DATE_EPOCH into the SDE_FILE. >>>>>> +# >>>>> >>>>> But I can't really get this to work. What did work for me was to >>>>> replace "do_create_source_date_epoch_stamp()" in my recipe: >>>>> >>>>> do_create_source_date_epoch_stamp() { >>>>> mkdir -p ${SDE_DIR} >>>>> date -d "1981-03-03" "+%s" > ${SDE_FILE} >>>>> } >>>>> >>>>> What is the intended way to achieve the thing I'm trying to do here ? >>>>> >>>> >>>> FYI, JPEW has a proposed patch here >>>> >>>> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=d091d2aa53ea417f70c10f5ce89151820c3db9ce >>>> >>>> >>>> for allowing a recipe to just set SOURCE_DATE_EPOCH directly. >>>> >>>> But maybe that currently is at odds with SOURCE_DATE_EPOCH being in >>>> BB_HASHBASE_WHITELIST ? >>>> >>>> /Jacob >>> >>> >>> On the surface of it, my comment appears to be just wrong: It does >>> make sense to replace do_create_source_date_epoch_stamp() as you >>> suggest. >>> >>> Joshua's proposed patch looks promising: >>> >>> * Should the new function not be called first, so that it takes >>> priority over the git, known files, and youngest file functions? >>> If someone has explicitly set SOURCE_DATE_EPOCH, then they want >>> it to take priority. >>> >> Having that be the first option makes sense. The only case in which >> that might not work, is if a recipe does something like: >> >> SOURCE_DATE_EPOCH = "${@my_awesome_sde_calculation(d)}" >> >> e.g. uses a function to get the SDE instead of setting to a fixed >> value, but that's probably going to be extremely rare. >> >> >>> * >>> >>> >>> >>> * As you observe, SOURCE_DATE_EPOCH would need to be removed from >>> BB_HASHBASE_WHITELIST. I'm not sure why it was in the whitelist >>> in the first place. >>> >> I'm not sure why exactly it is whitelisted; I didn't write the >> original code that whitelisted it, but I've CC'd Juro in case he >> happens to remember. >> > After a discussion with Richard, we figured out why SOURCE_DATE_EPOCH > has to be whitelisted. The value of the variable *must* be calculable > at parse time before any task is ran, but in practice it's value is > only available once the __source_date_epoch.txt file is present, which > is after parsing. This causes the taskhash to be calculated > differently during parsing and task execution which causes taskhash > mismatch errors. > True. What to do? Would it work to use a different non-whitelisted variable in the recipe, eg. SOURCE_DATE_EPOCH_FIXED? [-- Attachment #2: Type: text/html, Size: 5915 bytes --] ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Overriding SDE_FILE 2020-02-27 21:22 ` Douglas Royds @ 2020-02-27 21:49 ` Joshua Watt 2020-02-27 22:29 ` Douglas Royds 0 siblings, 1 reply; 14+ messages in thread From: Joshua Watt @ 2020-02-27 21:49 UTC (permalink / raw) To: Douglas Royds; +Cc: openembedded-core [-- Attachment #1: Type: text/plain, Size: 4012 bytes --] On 2/27/20 3:22 PM, Douglas Royds wrote: > > On 28/02/20 5:45 am, Joshua Watt wrote: > >> On 2/27/20 9:01 AM, Joshua Watt wrote: >>> On 2/26/20 11:46 PM, Douglas Royds wrote: >>>> >>>> On 26/02/20 4:53 am, Jacob Kroon wrote: >>>> >>>>> On 2/24/20 8:25 AM, Jacob Kroon wrote: >>>>>> Hi Douglas, >>>>>> >>>>>> You updated a comment in reproducible_build.bbclass, commit >>>>>> e7b891b76954c784f5a93bd0a1c91315673ce40d: >>>>>> >>>>>>> -# Once the value of SOURCE_DATE_EPOCH is determined, it is >>>>>>> stored in the recipe's ${SDE_FILE}. >>>>>>> +# Once the value of SOURCE_DATE_EPOCH is determined, it is >>>>>>> stored in the recipe's SDE_FILE. >>>>>>> +# If none of these mechanisms are suitable, replace the >>>>>>> do_deploy_source_date_epoch task >>>>>>> +# with recipe-specific functionality to write the appropriate >>>>>>> SOURCE_DATE_EPOCH into the SDE_FILE. >>>>>>> +# >>>>>> >>>>>> But I can't really get this to work. What did work for me was to >>>>>> replace "do_create_source_date_epoch_stamp()" in my recipe: >>>>>> >>>>>> do_create_source_date_epoch_stamp() { >>>>>> mkdir -p ${SDE_DIR} >>>>>> date -d "1981-03-03" "+%s" > ${SDE_FILE} >>>>>> } >>>>>> >>>>>> What is the intended way to achieve the thing I'm trying to do >>>>>> here ? >>>>>> >>>>> >>>>> FYI, JPEW has a proposed patch here >>>>> >>>>> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=d091d2aa53ea417f70c10f5ce89151820c3db9ce >>>>> >>>>> >>>>> for allowing a recipe to just set SOURCE_DATE_EPOCH directly. >>>>> >>>>> But maybe that currently is at odds with SOURCE_DATE_EPOCH being >>>>> in BB_HASHBASE_WHITELIST ? >>>>> >>>>> /Jacob >>>> >>>> >>>> On the surface of it, my comment appears to be just wrong: It does >>>> make sense to replace do_create_source_date_epoch_stamp() as you >>>> suggest. >>>> >>>> Joshua's proposed patch looks promising: >>>> >>>> * Should the new function not be called first, so that it takes >>>> priority over the git, known files, and youngest file >>>> functions? If someone has explicitly set SOURCE_DATE_EPOCH, >>>> then they want it to take priority. >>>> >>> Having that be the first option makes sense. The only case in which >>> that might not work, is if a recipe does something like: >>> >>> SOURCE_DATE_EPOCH = "${@my_awesome_sde_calculation(d)}" >>> >>> e.g. uses a function to get the SDE instead of setting to a fixed >>> value, but that's probably going to be extremely rare. >>> >>> >>>> * >>>> >>>> >>>> >>>> * As you observe, SOURCE_DATE_EPOCH would need to be removed from >>>> BB_HASHBASE_WHITELIST. I'm not sure why it was in the whitelist >>>> in the first place. >>>> >>> I'm not sure why exactly it is whitelisted; I didn't write the >>> original code that whitelisted it, but I've CC'd Juro in case he >>> happens to remember. >>> >> After a discussion with Richard, we figured out why SOURCE_DATE_EPOCH >> has to be whitelisted. The value of the variable *must* be calculable >> at parse time before any task is ran, but in practice it's value is >> only available once the __source_date_epoch.txt file is present, >> which is after parsing. This causes the taskhash to be calculated >> differently during parsing and task execution which causes taskhash >> mismatch errors. >> > > True. > > What to do? Would it work to use a different non-whitelisted variable > in the recipe, eg. SOURCE_DATE_EPOCH_FIXED? > Ya, that would work. You'd have to figure out how to get the variable to be included in each taskhash even though it's not directly referenced, but I'm sure that's possible. Another option that's at lot more "magic" would be something like this: http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=2b524916cf35238ff3deea34017e8a4cd73926cd That's really weird, and I'm not sure I like it, but worth a thought. [-- Attachment #2: Type: text/html, Size: 7012 bytes --] ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Overriding SDE_FILE 2020-02-27 21:49 ` Joshua Watt @ 2020-02-27 22:29 ` Douglas Royds 2020-02-27 22:34 ` Joshua Watt 0 siblings, 1 reply; 14+ messages in thread From: Douglas Royds @ 2020-02-27 22:29 UTC (permalink / raw) To: Joshua Watt; +Cc: openembedded-core [-- Attachment #1: Type: text/plain, Size: 4410 bytes --] On 28/02/20 10:49 am, Joshua Watt wrote: > > On 2/27/20 3:22 PM, Douglas Royds wrote: >> >> On 28/02/20 5:45 am, Joshua Watt wrote: >> >>> On 2/27/20 9:01 AM, Joshua Watt wrote: >>>> On 2/26/20 11:46 PM, Douglas Royds wrote: >>>>> >>>>> On 26/02/20 4:53 am, Jacob Kroon wrote: >>>>> >>>>>> On 2/24/20 8:25 AM, Jacob Kroon wrote: >>>>>>> Hi Douglas, >>>>>>> >>>>>>> You updated a comment in reproducible_build.bbclass, commit >>>>>>> e7b891b76954c784f5a93bd0a1c91315673ce40d: >>>>>>> >>>>>>>> -# Once the value of SOURCE_DATE_EPOCH is determined, it is >>>>>>>> stored in the recipe's ${SDE_FILE}. >>>>>>>> +# Once the value of SOURCE_DATE_EPOCH is determined, it is >>>>>>>> stored in the recipe's SDE_FILE. >>>>>>>> +# If none of these mechanisms are suitable, replace the >>>>>>>> do_deploy_source_date_epoch task >>>>>>>> +# with recipe-specific functionality to write the appropriate >>>>>>>> SOURCE_DATE_EPOCH into the SDE_FILE. >>>>>>>> +# >>>>>>> >>>>>>> But I can't really get this to work. What did work for me was to >>>>>>> replace "do_create_source_date_epoch_stamp()" in my recipe: >>>>>>> >>>>>>> do_create_source_date_epoch_stamp() { >>>>>>> mkdir -p ${SDE_DIR} >>>>>>> date -d "1981-03-03" "+%s" > ${SDE_FILE} >>>>>>> } >>>>>>> >>>>>>> What is the intended way to achieve the thing I'm trying to do >>>>>>> here ? >>>>>>> >>>>>> >>>>>> FYI, JPEW has a proposed patch here >>>>>> >>>>>> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=d091d2aa53ea417f70c10f5ce89151820c3db9ce >>>>>> >>>>>> >>>>>> for allowing a recipe to just set SOURCE_DATE_EPOCH directly. >>>>>> >>>>>> But maybe that currently is at odds with SOURCE_DATE_EPOCH being >>>>>> in BB_HASHBASE_WHITELIST ? >>>>>> >>>>>> /Jacob >>>>> >>>>> >>>>> On the surface of it, my comment appears to be just wrong: It does >>>>> make sense to replace do_create_source_date_epoch_stamp() as you >>>>> suggest. >>>>> >>>>> Joshua's proposed patch looks promising: >>>>> >>>>> * Should the new function not be called first, so that it takes >>>>> priority over the git, known files, and youngest file >>>>> functions? If someone has explicitly set SOURCE_DATE_EPOCH, >>>>> then they want it to take priority. >>>>> >>>> Having that be the first option makes sense. The only case in which >>>> that might not work, is if a recipe does something like: >>>> >>>> SOURCE_DATE_EPOCH = "${@my_awesome_sde_calculation(d)}" >>>> >>>> e.g. uses a function to get the SDE instead of setting to a fixed >>>> value, but that's probably going to be extremely rare. >>>> >>>> >>>>> * >>>>> >>>>> >>>>> >>>>> * As you observe, SOURCE_DATE_EPOCH would need to be removed >>>>> from BB_HASHBASE_WHITELIST. I'm not sure why it was in the >>>>> whitelist in the first place. >>>>> >>>> I'm not sure why exactly it is whitelisted; I didn't write the >>>> original code that whitelisted it, but I've CC'd Juro in case he >>>> happens to remember. >>>> >>> After a discussion with Richard, we figured out why >>> SOURCE_DATE_EPOCH has to be whitelisted. The value of the variable >>> *must* be calculable at parse time before any task is ran, but in >>> practice it's value is only available once the >>> __source_date_epoch.txt file is present, which is after parsing. >>> This causes the taskhash to be calculated differently during parsing >>> and task execution which causes taskhash mismatch errors. >>> >> >> True. >> >> What to do? Would it work to use a different non-whitelisted variable >> in the recipe, eg. SOURCE_DATE_EPOCH_FIXED? >> > > Ya, that would work. You'd have to figure out how to get the variable > to be included in each taskhash even though it's not directly > referenced, but I'm sure that's possible. > > > Another option that's at lot more "magic" would be something like > this: > http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=2b524916cf35238ff3deea34017e8a4cd73926cd > > > That's really weird, and I'm not sure I like it, but worth a thought. > If it works, it's good. I like the fact that the user can just set SOURCE_DATE_EPOCH directly in their recipe. How about ... BB_HASHBASE_WHITELIST += "${@'SOURCE_DATE_EPOCH' if not source_date_epoch_var(d) else ''}" [-- Attachment #2: Type: text/html, Size: 8214 bytes --] ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Overriding SDE_FILE 2020-02-27 22:29 ` Douglas Royds @ 2020-02-27 22:34 ` Joshua Watt 2020-02-27 22:35 ` Douglas Royds 0 siblings, 1 reply; 14+ messages in thread From: Joshua Watt @ 2020-02-27 22:34 UTC (permalink / raw) To: Douglas Royds; +Cc: openembedded-core [-- Attachment #1: Type: text/plain, Size: 4712 bytes --] On 2/27/20 4:29 PM, Douglas Royds wrote: > > On 28/02/20 10:49 am, Joshua Watt wrote: > >> >> On 2/27/20 3:22 PM, Douglas Royds wrote: >>> >>> On 28/02/20 5:45 am, Joshua Watt wrote: >>> >>>> On 2/27/20 9:01 AM, Joshua Watt wrote: >>>>> On 2/26/20 11:46 PM, Douglas Royds wrote: >>>>>> >>>>>> On 26/02/20 4:53 am, Jacob Kroon wrote: >>>>>> >>>>>>> On 2/24/20 8:25 AM, Jacob Kroon wrote: >>>>>>>> Hi Douglas, >>>>>>>> >>>>>>>> You updated a comment in reproducible_build.bbclass, commit >>>>>>>> e7b891b76954c784f5a93bd0a1c91315673ce40d: >>>>>>>> >>>>>>>>> -# Once the value of SOURCE_DATE_EPOCH is determined, it is >>>>>>>>> stored in the recipe's ${SDE_FILE}. >>>>>>>>> +# Once the value of SOURCE_DATE_EPOCH is determined, it is >>>>>>>>> stored in the recipe's SDE_FILE. >>>>>>>>> +# If none of these mechanisms are suitable, replace the >>>>>>>>> do_deploy_source_date_epoch task >>>>>>>>> +# with recipe-specific functionality to write the appropriate >>>>>>>>> SOURCE_DATE_EPOCH into the SDE_FILE. >>>>>>>>> +# >>>>>>>> >>>>>>>> But I can't really get this to work. What did work for me was >>>>>>>> to replace "do_create_source_date_epoch_stamp()" in my recipe: >>>>>>>> >>>>>>>> do_create_source_date_epoch_stamp() { >>>>>>>> mkdir -p ${SDE_DIR} >>>>>>>> date -d "1981-03-03" "+%s" > ${SDE_FILE} >>>>>>>> } >>>>>>>> >>>>>>>> What is the intended way to achieve the thing I'm trying to do >>>>>>>> here ? >>>>>>>> >>>>>>> >>>>>>> FYI, JPEW has a proposed patch here >>>>>>> >>>>>>> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=d091d2aa53ea417f70c10f5ce89151820c3db9ce >>>>>>> >>>>>>> >>>>>>> for allowing a recipe to just set SOURCE_DATE_EPOCH directly. >>>>>>> >>>>>>> But maybe that currently is at odds with SOURCE_DATE_EPOCH being >>>>>>> in BB_HASHBASE_WHITELIST ? >>>>>>> >>>>>>> /Jacob >>>>>> >>>>>> >>>>>> On the surface of it, my comment appears to be just wrong: It >>>>>> does make sense to replace do_create_source_date_epoch_stamp() as >>>>>> you suggest. >>>>>> >>>>>> Joshua's proposed patch looks promising: >>>>>> >>>>>> * Should the new function not be called first, so that it takes >>>>>> priority over the git, known files, and youngest file >>>>>> functions? If someone has explicitly set SOURCE_DATE_EPOCH, >>>>>> then they want it to take priority. >>>>>> >>>>> Having that be the first option makes sense. The only case in >>>>> which that might not work, is if a recipe does something like: >>>>> >>>>> SOURCE_DATE_EPOCH = "${@my_awesome_sde_calculation(d)}" >>>>> >>>>> e.g. uses a function to get the SDE instead of setting to a fixed >>>>> value, but that's probably going to be extremely rare. >>>>> >>>>> >>>>>> * >>>>>> >>>>>> >>>>>> >>>>>> * As you observe, SOURCE_DATE_EPOCH would need to be removed >>>>>> from BB_HASHBASE_WHITELIST. I'm not sure why it was in the >>>>>> whitelist in the first place. >>>>>> >>>>> I'm not sure why exactly it is whitelisted; I didn't write the >>>>> original code that whitelisted it, but I've CC'd Juro in case he >>>>> happens to remember. >>>>> >>>> After a discussion with Richard, we figured out why >>>> SOURCE_DATE_EPOCH has to be whitelisted. The value of the variable >>>> *must* be calculable at parse time before any task is ran, but in >>>> practice it's value is only available once the >>>> __source_date_epoch.txt file is present, which is after parsing. >>>> This causes the taskhash to be calculated differently during >>>> parsing and task execution which causes taskhash mismatch errors. >>>> >>> >>> True. >>> >>> What to do? Would it work to use a different non-whitelisted >>> variable in the recipe, eg. SOURCE_DATE_EPOCH_FIXED? >>> >> >> Ya, that would work. You'd have to figure out how to get the variable >> to be included in each taskhash even though it's not directly >> referenced, but I'm sure that's possible. >> >> >> Another option that's at lot more "magic" would be something like >> this: >> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=2b524916cf35238ff3deea34017e8a4cd73926cd >> >> >> That's really weird, and I'm not sure I like it, but worth a thought. >> > If it works, it's good. I like the fact that the user can just set > SOURCE_DATE_EPOCH directly in their recipe. > > How about ... > > BB_HASHBASE_WHITELIST += "${@'SOURCE_DATE_EPOCH' if not > source_date_epoch_var(d) else ''}" > Ya, that works. The only reason I chose "is None" was in case a user really wanted to do SOURCE_DATE_EPOCH = "0" [-- Attachment #2: Type: text/html, Size: 8780 bytes --] ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Overriding SDE_FILE 2020-02-27 22:34 ` Joshua Watt @ 2020-02-27 22:35 ` Douglas Royds 2020-02-28 13:39 ` Jacob Kroon 0 siblings, 1 reply; 14+ messages in thread From: Douglas Royds @ 2020-02-27 22:35 UTC (permalink / raw) To: Joshua Watt; +Cc: openembedded-core [-- Attachment #1: Type: text/plain, Size: 4916 bytes --] On 28/02/20 11:34 am, Joshua Watt wrote: > > On 2/27/20 4:29 PM, Douglas Royds wrote: >> >> On 28/02/20 10:49 am, Joshua Watt wrote: >> >>> >>> On 2/27/20 3:22 PM, Douglas Royds wrote: >>>> >>>> On 28/02/20 5:45 am, Joshua Watt wrote: >>>> >>>>> On 2/27/20 9:01 AM, Joshua Watt wrote: >>>>>> On 2/26/20 11:46 PM, Douglas Royds wrote: >>>>>>> >>>>>>> On 26/02/20 4:53 am, Jacob Kroon wrote: >>>>>>> >>>>>>>> On 2/24/20 8:25 AM, Jacob Kroon wrote: >>>>>>>>> Hi Douglas, >>>>>>>>> >>>>>>>>> You updated a comment in reproducible_build.bbclass, commit >>>>>>>>> e7b891b76954c784f5a93bd0a1c91315673ce40d: >>>>>>>>> >>>>>>>>>> -# Once the value of SOURCE_DATE_EPOCH is determined, it is >>>>>>>>>> stored in the recipe's ${SDE_FILE}. >>>>>>>>>> +# Once the value of SOURCE_DATE_EPOCH is determined, it is >>>>>>>>>> stored in the recipe's SDE_FILE. >>>>>>>>>> +# If none of these mechanisms are suitable, replace the >>>>>>>>>> do_deploy_source_date_epoch task >>>>>>>>>> +# with recipe-specific functionality to write the >>>>>>>>>> appropriate SOURCE_DATE_EPOCH into the SDE_FILE. >>>>>>>>>> +# >>>>>>>>> >>>>>>>>> But I can't really get this to work. What did work for me was >>>>>>>>> to replace "do_create_source_date_epoch_stamp()" in my recipe: >>>>>>>>> >>>>>>>>> do_create_source_date_epoch_stamp() { >>>>>>>>> mkdir -p ${SDE_DIR} >>>>>>>>> date -d "1981-03-03" "+%s" > ${SDE_FILE} >>>>>>>>> } >>>>>>>>> >>>>>>>>> What is the intended way to achieve the thing I'm trying to do >>>>>>>>> here ? >>>>>>>>> >>>>>>>> >>>>>>>> FYI, JPEW has a proposed patch here >>>>>>>> >>>>>>>> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=d091d2aa53ea417f70c10f5ce89151820c3db9ce >>>>>>>> >>>>>>>> >>>>>>>> for allowing a recipe to just set SOURCE_DATE_EPOCH directly. >>>>>>>> >>>>>>>> But maybe that currently is at odds with SOURCE_DATE_EPOCH >>>>>>>> being in BB_HASHBASE_WHITELIST ? >>>>>>>> >>>>>>>> /Jacob >>>>>>> >>>>>>> >>>>>>> On the surface of it, my comment appears to be just wrong: It >>>>>>> does make sense to replace do_create_source_date_epoch_stamp() >>>>>>> as you suggest. >>>>>>> >>>>>>> Joshua's proposed patch looks promising: >>>>>>> >>>>>>> * Should the new function not be called first, so that it >>>>>>> takes priority over the git, known files, and youngest file >>>>>>> functions? If someone has explicitly set SOURCE_DATE_EPOCH, >>>>>>> then they want it to take priority. >>>>>>> >>>>>> Having that be the first option makes sense. The only case in >>>>>> which that might not work, is if a recipe does something like: >>>>>> >>>>>> SOURCE_DATE_EPOCH = "${@my_awesome_sde_calculation(d)}" >>>>>> >>>>>> e.g. uses a function to get the SDE instead of setting to a fixed >>>>>> value, but that's probably going to be extremely rare. >>>>>> >>>>>> >>>>>>> * >>>>>>> >>>>>>> >>>>>>> >>>>>>> * As you observe, SOURCE_DATE_EPOCH would need to be removed >>>>>>> from BB_HASHBASE_WHITELIST. I'm not sure why it was in the >>>>>>> whitelist in the first place. >>>>>>> >>>>>> I'm not sure why exactly it is whitelisted; I didn't write the >>>>>> original code that whitelisted it, but I've CC'd Juro in case he >>>>>> happens to remember. >>>>>> >>>>> After a discussion with Richard, we figured out why >>>>> SOURCE_DATE_EPOCH has to be whitelisted. The value of the variable >>>>> *must* be calculable at parse time before any task is ran, but in >>>>> practice it's value is only available once the >>>>> __source_date_epoch.txt file is present, which is after parsing. >>>>> This causes the taskhash to be calculated differently during >>>>> parsing and task execution which causes taskhash mismatch errors. >>>>> >>>> >>>> True. >>>> >>>> What to do? Would it work to use a different non-whitelisted >>>> variable in the recipe, eg. SOURCE_DATE_EPOCH_FIXED? >>>> >>> >>> Ya, that would work. You'd have to figure out how to get the >>> variable to be included in each taskhash even though it's not >>> directly referenced, but I'm sure that's possible. >>> >>> >>> Another option that's at lot more "magic" would be something like >>> this: >>> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=2b524916cf35238ff3deea34017e8a4cd73926cd >>> >>> >>> That's really weird, and I'm not sure I like it, but worth a thought. >>> >> If it works, it's good. I like the fact that the user can just set >> SOURCE_DATE_EPOCH directly in their recipe. >> >> How about ... >> >> BB_HASHBASE_WHITELIST += "${@'SOURCE_DATE_EPOCH' if not >> source_date_epoch_var(d) else ''}" >> > Ya, that works. The only reason I chose "is None" was in case a user > really wanted to do > > SOURCE_DATE_EPOCH = "0" > True, best to allow for that. [-- Attachment #2: Type: text/html, Size: 9882 bytes --] ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Overriding SDE_FILE 2020-02-27 22:35 ` Douglas Royds @ 2020-02-28 13:39 ` Jacob Kroon 2020-02-28 13:41 ` Jacob Kroon 0 siblings, 1 reply; 14+ messages in thread From: Jacob Kroon @ 2020-02-28 13:39 UTC (permalink / raw) To: Douglas Royds, Joshua Watt; +Cc: openembedded-core On 2/27/20 11:35 PM, Douglas Royds via Openembedded-core wrote: > On 28/02/20 11:34 am, Joshua Watt wrote: > >> >> On 2/27/20 4:29 PM, Douglas Royds wrote: >>> >>> On 28/02/20 10:49 am, Joshua Watt wrote: >>> >>>> >>>> On 2/27/20 3:22 PM, Douglas Royds wrote: >>>>> >>>>> On 28/02/20 5:45 am, Joshua Watt wrote: >>>>> >>>>>> On 2/27/20 9:01 AM, Joshua Watt wrote: >>>>>>> On 2/26/20 11:46 PM, Douglas Royds wrote: >>>>>>>> >>>>>>>> On 26/02/20 4:53 am, Jacob Kroon wrote: >>>>>>>> >>>>>>>>> On 2/24/20 8:25 AM, Jacob Kroon wrote: >>>>>>>>>> Hi Douglas, >>>>>>>>>> >>>>>>>>>> You updated a comment in reproducible_build.bbclass, commit >>>>>>>>>> e7b891b76954c784f5a93bd0a1c91315673ce40d: >>>>>>>>>> >>>>>>>>>>> -# Once the value of SOURCE_DATE_EPOCH is determined, it is >>>>>>>>>>> stored in the recipe's ${SDE_FILE}. >>>>>>>>>>> +# Once the value of SOURCE_DATE_EPOCH is determined, it is >>>>>>>>>>> stored in the recipe's SDE_FILE. >>>>>>>>>>> +# If none of these mechanisms are suitable, replace the >>>>>>>>>>> do_deploy_source_date_epoch task >>>>>>>>>>> +# with recipe-specific functionality to write the >>>>>>>>>>> appropriate SOURCE_DATE_EPOCH into the SDE_FILE. >>>>>>>>>>> +# >>>>>>>>>> >>>>>>>>>> But I can't really get this to work. What did work for me was >>>>>>>>>> to replace "do_create_source_date_epoch_stamp()" in my recipe: >>>>>>>>>> >>>>>>>>>> do_create_source_date_epoch_stamp() { >>>>>>>>>> mkdir -p ${SDE_DIR} >>>>>>>>>> date -d "1981-03-03" "+%s" > ${SDE_FILE} >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> What is the intended way to achieve the thing I'm trying to do >>>>>>>>>> here ? >>>>>>>>>> >>>>>>>>> >>>>>>>>> FYI, JPEW has a proposed patch here >>>>>>>>> >>>>>>>>> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=d091d2aa53ea417f70c10f5ce89151820c3db9ce >>>>>>>>> >>>>>>>>> >>>>>>>>> for allowing a recipe to just set SOURCE_DATE_EPOCH directly. >>>>>>>>> >>>>>>>>> But maybe that currently is at odds with SOURCE_DATE_EPOCH >>>>>>>>> being in BB_HASHBASE_WHITELIST ? >>>>>>>>> >>>>>>>>> /Jacob >>>>>>>> >>>>>>>> >>>>>>>> On the surface of it, my comment appears to be just wrong: It >>>>>>>> does make sense to replace do_create_source_date_epoch_stamp() >>>>>>>> as you suggest. >>>>>>>> >>>>>>>> Joshua's proposed patch looks promising: >>>>>>>> >>>>>>>> * Should the new function not be called first, so that it >>>>>>>> takes priority over the git, known files, and youngest file >>>>>>>> functions? If someone has explicitly set SOURCE_DATE_EPOCH, >>>>>>>> then they want it to take priority. >>>>>>>> >>>>>>> Having that be the first option makes sense. The only case in >>>>>>> which that might not work, is if a recipe does something like: >>>>>>> >>>>>>> SOURCE_DATE_EPOCH = "${@my_awesome_sde_calculation(d)}" >>>>>>> >>>>>>> e.g. uses a function to get the SDE instead of setting to a fixed >>>>>>> value, but that's probably going to be extremely rare. >>>>>>> >>>>>>> >>>>>>>> * >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> * As you observe, SOURCE_DATE_EPOCH would need to be removed >>>>>>>> from BB_HASHBASE_WHITELIST. I'm not sure why it was in the >>>>>>>> whitelist in the first place. >>>>>>>> >>>>>>> I'm not sure why exactly it is whitelisted; I didn't write the >>>>>>> original code that whitelisted it, but I've CC'd Juro in case he >>>>>>> happens to remember. >>>>>>> >>>>>> After a discussion with Richard, we figured out why >>>>>> SOURCE_DATE_EPOCH has to be whitelisted. The value of the variable >>>>>> *must* be calculable at parse time before any task is ran, but in >>>>>> practice it's value is only available once the >>>>>> __source_date_epoch.txt file is present, which is after parsing. >>>>>> This causes the taskhash to be calculated differently during >>>>>> parsing and task execution which causes taskhash mismatch errors. >>>>>> >>>>> >>>>> True. >>>>> >>>>> What to do? Would it work to use a different non-whitelisted >>>>> variable in the recipe, eg. SOURCE_DATE_EPOCH_FIXED? >>>>> >>>> >>>> Ya, that would work. You'd have to figure out how to get the >>>> variable to be included in each taskhash even though it's not >>>> directly referenced, but I'm sure that's possible. >>>> >>>> >>>> Another option that's at lot more "magic" would be something like >>>> this: >>>> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=2b524916cf35238ff3deea34017e8a4cd73926cd >>>> >>>> >>>> That's really weird, and I'm not sure I like it, but worth a thought. >>>> >>> If it works, it's good. I like the fact that the user can just set >>> SOURCE_DATE_EPOCH directly in their recipe. >>> >>> How about ... >>> >>> BB_HASHBASE_WHITELIST += "${@'SOURCE_DATE_EPOCH' if not >>> source_date_epoch_var(d) else ''}" >>> >> Ya, that works. The only reason I chose "is None" was in case a user >> really wanted to do >> >> SOURCE_DATE_EPOCH = "0" >> > True, best to allow for that. > Joshua, I just tried the two top patches in your reproducible branch; I can see that SOURCE_DATE_EPOCH is not in BB_HASHBASE_WHITELIST, however, changing its value in my recipe doesn't trigger rebuilding for some reason.. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Overriding SDE_FILE 2020-02-28 13:39 ` Jacob Kroon @ 2020-02-28 13:41 ` Jacob Kroon 2020-02-28 14:18 ` Joshua Watt 0 siblings, 1 reply; 14+ messages in thread From: Jacob Kroon @ 2020-02-28 13:41 UTC (permalink / raw) To: Douglas Royds, Joshua Watt; +Cc: openembedded-core On 2/28/20 2:39 PM, Jacob Kroon wrote: > On 2/27/20 11:35 PM, Douglas Royds via Openembedded-core wrote: >> On 28/02/20 11:34 am, Joshua Watt wrote: >> >>> >>> On 2/27/20 4:29 PM, Douglas Royds wrote: >>>> >>>> On 28/02/20 10:49 am, Joshua Watt wrote: >>>> >>>>> >>>>> On 2/27/20 3:22 PM, Douglas Royds wrote: >>>>>> >>>>>> On 28/02/20 5:45 am, Joshua Watt wrote: >>>>>> >>>>>>> On 2/27/20 9:01 AM, Joshua Watt wrote: >>>>>>>> On 2/26/20 11:46 PM, Douglas Royds wrote: >>>>>>>>> >>>>>>>>> On 26/02/20 4:53 am, Jacob Kroon wrote: >>>>>>>>> >>>>>>>>>> On 2/24/20 8:25 AM, Jacob Kroon wrote: >>>>>>>>>>> Hi Douglas, >>>>>>>>>>> >>>>>>>>>>> You updated a comment in reproducible_build.bbclass, commit >>>>>>>>>>> e7b891b76954c784f5a93bd0a1c91315673ce40d: >>>>>>>>>>> >>>>>>>>>>>> -# Once the value of SOURCE_DATE_EPOCH is determined, it is >>>>>>>>>>>> stored in the recipe's ${SDE_FILE}. >>>>>>>>>>>> +# Once the value of SOURCE_DATE_EPOCH is determined, it is >>>>>>>>>>>> stored in the recipe's SDE_FILE. >>>>>>>>>>>> +# If none of these mechanisms are suitable, replace the >>>>>>>>>>>> do_deploy_source_date_epoch task >>>>>>>>>>>> +# with recipe-specific functionality to write the >>>>>>>>>>>> appropriate SOURCE_DATE_EPOCH into the SDE_FILE. >>>>>>>>>>>> +# >>>>>>>>>>> >>>>>>>>>>> But I can't really get this to work. What did work for me was >>>>>>>>>>> to replace "do_create_source_date_epoch_stamp()" in my recipe: >>>>>>>>>>> >>>>>>>>>>> do_create_source_date_epoch_stamp() { >>>>>>>>>>> mkdir -p ${SDE_DIR} >>>>>>>>>>> date -d "1981-03-03" "+%s" > ${SDE_FILE} >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> What is the intended way to achieve the thing I'm trying to >>>>>>>>>>> do here ? >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> FYI, JPEW has a proposed patch here >>>>>>>>>> >>>>>>>>>> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=d091d2aa53ea417f70c10f5ce89151820c3db9ce >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> for allowing a recipe to just set SOURCE_DATE_EPOCH directly. >>>>>>>>>> >>>>>>>>>> But maybe that currently is at odds with SOURCE_DATE_EPOCH >>>>>>>>>> being in BB_HASHBASE_WHITELIST ? >>>>>>>>>> >>>>>>>>>> /Jacob >>>>>>>>> >>>>>>>>> >>>>>>>>> On the surface of it, my comment appears to be just wrong: It >>>>>>>>> does make sense to replace do_create_source_date_epoch_stamp() >>>>>>>>> as you suggest. >>>>>>>>> >>>>>>>>> Joshua's proposed patch looks promising: >>>>>>>>> >>>>>>>>> * Should the new function not be called first, so that it >>>>>>>>> takes priority over the git, known files, and youngest file >>>>>>>>> functions? If someone has explicitly set SOURCE_DATE_EPOCH, >>>>>>>>> then they want it to take priority. >>>>>>>>> >>>>>>>> Having that be the first option makes sense. The only case in >>>>>>>> which that might not work, is if a recipe does something like: >>>>>>>> >>>>>>>> SOURCE_DATE_EPOCH = "${@my_awesome_sde_calculation(d)}" >>>>>>>> >>>>>>>> e.g. uses a function to get the SDE instead of setting to a >>>>>>>> fixed value, but that's probably going to be extremely rare. >>>>>>>> >>>>>>>> >>>>>>>>> * >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> * As you observe, SOURCE_DATE_EPOCH would need to be removed >>>>>>>>> from BB_HASHBASE_WHITELIST. I'm not sure why it was in the >>>>>>>>> whitelist in the first place. >>>>>>>>> >>>>>>>> I'm not sure why exactly it is whitelisted; I didn't write the >>>>>>>> original code that whitelisted it, but I've CC'd Juro in case he >>>>>>>> happens to remember. >>>>>>>> >>>>>>> After a discussion with Richard, we figured out why >>>>>>> SOURCE_DATE_EPOCH has to be whitelisted. The value of the >>>>>>> variable *must* be calculable at parse time before any task is >>>>>>> ran, but in practice it's value is only available once the >>>>>>> __source_date_epoch.txt file is present, which is after parsing. >>>>>>> This causes the taskhash to be calculated differently during >>>>>>> parsing and task execution which causes taskhash mismatch errors. >>>>>>> >>>>>> >>>>>> True. >>>>>> >>>>>> What to do? Would it work to use a different non-whitelisted >>>>>> variable in the recipe, eg. SOURCE_DATE_EPOCH_FIXED? >>>>>> >>>>> >>>>> Ya, that would work. You'd have to figure out how to get the >>>>> variable to be included in each taskhash even though it's not >>>>> directly referenced, but I'm sure that's possible. >>>>> >>>>> >>>>> Another option that's at lot more "magic" would be something like >>>>> this: >>>>> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=2b524916cf35238ff3deea34017e8a4cd73926cd >>>>> >>>>> >>>>> That's really weird, and I'm not sure I like it, but worth a thought. >>>>> >>>> If it works, it's good. I like the fact that the user can just set >>>> SOURCE_DATE_EPOCH directly in their recipe. >>>> >>>> How about ... >>>> >>>> BB_HASHBASE_WHITELIST += "${@'SOURCE_DATE_EPOCH' if not >>>> source_date_epoch_var(d) else ''}" >>>> >>> Ya, that works. The only reason I chose "is None" was in case a user >>> really wanted to do >>> >>> SOURCE_DATE_EPOCH = "0" >>> >> True, best to allow for that. >> > > Joshua, I just tried the two top patches in your reproducible branch; I > can see that SOURCE_DATE_EPOCH is not in BB_HASHBASE_WHITELIST, however, > changing its value in my recipe doesn't trigger rebuilding for some > reason.. In case it makes a difference, I only reference ${SOURCE_DATE_EPOCH} in my do_install() task. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Overriding SDE_FILE 2020-02-28 13:41 ` Jacob Kroon @ 2020-02-28 14:18 ` Joshua Watt 2020-02-28 14:32 ` Jacob Kroon 0 siblings, 1 reply; 14+ messages in thread From: Joshua Watt @ 2020-02-28 14:18 UTC (permalink / raw) To: Jacob Kroon, Douglas Royds; +Cc: openembedded-core On 2/28/20 7:41 AM, Jacob Kroon wrote: > On 2/28/20 2:39 PM, Jacob Kroon wrote: >> On 2/27/20 11:35 PM, Douglas Royds via Openembedded-core wrote: >>> On 28/02/20 11:34 am, Joshua Watt wrote: >>> >>>> >>>> On 2/27/20 4:29 PM, Douglas Royds wrote: >>>>> >>>>> On 28/02/20 10:49 am, Joshua Watt wrote: >>>>> >>>>>> >>>>>> On 2/27/20 3:22 PM, Douglas Royds wrote: >>>>>>> >>>>>>> On 28/02/20 5:45 am, Joshua Watt wrote: >>>>>>> >>>>>>>> On 2/27/20 9:01 AM, Joshua Watt wrote: >>>>>>>>> On 2/26/20 11:46 PM, Douglas Royds wrote: >>>>>>>>>> >>>>>>>>>> On 26/02/20 4:53 am, Jacob Kroon wrote: >>>>>>>>>> >>>>>>>>>>> On 2/24/20 8:25 AM, Jacob Kroon wrote: >>>>>>>>>>>> Hi Douglas, >>>>>>>>>>>> >>>>>>>>>>>> You updated a comment in reproducible_build.bbclass, commit >>>>>>>>>>>> e7b891b76954c784f5a93bd0a1c91315673ce40d: >>>>>>>>>>>> >>>>>>>>>>>>> -# Once the value of SOURCE_DATE_EPOCH is determined, it >>>>>>>>>>>>> is stored in the recipe's ${SDE_FILE}. >>>>>>>>>>>>> +# Once the value of SOURCE_DATE_EPOCH is determined, it >>>>>>>>>>>>> is stored in the recipe's SDE_FILE. >>>>>>>>>>>>> +# If none of these mechanisms are suitable, replace the >>>>>>>>>>>>> do_deploy_source_date_epoch task >>>>>>>>>>>>> +# with recipe-specific functionality to write the >>>>>>>>>>>>> appropriate SOURCE_DATE_EPOCH into the SDE_FILE. >>>>>>>>>>>>> +# >>>>>>>>>>>> >>>>>>>>>>>> But I can't really get this to work. What did work for me >>>>>>>>>>>> was to replace "do_create_source_date_epoch_stamp()" in my >>>>>>>>>>>> recipe: >>>>>>>>>>>> >>>>>>>>>>>> do_create_source_date_epoch_stamp() { >>>>>>>>>>>> mkdir -p ${SDE_DIR} >>>>>>>>>>>> date -d "1981-03-03" "+%s" > ${SDE_FILE} >>>>>>>>>>>> } >>>>>>>>>>>> >>>>>>>>>>>> What is the intended way to achieve the thing I'm trying to >>>>>>>>>>>> do here ? >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> FYI, JPEW has a proposed patch here >>>>>>>>>>> >>>>>>>>>>> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=d091d2aa53ea417f70c10f5ce89151820c3db9ce >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> for allowing a recipe to just set SOURCE_DATE_EPOCH directly. >>>>>>>>>>> >>>>>>>>>>> But maybe that currently is at odds with SOURCE_DATE_EPOCH >>>>>>>>>>> being in BB_HASHBASE_WHITELIST ? >>>>>>>>>>> >>>>>>>>>>> /Jacob >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On the surface of it, my comment appears to be just wrong: It >>>>>>>>>> does make sense to replace >>>>>>>>>> do_create_source_date_epoch_stamp() as you suggest. >>>>>>>>>> >>>>>>>>>> Joshua's proposed patch looks promising: >>>>>>>>>> >>>>>>>>>> * Should the new function not be called first, so that it >>>>>>>>>> takes priority over the git, known files, and youngest file >>>>>>>>>> functions? If someone has explicitly set SOURCE_DATE_EPOCH, >>>>>>>>>> then they want it to take priority. >>>>>>>>>> >>>>>>>>> Having that be the first option makes sense. The only case in >>>>>>>>> which that might not work, is if a recipe does something like: >>>>>>>>> >>>>>>>>> SOURCE_DATE_EPOCH = "${@my_awesome_sde_calculation(d)}" >>>>>>>>> >>>>>>>>> e.g. uses a function to get the SDE instead of setting to a >>>>>>>>> fixed value, but that's probably going to be extremely rare. >>>>>>>>> >>>>>>>>> >>>>>>>>>> * >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> * As you observe, SOURCE_DATE_EPOCH would need to be removed >>>>>>>>>> from BB_HASHBASE_WHITELIST. I'm not sure why it was in the >>>>>>>>>> whitelist in the first place. >>>>>>>>>> >>>>>>>>> I'm not sure why exactly it is whitelisted; I didn't write the >>>>>>>>> original code that whitelisted it, but I've CC'd Juro in case >>>>>>>>> he happens to remember. >>>>>>>>> >>>>>>>> After a discussion with Richard, we figured out why >>>>>>>> SOURCE_DATE_EPOCH has to be whitelisted. The value of the >>>>>>>> variable *must* be calculable at parse time before any task is >>>>>>>> ran, but in practice it's value is only available once the >>>>>>>> __source_date_epoch.txt file is present, which is after >>>>>>>> parsing. This causes the taskhash to be calculated differently >>>>>>>> during parsing and task execution which causes taskhash >>>>>>>> mismatch errors. >>>>>>>> >>>>>>> >>>>>>> True. >>>>>>> >>>>>>> What to do? Would it work to use a different non-whitelisted >>>>>>> variable in the recipe, eg. SOURCE_DATE_EPOCH_FIXED? >>>>>>> >>>>>> >>>>>> Ya, that would work. You'd have to figure out how to get the >>>>>> variable to be included in each taskhash even though it's not >>>>>> directly referenced, but I'm sure that's possible. >>>>>> >>>>>> >>>>>> Another option that's at lot more "magic" would be something like >>>>>> this: >>>>>> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=2b524916cf35238ff3deea34017e8a4cd73926cd >>>>>> >>>>>> >>>>>> That's really weird, and I'm not sure I like it, but worth a >>>>>> thought. >>>>>> >>>>> If it works, it's good. I like the fact that the user can just set >>>>> SOURCE_DATE_EPOCH directly in their recipe. >>>>> >>>>> How about ... >>>>> >>>>> BB_HASHBASE_WHITELIST += "${@'SOURCE_DATE_EPOCH' if not >>>>> source_date_epoch_var(d) else ''}" >>>>> >>>> Ya, that works. The only reason I chose "is None" was in case a >>>> user really wanted to do >>>> >>>> SOURCE_DATE_EPOCH = "0" >>>> >>> True, best to allow for that. >>> >> >> Joshua, I just tried the two top patches in your reproducible branch; >> I can see that SOURCE_DATE_EPOCH is not in BB_HASHBASE_WHITELIST, >> however, changing its value in my recipe doesn't trigger rebuilding >> for some reason.. > > In case it makes a difference, I only reference ${SOURCE_DATE_EPOCH} > in my do_install() task. I don't think it matters. My understanding is that an export variable (like SOURCE_DATE_EPOCH) is included in the taskhash of all tasks unless it is specifically whitelisted. It's possible there is some bug I haven't throughly tested it yet.... or it just can't work that way at all for some reason. You might be able to dump the task signature for your do_install task to see if SOURCE_DATE_EPOCH is being included. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Overriding SDE_FILE 2020-02-28 14:18 ` Joshua Watt @ 2020-02-28 14:32 ` Jacob Kroon 0 siblings, 0 replies; 14+ messages in thread From: Jacob Kroon @ 2020-02-28 14:32 UTC (permalink / raw) To: Joshua Watt, Douglas Royds; +Cc: openembedded-core On 2/28/20 3:18 PM, Joshua Watt wrote: > > On 2/28/20 7:41 AM, Jacob Kroon wrote: >> On 2/28/20 2:39 PM, Jacob Kroon wrote: >>> On 2/27/20 11:35 PM, Douglas Royds via Openembedded-core wrote: >>>> On 28/02/20 11:34 am, Joshua Watt wrote: >>>> >>>>> >>>>> On 2/27/20 4:29 PM, Douglas Royds wrote: >>>>>> >>>>>> On 28/02/20 10:49 am, Joshua Watt wrote: >>>>>> >>>>>>> >>>>>>> On 2/27/20 3:22 PM, Douglas Royds wrote: >>>>>>>> >>>>>>>> On 28/02/20 5:45 am, Joshua Watt wrote: >>>>>>>> >>>>>>>>> On 2/27/20 9:01 AM, Joshua Watt wrote: >>>>>>>>>> On 2/26/20 11:46 PM, Douglas Royds wrote: >>>>>>>>>>> >>>>>>>>>>> On 26/02/20 4:53 am, Jacob Kroon wrote: >>>>>>>>>>> >>>>>>>>>>>> On 2/24/20 8:25 AM, Jacob Kroon wrote: >>>>>>>>>>>>> Hi Douglas, >>>>>>>>>>>>> >>>>>>>>>>>>> You updated a comment in reproducible_build.bbclass, commit >>>>>>>>>>>>> e7b891b76954c784f5a93bd0a1c91315673ce40d: >>>>>>>>>>>>> >>>>>>>>>>>>>> -# Once the value of SOURCE_DATE_EPOCH is determined, it >>>>>>>>>>>>>> is stored in the recipe's ${SDE_FILE}. >>>>>>>>>>>>>> +# Once the value of SOURCE_DATE_EPOCH is determined, it >>>>>>>>>>>>>> is stored in the recipe's SDE_FILE. >>>>>>>>>>>>>> +# If none of these mechanisms are suitable, replace the >>>>>>>>>>>>>> do_deploy_source_date_epoch task >>>>>>>>>>>>>> +# with recipe-specific functionality to write the >>>>>>>>>>>>>> appropriate SOURCE_DATE_EPOCH into the SDE_FILE. >>>>>>>>>>>>>> +# >>>>>>>>>>>>> >>>>>>>>>>>>> But I can't really get this to work. What did work for me >>>>>>>>>>>>> was to replace "do_create_source_date_epoch_stamp()" in my >>>>>>>>>>>>> recipe: >>>>>>>>>>>>> >>>>>>>>>>>>> do_create_source_date_epoch_stamp() { >>>>>>>>>>>>> mkdir -p ${SDE_DIR} >>>>>>>>>>>>> date -d "1981-03-03" "+%s" > ${SDE_FILE} >>>>>>>>>>>>> } >>>>>>>>>>>>> >>>>>>>>>>>>> What is the intended way to achieve the thing I'm trying to >>>>>>>>>>>>> do here ? >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> FYI, JPEW has a proposed patch here >>>>>>>>>>>> >>>>>>>>>>>> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=d091d2aa53ea417f70c10f5ce89151820c3db9ce >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> for allowing a recipe to just set SOURCE_DATE_EPOCH directly. >>>>>>>>>>>> >>>>>>>>>>>> But maybe that currently is at odds with SOURCE_DATE_EPOCH >>>>>>>>>>>> being in BB_HASHBASE_WHITELIST ? >>>>>>>>>>>> >>>>>>>>>>>> /Jacob >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On the surface of it, my comment appears to be just wrong: It >>>>>>>>>>> does make sense to replace >>>>>>>>>>> do_create_source_date_epoch_stamp() as you suggest. >>>>>>>>>>> >>>>>>>>>>> Joshua's proposed patch looks promising: >>>>>>>>>>> >>>>>>>>>>> * Should the new function not be called first, so that it >>>>>>>>>>> takes priority over the git, known files, and youngest file >>>>>>>>>>> functions? If someone has explicitly set SOURCE_DATE_EPOCH, >>>>>>>>>>> then they want it to take priority. >>>>>>>>>>> >>>>>>>>>> Having that be the first option makes sense. The only case in >>>>>>>>>> which that might not work, is if a recipe does something like: >>>>>>>>>> >>>>>>>>>> SOURCE_DATE_EPOCH = "${@my_awesome_sde_calculation(d)}" >>>>>>>>>> >>>>>>>>>> e.g. uses a function to get the SDE instead of setting to a >>>>>>>>>> fixed value, but that's probably going to be extremely rare. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> * >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> * As you observe, SOURCE_DATE_EPOCH would need to be removed >>>>>>>>>>> from BB_HASHBASE_WHITELIST. I'm not sure why it was in the >>>>>>>>>>> whitelist in the first place. >>>>>>>>>>> >>>>>>>>>> I'm not sure why exactly it is whitelisted; I didn't write the >>>>>>>>>> original code that whitelisted it, but I've CC'd Juro in case >>>>>>>>>> he happens to remember. >>>>>>>>>> >>>>>>>>> After a discussion with Richard, we figured out why >>>>>>>>> SOURCE_DATE_EPOCH has to be whitelisted. The value of the >>>>>>>>> variable *must* be calculable at parse time before any task is >>>>>>>>> ran, but in practice it's value is only available once the >>>>>>>>> __source_date_epoch.txt file is present, which is after >>>>>>>>> parsing. This causes the taskhash to be calculated differently >>>>>>>>> during parsing and task execution which causes taskhash >>>>>>>>> mismatch errors. >>>>>>>>> >>>>>>>> >>>>>>>> True. >>>>>>>> >>>>>>>> What to do? Would it work to use a different non-whitelisted >>>>>>>> variable in the recipe, eg. SOURCE_DATE_EPOCH_FIXED? >>>>>>>> >>>>>>> >>>>>>> Ya, that would work. You'd have to figure out how to get the >>>>>>> variable to be included in each taskhash even though it's not >>>>>>> directly referenced, but I'm sure that's possible. >>>>>>> >>>>>>> >>>>>>> Another option that's at lot more "magic" would be something like >>>>>>> this: >>>>>>> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=2b524916cf35238ff3deea34017e8a4cd73926cd >>>>>>> >>>>>>> >>>>>>> That's really weird, and I'm not sure I like it, but worth a >>>>>>> thought. >>>>>>> >>>>>> If it works, it's good. I like the fact that the user can just set >>>>>> SOURCE_DATE_EPOCH directly in their recipe. >>>>>> >>>>>> How about ... >>>>>> >>>>>> BB_HASHBASE_WHITELIST += "${@'SOURCE_DATE_EPOCH' if not >>>>>> source_date_epoch_var(d) else ''}" >>>>>> >>>>> Ya, that works. The only reason I chose "is None" was in case a >>>>> user really wanted to do >>>>> >>>>> SOURCE_DATE_EPOCH = "0" >>>>> >>>> True, best to allow for that. >>>> >>> >>> Joshua, I just tried the two top patches in your reproducible branch; >>> I can see that SOURCE_DATE_EPOCH is not in BB_HASHBASE_WHITELIST, >>> however, changing its value in my recipe doesn't trigger rebuilding >>> for some reason.. >> >> In case it makes a difference, I only reference ${SOURCE_DATE_EPOCH} >> in my do_install() task. > > > I don't think it matters. My understanding is that an export variable > (like SOURCE_DATE_EPOCH) is included in the taskhash of all tasks unless > it is specifically whitelisted. It's possible there is some bug I > haven't throughly tested it yet.... or it just can't work that way at > all for some reason. You might be able to dump the task signature for > your do_install task to see if SOURCE_DATE_EPOCH is being included. > Using bitbake-dumpsig I can see that SOURCE_DATE_EPOCH is still in "basewhitelist:" ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2020-02-28 14:32 UTC | newest] Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-02-24 7:25 Overriding SDE_FILE Jacob Kroon 2020-02-25 15:53 ` Jacob Kroon 2020-02-27 5:46 ` Douglas Royds 2020-02-27 15:01 ` Joshua Watt 2020-02-27 16:45 ` Joshua Watt 2020-02-27 21:22 ` Douglas Royds 2020-02-27 21:49 ` Joshua Watt 2020-02-27 22:29 ` Douglas Royds 2020-02-27 22:34 ` Joshua Watt 2020-02-27 22:35 ` Douglas Royds 2020-02-28 13:39 ` Jacob Kroon 2020-02-28 13:41 ` Jacob Kroon 2020-02-28 14:18 ` Joshua Watt 2020-02-28 14:32 ` Jacob Kroon
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.