All of lore.kernel.org
 help / color / mirror / Atom feed
From: Richard Purdie <richard.purdie@linuxfoundation.org>
To: Joshua Watt <jpewhacker@gmail.com>,
	openembedded-core@lists.openembedded.org,
	bitbake-devel@lists.openembedded.org
Subject: Re: [OE-core][PATCH v3 13/17] bitbake: runqueue: Pass dependency ID to hash validate
Date: Wed, 05 Dec 2018 22:52:22 +0000	[thread overview]
Message-ID: <5d5b3aba3171a2fc2ef94157769ec0b0e46e49f9.camel@linuxfoundation.org> (raw)
In-Reply-To: <20181204034245.25461-14-JPEWhacker@gmail.com>

On Mon, 2018-12-03 at 21:42 -0600, Joshua Watt wrote:
> If the dependency ID is being used to track task dependencies, the
> hash
> validation function needs to know about it in order to properly
> validate
> the hash.
> 
> TODO: This currently isn't going to be backward compatible with older
> hashvalidate functions. Is that necessary, and if so are there any
> suggestions for a good approach?

That is necessary or bitbake and the metadata need to be in lock step.

@@ -1567,15 +1568,16 @@ class RunQueue:
>              sq_fn.append(fn)
>              sq_hashfn.append(self.rqdata.dataCaches[mc].hashfn[taskfn])
>              sq_hash.append(self.rqdata.runtaskentries[tid].hash)
> +            sq_depid.append(self.rqdata.runtaskentries[tid].depid)
>              sq_taskname.append(taskname)
>              sq_task.append(tid)
> -        locs = { "sq_fn" : sq_fn, "sq_task" : sq_taskname, "sq_hash" : sq_hash, "sq_hashfn" : sq_hashfn, "d" : self.cooker.data }
> +        locs = { "sq_fn" : sq_fn, "sq_task" : sq_taskname, "sq_hash" : sq_hash, "sq_hashfn" : sq_hashfn, "sq_depid" : sq_depid, "d" : self.cooker.data }
>          try:
> -            call = self.hashvalidate + "(sq_fn, sq_task, sq_hash, sq_hashfn, d, siginfo=True)"
> +            call = self.hashvalidate + "(sq_fn, sq_task, sq_hash, sq_hashfn, sq_depid, d, siginfo=True)"
>              valid = bb.utils.better_eval(call, locs)
>          # Handle version with no siginfo parameter
>          except TypeError:
> -            call = self.hashvalidate + "(sq_fn, sq_task, sq_hash, sq_hashfn, d)"
> +            call = self.hashvalidate + "(sq_fn, sq_task, sq_hash, sq_hashfn, sq_depid, d)"
>              valid = bb.utils.better_eval(call, locs)
>          for v in valid:
> 

The hint on how we've handled this before is in the code, the TypeError
was the fallback code for older metadata (which you just broke!). We
may need to do something similar here.

Cheers,

Richard



WARNING: multiple messages have this Message-ID (diff)
From: Richard Purdie <richard.purdie@linuxfoundation.org>
To: Joshua Watt <jpewhacker@gmail.com>,
	openembedded-core@lists.openembedded.org,
	bitbake-devel@lists.openembedded.org
Subject: Re: [bitbake-devel] [PATCH v3 13/17] bitbake: runqueue: Pass dependency ID to hash validate
Date: Wed, 05 Dec 2018 22:52:22 +0000	[thread overview]
Message-ID: <5d5b3aba3171a2fc2ef94157769ec0b0e46e49f9.camel@linuxfoundation.org> (raw)
In-Reply-To: <20181204034245.25461-14-JPEWhacker@gmail.com>

On Mon, 2018-12-03 at 21:42 -0600, Joshua Watt wrote:
> If the dependency ID is being used to track task dependencies, the
> hash
> validation function needs to know about it in order to properly
> validate
> the hash.
> 
> TODO: This currently isn't going to be backward compatible with older
> hashvalidate functions. Is that necessary, and if so are there any
> suggestions for a good approach?

That is necessary or bitbake and the metadata need to be in lock step.

@@ -1567,15 +1568,16 @@ class RunQueue:
>              sq_fn.append(fn)
>              sq_hashfn.append(self.rqdata.dataCaches[mc].hashfn[taskfn])
>              sq_hash.append(self.rqdata.runtaskentries[tid].hash)
> +            sq_depid.append(self.rqdata.runtaskentries[tid].depid)
>              sq_taskname.append(taskname)
>              sq_task.append(tid)
> -        locs = { "sq_fn" : sq_fn, "sq_task" : sq_taskname, "sq_hash" : sq_hash, "sq_hashfn" : sq_hashfn, "d" : self.cooker.data }
> +        locs = { "sq_fn" : sq_fn, "sq_task" : sq_taskname, "sq_hash" : sq_hash, "sq_hashfn" : sq_hashfn, "sq_depid" : sq_depid, "d" : self.cooker.data }
>          try:
> -            call = self.hashvalidate + "(sq_fn, sq_task, sq_hash, sq_hashfn, d, siginfo=True)"
> +            call = self.hashvalidate + "(sq_fn, sq_task, sq_hash, sq_hashfn, sq_depid, d, siginfo=True)"
>              valid = bb.utils.better_eval(call, locs)
>          # Handle version with no siginfo parameter
>          except TypeError:
> -            call = self.hashvalidate + "(sq_fn, sq_task, sq_hash, sq_hashfn, d)"
> +            call = self.hashvalidate + "(sq_fn, sq_task, sq_hash, sq_hashfn, sq_depid, d)"
>              valid = bb.utils.better_eval(call, locs)
>          for v in valid:
> 

The hint on how we've handled this before is in the code, the TypeError
was the fallback code for older metadata (which you just broke!). We
may need to do something similar here.

Cheers,

Richard



  reply	other threads:[~2018-12-05 22:52 UTC|newest]

Thread overview: 158+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-07-16 20:37 [RFC 0/9] Hash Equivalency Server Joshua Watt
2018-07-16 20:37 ` [RFC 1/9] bitbake-worker: Pass taskhash as runtask parameter Joshua Watt
2018-07-16 20:37 ` [RFC 2/9] siggen: Split out stampfile hash fetch Joshua Watt
2018-07-16 20:37 ` [RFC 3/9] siggen: Split out task depend ID Joshua Watt
2018-07-16 20:37 ` [RFC 4/9] runqueue: Track task dependency ID Joshua Watt
2018-07-16 20:37 ` [RFC 5/9] runqueue: Pass dependency ID to task Joshua Watt
2018-07-16 20:37 ` [RFC 6/9] runqueue: Pass dependency ID to hash validate Joshua Watt
2018-07-16 20:37 ` [RFC 7/9] classes/sstate: Handle depid in hash check Joshua Watt
2018-07-16 20:37 ` [RFC 8/9] hashserver: Add initial reference server Joshua Watt
2018-07-17 12:11   ` Richard Purdie
2018-07-17 12:11     ` [bitbake-devel] " Richard Purdie
2018-07-17 13:44     ` Joshua Watt
2018-07-17 13:44       ` [bitbake-devel] " Joshua Watt
2018-07-18 13:53     ` Joshua Watt
2018-07-18 13:53       ` [bitbake-devel] " Joshua Watt
2018-07-16 20:37 ` [RFC 9/9] sstate: Implement hash equivalence sstate Joshua Watt
2018-08-09 22:08 ` [RFC v2 00/16] Hash Equivalency Server Joshua Watt
2018-08-09 22:08   ` [RFC v2 01/16] bitbake: fork: Add os.fork() wrappers Joshua Watt
2018-08-09 22:08   ` [RFC v2 02/16] bitbake: persist_data: Fix leaking cursors causing deadlock Joshua Watt
2018-08-09 22:08   ` [RFC v2 03/16] bitbake: persist_data: Add key constraints Joshua Watt
2018-08-09 22:08   ` [RFC v2 04/16] bitbake: persist_data: Enable Write Ahead Log Joshua Watt
2018-08-09 22:08   ` [RFC v2 05/16] bitbake: persist_data: Disable enable_shared_cache Joshua Watt
2018-08-09 22:08   ` [RFC v2 06/16] bitbake: persist_data: Close databases across fork Joshua Watt
2018-08-09 22:08   ` [RFC v2 07/16] bitbake: tests/persist_data: Add tests Joshua Watt
2018-08-09 22:08   ` [RFC v2 08/16] bitbake: bitbake-worker: Pass taskhash as runtask parameter Joshua Watt
2018-08-09 22:08   ` [RFC v2 09/16] bitbake: siggen: Split out stampfile hash fetch Joshua Watt
2018-08-09 22:08   ` [RFC v2 10/16] bitbake: siggen: Split out task depend ID Joshua Watt
2018-08-09 22:08   ` [RFC v2 11/16] bitbake: runqueue: Track task dependency ID Joshua Watt
2018-08-09 22:08   ` [RFC v2 12/16] bitbake: runqueue: Pass dependency ID to task Joshua Watt
2018-08-09 22:08   ` [RFC v2 13/16] bitbake: runqueue: Pass dependency ID to hash validate Joshua Watt
2018-08-09 22:08   ` [RFC v2 14/16] classes/sstate: Handle depid in hash check Joshua Watt
2018-08-09 22:08   ` [RFC v2 15/16] bitbake: hashserv: Add hash equivalence reference server Joshua Watt
2018-08-09 22:08   ` [RFC v2 16/16] sstate: Implement hash equivalence sstate Joshua Watt
2018-12-04  3:42   ` [OE-core][PATCH v3 00/17] Hash Equivalency Server Joshua Watt
2018-12-04  3:42     ` [PATCH " Joshua Watt
2018-12-04  3:42     ` [OE-core][PATCH v3 01/17] bitbake: fork: Add os.fork() wrappers Joshua Watt
2018-12-04  3:42       ` [PATCH " Joshua Watt
2018-12-04  3:42     ` [OE-core][PATCH v3 02/17] bitbake: persist_data: Fix leaking cursors causing deadlock Joshua Watt
2018-12-04  3:42       ` [PATCH " Joshua Watt
2018-12-04  3:42     ` [OE-core][PATCH v3 03/17] bitbake: persist_data: Add key constraints Joshua Watt
2018-12-04  3:42       ` [PATCH " Joshua Watt
2018-12-04  3:42     ` [OE-core][PATCH v3 04/17] bitbake: persist_data: Enable Write Ahead Log Joshua Watt
2018-12-04  3:42       ` [PATCH " Joshua Watt
2018-12-04  3:42     ` [OE-core][PATCH v3 05/17] bitbake: persist_data: Disable enable_shared_cache Joshua Watt
2018-12-04  3:42       ` [PATCH " Joshua Watt
2018-12-04  3:42     ` [OE-core][PATCH v3 06/17] bitbake: persist_data: Close databases across fork Joshua Watt
2018-12-04  3:42       ` [PATCH " Joshua Watt
2018-12-04  3:42     ` [OE-core][PATCH v3 07/17] bitbake: tests/persist_data: Add tests Joshua Watt
2018-12-04  3:42       ` [PATCH " Joshua Watt
2018-12-04  3:42     ` [OE-core][PATCH v3 08/17] bitbake: bitbake-worker: Pass taskhash as runtask parameter Joshua Watt
2018-12-04  3:42       ` [PATCH " Joshua Watt
2018-12-04  3:42     ` [OE-core][PATCH v3 09/17] bitbake: siggen: Split out stampfile hash fetch Joshua Watt
2018-12-04  3:42       ` [PATCH " Joshua Watt
2018-12-04  3:42     ` [OE-core][PATCH v3 10/17] bitbake: siggen: Split out task depend ID Joshua Watt
2018-12-04  3:42       ` [PATCH " Joshua Watt
2018-12-05 22:50       ` [OE-core][PATCH " Richard Purdie
2018-12-05 22:50         ` [bitbake-devel] [PATCH " Richard Purdie
2018-12-06 14:58         ` [OE-core][PATCH " Joshua Watt
2018-12-06 14:58           ` [bitbake-devel] [PATCH " Joshua Watt
2018-12-04  3:42     ` [OE-core][PATCH v3 11/17] bitbake: runqueue: Track task dependency ID Joshua Watt
2018-12-04  3:42       ` [PATCH " Joshua Watt
2018-12-04  3:42     ` [OE-core][PATCH v3 12/17] bitbake: runqueue: Pass dependency ID to task Joshua Watt
2018-12-04  3:42       ` [PATCH " Joshua Watt
2018-12-04  3:42     ` [OE-core][PATCH v3 13/17] bitbake: runqueue: Pass dependency ID to hash validate Joshua Watt
2018-12-04  3:42       ` [PATCH " Joshua Watt
2018-12-05 22:52       ` Richard Purdie [this message]
2018-12-05 22:52         ` [bitbake-devel] " Richard Purdie
2018-12-04  3:42     ` [OE-core][PATCH v3 14/17] classes/sstate: Handle depid in hash check Joshua Watt
2018-12-04  3:42       ` [PATCH " Joshua Watt
2018-12-04  3:42     ` [OE-core][PATCH v3 15/17] bitbake: hashserv: Add hash equivalence reference server Joshua Watt
2018-12-04  3:42       ` [PATCH " Joshua Watt
2018-12-04  3:42     ` [OE-core][PATCH v3 16/17] sstate: Implement hash equivalence sstate Joshua Watt
2018-12-04  3:42       ` [PATCH " Joshua Watt
2018-12-04  3:42     ` [OE-core][PATCH v3 17/17] classes/image-buildinfo: Remove unused argument Joshua Watt
2018-12-04  3:42       ` [PATCH " Joshua Watt
2018-12-18 15:30     ` [OE-core][PATCH v4 00/10] Hash Equivalency Server Joshua Watt
2018-12-18 15:30       ` [PATCH " Joshua Watt
2018-12-18 15:30       ` [OE-core][PATCH v4 01/10] bitbake: fork: Add os.fork() wrappers Joshua Watt
2018-12-18 15:30         ` [PATCH " Joshua Watt
2018-12-18 15:30       ` [OE-core][PATCH v4 02/10] bitbake: persist_data: Close databases across fork Joshua Watt
2018-12-18 15:30         ` [PATCH " Joshua Watt
2018-12-18 15:30       ` [OE-core][PATCH v4 03/10] bitbake: tests/persist_data: Add tests Joshua Watt
2018-12-18 15:30         ` [PATCH " Joshua Watt
2018-12-18 15:30       ` [OE-core][PATCH v4 04/10] bitbake: siggen: Split out task unique hash Joshua Watt
2018-12-18 15:30         ` [PATCH " Joshua Watt
2018-12-18 15:30       ` [OE-core][PATCH v4 05/10] bitbake: runqueue: Track " Joshua Watt
2018-12-18 15:30         ` [PATCH " Joshua Watt
2018-12-18 15:30       ` [OE-core][PATCH v4 06/10] bitbake: runqueue: Pass unique hash to task Joshua Watt
2018-12-18 15:30         ` [PATCH " Joshua Watt
2018-12-18 15:30       ` [OE-core][PATCH v4 07/10] bitbake: runqueue: Pass unique hash to hash validate Joshua Watt
2018-12-18 15:30         ` [PATCH " Joshua Watt
2018-12-18 16:24         ` [OE-core] " Richard Purdie
2018-12-18 16:24           ` Richard Purdie
2018-12-18 16:31           ` [OE-core] " Joshua Watt
2018-12-18 16:31             ` Joshua Watt
2018-12-18 15:30       ` [OE-core][PATCH v4 08/10] classes/sstate: Handle unihash in hash check Joshua Watt
2018-12-18 15:30         ` [PATCH " Joshua Watt
2018-12-18 15:31       ` [OE-core][PATCH v4 09/10] bitbake: hashserv: Add hash equivalence reference server Joshua Watt
2018-12-18 15:31         ` [PATCH " Joshua Watt
2018-12-18 15:31       ` [OE-core][PATCH v4 10/10] sstate: Implement hash equivalence sstate Joshua Watt
2018-12-18 15:31         ` [PATCH " Joshua Watt
2018-12-19  3:10       ` [OE-core][PATCH v5 0/8] Hash Equivalency Server Joshua Watt
2018-12-19  3:10         ` [PATCH " Joshua Watt
2018-12-19  3:10         ` [OE-core][PATCH v5 1/8] bitbake: tests/persist_data: Add tests Joshua Watt
2018-12-19  3:10           ` [PATCH " Joshua Watt
2018-12-19  3:10         ` [OE-core][PATCH v5 2/8] bitbake: siggen: Split out task unique hash Joshua Watt
2018-12-19  3:10           ` [PATCH " Joshua Watt
2018-12-19  3:10         ` [OE-core][PATCH v5 3/8] bitbake: runqueue: Track " Joshua Watt
2018-12-19  3:10           ` [PATCH " Joshua Watt
2019-01-05  7:49           ` [OE-core] " Alejandro Hernandez
2019-01-05  7:49             ` Alejandro Hernandez
2019-01-06  3:09             ` [OE-core] " Joshua Watt
2019-01-06  3:09               ` Joshua Watt
2019-01-07  6:52               ` [OE-core] " Alejandro Hernandez
2019-01-07  6:52                 ` Alejandro Hernandez
2019-01-07 16:16               ` [OE-core] " akuster808
2019-01-07 16:16                 ` akuster808
2019-01-07 16:40                 ` [OE-core] " Joshua Watt
2019-01-07 16:40                   ` Joshua Watt
2018-12-19  3:10         ` [OE-core][PATCH v5 4/8] bitbake: runqueue: Pass unique hash to task Joshua Watt
2018-12-19  3:10           ` [PATCH " Joshua Watt
2018-12-19  3:10         ` [OE-core][PATCH v5 5/8] bitbake: runqueue: Pass unique hash to hash validate Joshua Watt
2018-12-19  3:10           ` [PATCH " Joshua Watt
2018-12-19  3:10         ` [OE-core][PATCH v5 6/8] classes/sstate: Handle unihash in hash check Joshua Watt
2018-12-19  3:10           ` [PATCH " Joshua Watt
2018-12-19  3:10         ` [OE-core][PATCH v5 7/8] bitbake: hashserv: Add hash equivalence reference server Joshua Watt
2018-12-19  3:10           ` [PATCH " Joshua Watt
2018-12-19  3:10         ` [OE-core][PATCH v5 8/8] sstate: Implement hash equivalence sstate Joshua Watt
2018-12-19  3:10           ` [PATCH " Joshua Watt
2018-12-19  3:33       ` ✗ patchtest: failure for Hash Equivalency Server (rev3) Patchwork
2019-01-04  2:42       ` [OE-core][PATCH v6 0/3] Hash Equivalency Server Joshua Watt
2019-01-04  2:42         ` [PATCH " Joshua Watt
2019-01-04  2:42         ` [OE-core][PATCH v6 1/3] classes/sstate: Handle unihash in hash check Joshua Watt
2019-01-04  2:42           ` [PATCH " Joshua Watt
2019-01-04  7:01           ` [OE-core][PATCH " Richard Purdie
2019-01-04  7:01             ` [bitbake-devel] [PATCH " Richard Purdie
2019-01-04  2:42         ` [OE-core][PATCH v6 2/3] bitbake: hashserv: Add hash equivalence reference server Joshua Watt
2019-01-04  2:42           ` [PATCH " Joshua Watt
2019-01-04  2:42         ` [OE-core][PATCH v6 3/3] sstate: Implement hash equivalence sstate Joshua Watt
2019-01-04  2:42           ` [PATCH " Joshua Watt
2019-01-04 16:20         ` [OE-core][PATCH v7 0/3] Hash Equivalency Server Joshua Watt
2019-01-04 16:20           ` [PATCH " Joshua Watt
2019-01-04 16:20           ` [OE-core][PATCH v7 1/3] classes/sstate: Handle unihash in hash check Joshua Watt
2019-01-04 16:20             ` [PATCH " Joshua Watt
2019-01-04 16:20           ` [OE-core][PATCH v7 2/3] bitbake: hashserv: Add hash equivalence reference server Joshua Watt
2019-01-04 16:20             ` [PATCH " Joshua Watt
2019-01-04 16:20           ` [OE-core][PATCH v7 3/3] sstate: Implement hash equivalence sstate Joshua Watt
2019-01-04 16:20             ` [PATCH " Joshua Watt
2019-01-08  6:29             ` [OE-core][PATCH " Jacob Kroon
2019-01-08  6:29               ` [bitbake-devel] [PATCH " Jacob Kroon
2019-01-09 17:09               ` [OE-core][PATCH " Joshua Watt
2019-01-09 17:09                 ` [bitbake-devel] [PATCH " Joshua Watt
2019-01-11 20:39                 ` [OE-core][PATCH " Peter Kjellerstedt
2019-01-11 20:39                   ` [bitbake-devel] [PATCH " Peter Kjellerstedt
2019-01-04 16:33         ` ✗ patchtest: failure for Hash Equivalency Server (rev5) Patchwork
2019-01-04  3:03       ` ✗ patchtest: failure for Hash Equivalency Server (rev4) Patchwork
2018-12-18 16:03     ` ✗ patchtest: failure for Hash Equivalency Server (rev2) Patchwork
2018-12-04  4:05   ` ✗ patchtest: failure for Hash Equivalency Server Patchwork

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=5d5b3aba3171a2fc2ef94157769ec0b0e46e49f9.camel@linuxfoundation.org \
    --to=richard.purdie@linuxfoundation.org \
    --cc=bitbake-devel@lists.openembedded.org \
    --cc=jpewhacker@gmail.com \
    --cc=openembedded-core@lists.openembedded.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.