All of lore.kernel.org
 help / color / mirror / Atom feed
From: Richard Purdie <richard.purdie@linuxfoundation.org>
To: bitbake-devel@lists.openembedded.org
Subject: [PATCH 00/26] runqueue changes
Date: Thu, 11 Jul 2019 01:05:46 +0100	[thread overview]
Message-ID: <8ace64fd460c657fee23dd39887825e418314c5e.camel@linuxfoundation.org> (raw)
In-Reply-To: <20190710235420.23825-1-richard.purdie@linuxfoundation.org>

This patchset reworks runqueue so that instead of having two distinct
phases, specifically "setscene", followed by "normal tasks" we have one
single phase.

In this single phase, both sets of tasks can run in parallel using the
same workers. Thankfully the code already used common classes and
functions so making this transition was relatively straight forward.

Initially all tasks are in the scenequeue, as it works through them,
they are marked as "buildable" which means normal tasks can execute.

The big change here are in the code which incrementally marks tasks as
available rather than being able to process tasks "en mass".

The most common failures from these changes are potentially:

a) tasks not migrating from the scenequeue
b) tasks not becomming buildable

I've made some tests on the autobuilder and believe I've tested many of
the common usecases. There is debug code to abort the builds with the
above errors when something unexpected happens.

I've also written an initial set of runqueue unittests. To do that I
had to create a dummy set of metadata so we had some dependency trees
to test with.

This patchset did remove an API we no longer need/use after RSS was
introduced and gets rid of what seems like a pointless form of event.

These changes set up the foundations so that we can work on two other
significant problems:

a) reuse of sstate artefacts between multiconfig build tasks
b) adapting the task queue 'on the fly' if the output of a task matches
   a previous output and hence we have sstate hash equivalency

Solving these would be great for OE!

I will follow up with more details on the future plans for the above
and this code soon.

Cheers,

Richard





  parent reply	other threads:[~2019-07-11  0:05 UTC|newest]

Thread overview: 31+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-07-10 23:53 [PATCH 01/26] runqueue: Tweak buildable variable handling in scheduler Richard Purdie
2019-07-10 23:53 ` [PATCH 02/26] runqueue: Drop unused BB_SETSCENE_VERIFY_FUNCTION2 Richard Purdie
2019-07-10 23:53 ` [PATCH 03/26] runqueue: Remove now uneeded code Richard Purdie
2019-07-10 23:53 ` [PATCH 04/26] runqueue: Move scenequeue data generation to a separate function Richard Purdie
2019-07-10 23:53 ` [PATCH 05/26] runqueue: Remove unused function parameter Richard Purdie
2019-07-10 23:54 ` [PATCH 06/26] runqueue: Factor out the process_setscene_whitelist checks Richard Purdie
2019-07-10 23:54 ` [PATCH 07/26] runqueue: Uniquely namespace the scenequeue functions Richard Purdie
2019-07-10 23:54 ` [PATCH 08/26] runqueue: Merge stats handling together for setscene/real tasks Richard Purdie
2019-07-10 23:54 ` [PATCH 09/26] runqueue: Merge scenequeue and real task queue code together Richard Purdie
2019-07-10 23:54 ` [PATCH 10/26] runqueue: Fix counter/task updating glitch Richard Purdie
2019-07-10 23:54 ` [PATCH 11/26] runqueue: Remove RunQueueExecuteScenequeue and RunQueueExecuteTasks Richard Purdie
2019-07-10 23:54 ` [PATCH 12/26] runqueue: Simplify _execute_runqueue logic Richard Purdie
2019-07-10 23:54 ` [PATCH 13/26] runqueue: Fold remains of the scenequeue setup into RunQueueExecute Richard Purdie
2019-07-10 23:54 ` [PATCH 14/26] event/runqueue: Drop StampUpdate event, its pointless/unused Richard Purdie
2019-07-10 23:54 ` [PATCH 15/26] runqueue: Add covered_tasks (or 'collated_deps') to scenequeue data Richard Purdie
2019-07-10 23:54 ` [PATCH 16/26] runqueue: Simplify scenequeue unskippable calculation Richard Purdie
2019-07-10 23:54 ` [PATCH 17/26] runqueue: Tweak comments and debug code Richard Purdie
2019-07-10 23:54 ` [PATCH 18/26] runqueue: Code simplification Richard Purdie
2019-07-10 23:54 ` [PATCH 19/26] runqueue: Remove pointless variable Richard Purdie
2019-07-10 23:54 ` [PATCH 20/26] runqueue: Further scheduler buildable tasks cleanup Richard Purdie
2019-07-10 23:54 ` [PATCH 21/26] runqueue: Clarify scenequeue_covered vs. tasks_covered Richard Purdie
2019-07-10 23:54 ` [PATCH 22/26] runqueue: Merge the queues and execute setscene and normal tasks in parallel Richard Purdie
2019-07-10 23:54 ` [PATCH 23/26] runqueue: Alter setscenewhitelist handling Richard Purdie
2019-07-10 23:54 ` [PATCH 24/26] runqueue: Complete the merge of scenequeue and normal task execution Richard Purdie
2019-07-10 23:54 ` [PATCH 25/26] siggen: Fix default handler Richard Purdie
2019-07-10 23:54 ` [PATCH 26/26] tests: Add initial scenario based test for runqueue Richard Purdie
2019-07-11  0:05 ` Richard Purdie [this message]
     [not found]   ` <63270c2a-f78a-65e1-16c0-8ce271d4dd8e@mikrodidakt.se>
2019-07-11 11:33     ` [PATCH 00/26] runqueue changes richard.purdie
2019-07-11 16:21   ` Richard Purdie
2019-07-12  0:57     ` Khem Raj
2019-07-12  8:13       ` richard.purdie

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=8ace64fd460c657fee23dd39887825e418314c5e.camel@linuxfoundation.org \
    --to=richard.purdie@linuxfoundation.org \
    --cc=bitbake-devel@lists.openembedded.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.