* [PATCH 0/2] Multi-threaded RPM support @ 2017-06-01 15:15 Alexander Kanavin 2017-06-01 15:15 ` [PATCH 1/2] package_rpm.bbclass: use multithreaded xz compression Alexander Kanavin ` (4 more replies) 0 siblings, 5 replies; 24+ messages in thread From: Alexander Kanavin @ 2017-06-01 15:15 UTC (permalink / raw) To: openembedded-core These two patches add multi-threaded features to RPM to speed up the do_package_write_rpm task. Specifically: 1) Multi-threaded xz compressor is used instead of default single-threaded gz. This has the most dramatic effect when a recipe produces a smaller number of large-sized packages. 2) Packages creation is run in multiple threads via thread pools. This is most beneficial when a recipe produces a large amount of small packages. Some not very scientific benchmarks for time and .rpm sizes (time is measured for do_package_write_rpm tasks only): webkitgtk before: 9m22s 1550M after: 1m50s 858M glibc-locale before: 3m2s 125M after: 40s 56M glibc before: 56s 54M after: 23s 38M perl before: 1m19s 63M after: 55s 42M python3 before: 40s 38M after: 28s 24M The following changes since commit bd063fa288b49b6e3ea77982d0ccc46610feb1ad: tune-mips32*.inc: use consistent comments across all three .inc files (2017-05-30 10:15:20 +0100) are available in the git repository at: git://git.yoctoproject.org/poky-contrib akanavin/parallel-rpm http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=akanavin/parallel-rpm Alexander Kanavin (2): package_rpm.bbclass: use multithreaded xz compression rpm: run binary package generation via thread pools meta/classes/package_rpm.bbclass | 2 + ...y-package-building-into-a-separate-functi.patch | 83 ++++++++ ...-binary-package-creation-via-thread-pools.patch | 137 +++++++++++++ ...c-make-operations-over-string-pools-threa.patch | 209 +++++++++++++++++++ ...c-remove-static-local-variables-from-buil.patch | 227 +++++++++++++++++++++ meta/recipes-devtools/rpm/rpm_git.bb | 4 + 6 files changed, 662 insertions(+) create mode 100644 meta/recipes-devtools/rpm/files/0001-Split-binary-package-building-into-a-separate-functi.patch create mode 100644 meta/recipes-devtools/rpm/files/0002-Run-binary-package-creation-via-thread-pools.patch create mode 100644 meta/recipes-devtools/rpm/files/0003-rpmstrpool.c-make-operations-over-string-pools-threa.patch create mode 100644 meta/recipes-devtools/rpm/files/0004-build-pack.c-remove-static-local-variables-from-buil.patch -- 2.11.0 ^ permalink raw reply [flat|nested] 24+ messages in thread
* [PATCH 1/2] package_rpm.bbclass: use multithreaded xz compression 2017-06-01 15:15 [PATCH 0/2] Multi-threaded RPM support Alexander Kanavin @ 2017-06-01 15:15 ` Alexander Kanavin 2017-06-01 17:11 ` Mark Hatle 2017-06-01 19:02 ` Andre McCurdy 2017-06-01 15:15 ` [PATCH 2/2] rpm: run binary package generation via thread pools Alexander Kanavin ` (3 subsequent siblings) 4 siblings, 2 replies; 24+ messages in thread From: Alexander Kanavin @ 2017-06-01 15:15 UTC (permalink / raw) To: openembedded-core RPM's default is single-threaded gz; the change greatly helps with both buildtimes (when there is a small number of large-sized packages) and disk space taken by resulting rpms. Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com> --- meta/classes/package_rpm.bbclass | 2 ++ 1 file changed, 2 insertions(+) diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass index a844c4d4360..dc241975c3b 100644 --- a/meta/classes/package_rpm.bbclass +++ b/meta/classes/package_rpm.bbclass @@ -644,6 +644,8 @@ python do_package_rpm () { cmd = cmd + " --define '_build_name_fmt %%{NAME}-%%{VERSION}-%%{RELEASE}.%%{ARCH}.rpm'" cmd = cmd + " --define '_use_internal_dependency_generator 0'" cmd = cmd + " --define '_binaries_in_noarch_packages_terminate_build 0'" + cmd = cmd + " --define '_binary_payload w6T.xzdio'" + cmd = cmd + " --define '_source_payload w6T.xzdio'" if perfiledeps: cmd = cmd + " --define '__find_requires " + outdepends + "'" cmd = cmd + " --define '__find_provides " + outprovides + "'" -- 2.11.0 ^ permalink raw reply related [flat|nested] 24+ messages in thread
* Re: [PATCH 1/2] package_rpm.bbclass: use multithreaded xz compression 2017-06-01 15:15 ` [PATCH 1/2] package_rpm.bbclass: use multithreaded xz compression Alexander Kanavin @ 2017-06-01 17:11 ` Mark Hatle 2017-06-01 17:44 ` Alexander Kanavin 2017-06-01 19:02 ` Andre McCurdy 1 sibling, 1 reply; 24+ messages in thread From: Mark Hatle @ 2017-06-01 17:11 UTC (permalink / raw) To: Alexander Kanavin, openembedded-core On 6/1/17 10:15 AM, Alexander Kanavin wrote: > RPM's default is single-threaded gz; the change greatly helps with > both buildtimes (when there is a small number of large-sized packages) > and disk space taken by resulting rpms. If you wnable XZ here, you need to make sure XZ compression is enabled within rpm itself. (I know how that worked in rpm5, not sure how rpm4 does it -- it may be enable that the binary be present to call out to.) Similarly to XZ, you can use pigz or other parallel compression tools to do similar with other schemes. Finally I have not investigated parallel XZ, but in some cases the resulting compressed file is different depending on compression artifacts due to parallelization. (The compressed file is different, not the uncompressed content.) This can be a concern to some who want fully reproducible builds (from source). So we need to make sure we have a way to disable this type of parallelization --if-- it causes a problem with different contents. (Note the fact RPM package metadata changes each time a new RPM is constructed, this is far -less- of a concern to me.. but something to be aware of.) --Mark > Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com> > --- > meta/classes/package_rpm.bbclass | 2 ++ > 1 file changed, 2 insertions(+) > > diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass > index a844c4d4360..dc241975c3b 100644 > --- a/meta/classes/package_rpm.bbclass > +++ b/meta/classes/package_rpm.bbclass > @@ -644,6 +644,8 @@ python do_package_rpm () { > cmd = cmd + " --define '_build_name_fmt %%{NAME}-%%{VERSION}-%%{RELEASE}.%%{ARCH}.rpm'" > cmd = cmd + " --define '_use_internal_dependency_generator 0'" > cmd = cmd + " --define '_binaries_in_noarch_packages_terminate_build 0'" > + cmd = cmd + " --define '_binary_payload w6T.xzdio'" > + cmd = cmd + " --define '_source_payload w6T.xzdio'" > if perfiledeps: > cmd = cmd + " --define '__find_requires " + outdepends + "'" > cmd = cmd + " --define '__find_provides " + outprovides + "'" > ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 1/2] package_rpm.bbclass: use multithreaded xz compression 2017-06-01 17:11 ` Mark Hatle @ 2017-06-01 17:44 ` Alexander Kanavin 0 siblings, 0 replies; 24+ messages in thread From: Alexander Kanavin @ 2017-06-01 17:44 UTC (permalink / raw) To: Mark Hatle, openembedded-core On 06/01/2017 08:11 PM, Mark Hatle wrote: > On 6/1/17 10:15 AM, Alexander Kanavin wrote: >> RPM's default is single-threaded gz; the change greatly helps with >> both buildtimes (when there is a small number of large-sized packages) >> and disk space taken by resulting rpms. > > If you wnable XZ here, you need to make sure XZ compression is enabled within > rpm itself. (I know how that worked in rpm5, not sure how rpm4 does it -- it > may be enable that the binary be present to call out to.) rpm4 does not call out to any binary, it is linking against liblzma from xz package, if it can find it during configure. Xz was already in list of dependencies, so nothing needed to be done. Same for other compression schemes. > Finally I have not investigated parallel XZ, but in some cases the resulting > compressed file is different depending on compression artifacts due to > parallelization. (The compressed file is different, not the uncompressed content.) > > This can be a concern to some who want fully reproducible builds (from source). > So we need to make sure we have a way to disable this type of parallelization > --if-- it causes a problem with different contents. Let's first see if it is an actual concern for anyone. If the uncompressed contents is exactly same, I have difficulty seeing what is the problem specifically. Alex ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 1/2] package_rpm.bbclass: use multithreaded xz compression 2017-06-01 15:15 ` [PATCH 1/2] package_rpm.bbclass: use multithreaded xz compression Alexander Kanavin 2017-06-01 17:11 ` Mark Hatle @ 2017-06-01 19:02 ` Andre McCurdy 2017-06-02 11:40 ` Alexander Kanavin 1 sibling, 1 reply; 24+ messages in thread From: Andre McCurdy @ 2017-06-01 19:02 UTC (permalink / raw) To: Alexander Kanavin; +Cc: OE Core mailing list On Thu, Jun 1, 2017 at 8:15 AM, Alexander Kanavin <alexander.kanavin@linux.intel.com> wrote: > RPM's default is single-threaded gz; the change greatly helps with > both buildtimes (when there is a small number of large-sized packages) > and disk space taken by resulting rpms. Do you have any test results for that? Obviously build times will be longer when building with a single CPU since xz compression needs more CPU than gzip, but I'm curious how many CPUs are required before the multiple threads win over and build times actually get shorter. 4? 8? One other concern with parallel xz compression is memory usage. Single threaded xz needs more memory than gzip and running multiple xz compression threads together will push the memory usage up again. Maybe not an issue if the peak is still acceptable, but something to be aware of (I've given up trying to build in 4GB, but hopefully 8GB build machines still have some life left in them...). > Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com> > --- > meta/classes/package_rpm.bbclass | 2 ++ > 1 file changed, 2 insertions(+) > > diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass > index a844c4d4360..dc241975c3b 100644 > --- a/meta/classes/package_rpm.bbclass > +++ b/meta/classes/package_rpm.bbclass > @@ -644,6 +644,8 @@ python do_package_rpm () { > cmd = cmd + " --define '_build_name_fmt %%{NAME}-%%{VERSION}-%%{RELEASE}.%%{ARCH}.rpm'" > cmd = cmd + " --define '_use_internal_dependency_generator 0'" > cmd = cmd + " --define '_binaries_in_noarch_packages_terminate_build 0'" > + cmd = cmd + " --define '_binary_payload w6T.xzdio'" > + cmd = cmd + " --define '_source_payload w6T.xzdio'" > if perfiledeps: > cmd = cmd + " --define '__find_requires " + outdepends + "'" > cmd = cmd + " --define '__find_provides " + outprovides + "'" > -- > 2.11.0 > > -- > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/openembedded-core ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 1/2] package_rpm.bbclass: use multithreaded xz compression 2017-06-01 19:02 ` Andre McCurdy @ 2017-06-02 11:40 ` Alexander Kanavin 2017-06-02 18:33 ` Andre McCurdy 0 siblings, 1 reply; 24+ messages in thread From: Alexander Kanavin @ 2017-06-02 11:40 UTC (permalink / raw) To: Andre McCurdy; +Cc: OE Core mailing list On 06/01/2017 10:02 PM, Andre McCurdy wrote: >> RPM's default is single-threaded gz; the change greatly helps with >> both buildtimes (when there is a small number of large-sized packages) >> and disk space taken by resulting rpms. > > Do you have any test results for that? Obviously build times will be > longer when building with a single CPU since xz compression needs more > CPU than gzip, but I'm curious how many CPUs are required before the > multiple threads win over and build times actually get shorter. 4? 8? Yes; it's in the cover letter. > One other concern with parallel xz compression is memory usage. Single > threaded xz needs more memory than gzip and running multiple xz > compression threads together will push the memory usage up again. > Maybe not an issue if the peak is still acceptable, but something to > be aware of (I've given up trying to build in 4GB, but hopefully 8GB > build machines still have some life left in them...). I haven't seen anything resembling low-memory or OOM in my testing, even when running 44 rpm instances at the same time, each with their own several threads, each thread handling a package, and compression for the package again being done in a multi-threaded way. But then I have 22 cores and 32G RAM here. So do try the patches please, and report the findings, YMMV. Alex ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 1/2] package_rpm.bbclass: use multithreaded xz compression 2017-06-02 11:40 ` Alexander Kanavin @ 2017-06-02 18:33 ` Andre McCurdy 0 siblings, 0 replies; 24+ messages in thread From: Andre McCurdy @ 2017-06-02 18:33 UTC (permalink / raw) To: Alexander Kanavin; +Cc: OE Core mailing list On Fri, Jun 2, 2017 at 4:40 AM, Alexander Kanavin <alexander.kanavin@linux.intel.com> wrote: > On 06/01/2017 10:02 PM, Andre McCurdy wrote: >>> >>> RPM's default is single-threaded gz; the change greatly helps with >>> both buildtimes (when there is a small number of large-sized packages) >>> and disk space taken by resulting rpms. >> >> Do you have any test results for that? Obviously build times will be >> longer when building with a single CPU since xz compression needs more >> CPU than gzip, but I'm curious how many CPUs are required before the >> multiple threads win over and build times actually get shorter. 4? 8? > > Yes; it's in the cover letter. Yes, I only saw that after sending my reply. I shouldn't read emails in reverse order... sorry about that. The cover letter benchmarks didn't mention the 22 CPU cores though, so it's good to have that clarified. Seeing some results from a 4 or 8 core machine would be interesting too. >> One other concern with parallel xz compression is memory usage. Single >> threaded xz needs more memory than gzip and running multiple xz >> compression threads together will push the memory usage up again. >> Maybe not an issue if the peak is still acceptable, but something to >> be aware of (I've given up trying to build in 4GB, but hopefully 8GB >> build machines still have some life left in them...).> > > I haven't seen anything resembling low-memory or OOM in my testing, even > when running 44 rpm instances at the same time, each with their own several > threads, each thread handling a package, and compression for the package > again being done in a multi-threaded way. But then I have 22 cores and 32G > RAM here. > > So do try the patches please, and report the findings, YMMV. > > Alex > ^ permalink raw reply [flat|nested] 24+ messages in thread
* [PATCH 2/2] rpm: run binary package generation via thread pools 2017-06-01 15:15 [PATCH 0/2] Multi-threaded RPM support Alexander Kanavin 2017-06-01 15:15 ` [PATCH 1/2] package_rpm.bbclass: use multithreaded xz compression Alexander Kanavin @ 2017-06-01 15:15 ` Alexander Kanavin 2017-06-01 16:02 ` Alexander Kanavin 2017-06-01 16:52 ` [PATCH 0/2] Multi-threaded RPM support Leonardo Sandoval ` (2 subsequent siblings) 4 siblings, 1 reply; 24+ messages in thread From: Alexander Kanavin @ 2017-06-01 15:15 UTC (permalink / raw) To: openembedded-core This greatly reduces build times when there is a large amount of small rpm packages to produce. The patches are rather invasive, and so will be submitted upstream. Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com> --- ...y-package-building-into-a-separate-functi.patch | 83 ++++++++ ...-binary-package-creation-via-thread-pools.patch | 137 +++++++++++++ ...c-make-operations-over-string-pools-threa.patch | 209 +++++++++++++++++++ ...c-remove-static-local-variables-from-buil.patch | 227 +++++++++++++++++++++ meta/recipes-devtools/rpm/rpm_git.bb | 4 + 5 files changed, 660 insertions(+) create mode 100644 meta/recipes-devtools/rpm/files/0001-Split-binary-package-building-into-a-separate-functi.patch create mode 100644 meta/recipes-devtools/rpm/files/0002-Run-binary-package-creation-via-thread-pools.patch create mode 100644 meta/recipes-devtools/rpm/files/0003-rpmstrpool.c-make-operations-over-string-pools-threa.patch create mode 100644 meta/recipes-devtools/rpm/files/0004-build-pack.c-remove-static-local-variables-from-buil.patch diff --git a/meta/recipes-devtools/rpm/files/0001-Split-binary-package-building-into-a-separate-functi.patch b/meta/recipes-devtools/rpm/files/0001-Split-binary-package-building-into-a-separate-functi.patch new file mode 100644 index 00000000000..b21c5119041 --- /dev/null +++ b/meta/recipes-devtools/rpm/files/0001-Split-binary-package-building-into-a-separate-functi.patch @@ -0,0 +1,83 @@ +From b841b699e519438a66b661247c94efff63d0700e Mon Sep 17 00:00:00 2001 +From: Alexander Kanavin <alex.kanavin@gmail.com> +Date: Thu, 25 May 2017 18:15:27 +0300 +Subject: [PATCH 01/14] Split binary package building into a separate function + +So that it can be run as a thread pool task. + +Upstream-Status: Pending +Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> +--- + build/pack.c | 33 +++++++++++++++++++++------------ + 1 file changed, 21 insertions(+), 12 deletions(-) + +diff --git a/build/pack.c b/build/pack.c +index 497300b96..891e6bdc3 100644 +--- a/build/pack.c ++++ b/build/pack.c +@@ -546,18 +546,13 @@ static rpmRC checkPackages(char *pkgcheck) + return RPMRC_OK; + } + +-rpmRC packageBinaries(rpmSpec spec, const char *cookie, int cheating) ++static rpmRC packageBinary(rpmSpec spec, Package pkg, const char *cookie, int cheating, char** filename) + { +- rpmRC rc; +- const char *errorString; +- Package pkg; +- char *pkglist = NULL; +- +- for (pkg = spec->packages; pkg != NULL; pkg = pkg->next) { +- char *fn; ++ const char *errorString; ++ rpmRC rc = RPMRC_OK; + + if (pkg->fileList == NULL) +- continue; ++ return rc; + + if ((rc = processScriptFiles(spec, pkg))) + return rc; +@@ -591,7 +586,7 @@ rpmRC packageBinaries(rpmSpec spec, const char *cookie, int cheating) + headerGetString(pkg->header, RPMTAG_NAME), errorString); + return RPMRC_FAIL; + } +- fn = rpmGetPath("%{_rpmdir}/", binRpm, NULL); ++ *filename = rpmGetPath("%{_rpmdir}/", binRpm, NULL); + if ((binDir = strchr(binRpm, '/')) != NULL) { + struct stat st; + char *dn; +@@ -613,14 +608,28 @@ rpmRC packageBinaries(rpmSpec spec, const char *cookie, int cheating) + free(binRpm); + } + +- rc = writeRPM(pkg, NULL, fn, NULL); ++ rc = writeRPM(pkg, NULL, *filename, NULL); + if (rc == RPMRC_OK) { + /* Do check each written package if enabled */ +- char *pkgcheck = rpmExpand("%{?_build_pkgcheck} ", fn, NULL); ++ char *pkgcheck = rpmExpand("%{?_build_pkgcheck} ", *filename, NULL); + if (pkgcheck[0] != ' ') { + rc = checkPackages(pkgcheck); + } + free(pkgcheck); ++ } ++ return rc; ++} ++ ++rpmRC packageBinaries(rpmSpec spec, const char *cookie, int cheating) ++{ ++ rpmRC rc; ++ Package pkg; ++ char *pkglist = NULL; ++ ++ for (pkg = spec->packages; pkg != NULL; pkg = pkg->next) { ++ char *fn = NULL; ++ rc = packageBinary(spec, pkg, cookie, cheating, &fn); ++ if (rc == RPMRC_OK) { + rstrcat(&pkglist, fn); + rstrcat(&pkglist, " "); + } +-- +2.11.0 + diff --git a/meta/recipes-devtools/rpm/files/0002-Run-binary-package-creation-via-thread-pools.patch b/meta/recipes-devtools/rpm/files/0002-Run-binary-package-creation-via-thread-pools.patch new file mode 100644 index 00000000000..2987a979698 --- /dev/null +++ b/meta/recipes-devtools/rpm/files/0002-Run-binary-package-creation-via-thread-pools.patch @@ -0,0 +1,137 @@ +From 45d92254cea0bceff38c42cbad89b301a0c07cea Mon Sep 17 00:00:00 2001 +From: Alexander Kanavin <alex.kanavin@gmail.com> +Date: Thu, 25 May 2017 19:30:20 +0300 +Subject: [PATCH 02/14] Run binary package creation via thread pools. + +Upstream-Status: Pending +Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> +--- + build/pack.c | 100 ++++++++++++++++++++++++++++++++++++++++++++++++++--------- + 1 file changed, 86 insertions(+), 14 deletions(-) + +diff --git a/build/pack.c b/build/pack.c +index 891e6bdc3..de2566d47 100644 +--- a/build/pack.c ++++ b/build/pack.c +@@ -14,6 +14,8 @@ + #include <rpm/rpmfileutil.h> + #include <rpm/rpmlog.h> + ++#include <prtpool.h> /* NSPR thread pools */ ++ + #include "rpmio/rpmio_internal.h" /* fdInitDigest, fdFiniDigest */ + #include "lib/fsm.h" + #include "lib/signature.h" +@@ -620,25 +622,95 @@ static rpmRC packageBinary(rpmSpec spec, Package pkg, const char *cookie, int ch + return rc; + } + +-rpmRC packageBinaries(rpmSpec spec, const char *cookie, int cheating) ++struct binaryPackageTaskData + { +- rpmRC rc; ++ rpmSpec spec; + Package pkg; ++ const char *cookie; ++ int cheating; ++ char *filename; ++ PRJob *job; ++ rpmRC result; ++ PRStatus status; ++ struct binaryPackageTaskData *next; ++}; ++ ++static void runBinaryPackageTask(void* taskdata) ++{ ++ struct binaryPackageTaskData *task = taskdata; ++ task->result = packageBinary(task->spec, task->pkg, task->cookie, task->cheating, &(task->filename)); ++} ++ ++static struct binaryPackageTaskData* runBinaryPackageTasks(rpmSpec spec, const char *cookie, int cheating) ++{ ++ struct binaryPackageTaskData *tasks = NULL; ++ struct binaryPackageTaskData *task = NULL; ++ struct binaryPackageTaskData *prev = NULL; ++ PRThreadPool *pool = PR_CreateThreadPool(4, sysconf(_SC_NPROCESSORS_ONLN), 0); ++ ++ for (Package pkg = spec->packages; pkg != NULL; pkg = pkg->next) { ++ task = rcalloc(1, sizeof(*task)); ++ task->spec = spec; ++ task->pkg = pkg; ++ task->cookie = cookie; ++ task->cheating = cheating; ++ if (pkg == spec->packages) { ++ // the first package needs to be processed ahead of others, as they copy ++ // changelog data from it, and so otherwise data races would happen ++ task->job = PR_QueueJob(pool, runBinaryPackageTask, task, PR_TRUE); ++ task->status = PR_JoinJob(task->job); ++ rpmlog(RPMLOG_NOTICE, _("Joined binary package job, status %d, result %d, filename %s\n"), task->status, task->result, task->filename); ++ tasks = task; ++ } ++ if (prev != NULL) { ++ prev->next = task; ++ } ++ prev = task; ++ } ++ ++ for (task = tasks; task != NULL; task = task->next) { ++ if (task != tasks) { ++ task->job = PR_QueueJob(pool, runBinaryPackageTask, task, PR_TRUE); ++ } ++ } ++ ++ for (task = tasks; task != NULL; task = task->next) { ++ if (task != tasks) { ++ task->status = PR_JoinJob(task->job); ++ rpmlog(RPMLOG_NOTICE, _("Joined binary package job, status %d, result %d, filename %s\n"), task->status, task->result, task->filename); ++ } ++ } ++ PR_ShutdownThreadPool(pool); ++ return tasks; ++} ++ ++static void freeBinaryPackageTasks(struct binaryPackageTaskData* tasks) ++{ ++ while (tasks != NULL) { ++ struct binaryPackageTaskData* next = tasks->next; ++ rfree(tasks->filename); ++ rfree(tasks); ++ tasks = next; ++ } ++} ++ ++rpmRC packageBinaries(rpmSpec spec, const char *cookie, int cheating) ++{ + char *pkglist = NULL; + +- for (pkg = spec->packages; pkg != NULL; pkg = pkg->next) { +- char *fn = NULL; +- rc = packageBinary(spec, pkg, cookie, cheating, &fn); +- if (rc == RPMRC_OK) { +- rstrcat(&pkglist, fn); +- rstrcat(&pkglist, " "); +- } +- free(fn); +- if (rc != RPMRC_OK) { +- pkglist = _free(pkglist); +- return rc; +- } ++ struct binaryPackageTaskData *tasks = runBinaryPackageTasks(spec, cookie, cheating); ++ ++ for (struct binaryPackageTaskData *task = tasks; task != NULL; task = task->next) { ++ if (task->status == PR_SUCCESS && task->result == RPMRC_OK) { ++ rstrcat(&pkglist, task->filename); ++ rstrcat(&pkglist, " "); ++ } else { ++ _free(pkglist); ++ freeBinaryPackageTasks(tasks); ++ return RPMRC_FAIL; ++ } + } ++ freeBinaryPackageTasks(tasks); + + /* Now check the package set if enabled */ + if (pkglist != NULL) { +-- +2.11.0 + diff --git a/meta/recipes-devtools/rpm/files/0003-rpmstrpool.c-make-operations-over-string-pools-threa.patch b/meta/recipes-devtools/rpm/files/0003-rpmstrpool.c-make-operations-over-string-pools-threa.patch new file mode 100644 index 00000000000..acda55d4ea7 --- /dev/null +++ b/meta/recipes-devtools/rpm/files/0003-rpmstrpool.c-make-operations-over-string-pools-threa.patch @@ -0,0 +1,209 @@ +From ed152aefce31aceb09bb606ca398bc396461fb49 Mon Sep 17 00:00:00 2001 +From: Alexander Kanavin <alex.kanavin@gmail.com> +Date: Tue, 30 May 2017 13:58:30 +0300 +Subject: [PATCH 03/14] rpmstrpool.c: make operations over string pools + thread-safe + +Otherwise multithreaded rpm building explodes in various ways due +to data races. + +Upstream-Status: Pending +Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> +--- + rpmio/rpmstrpool.c | 51 +++++++++++++++++++++++++++++++++++++++++---------- + 1 file changed, 41 insertions(+), 10 deletions(-) + +diff --git a/rpmio/rpmstrpool.c b/rpmio/rpmstrpool.c +index 30a57eb10..d85f5c3a4 100644 +--- a/rpmio/rpmstrpool.c ++++ b/rpmio/rpmstrpool.c +@@ -3,6 +3,7 @@ + #include <stdlib.h> + #include <rpm/rpmstring.h> + #include <rpm/rpmstrpool.h> ++#include <prlock.h> + #include "debug.h" + + #define STRDATA_CHUNKS 1024 +@@ -39,7 +40,8 @@ struct rpmstrPool_s { + + poolHash hash; /* string -> sid hash table */ + int frozen; /* are new id additions allowed? */ +- int nrefs; /* refcount */ ++ _Atomic int nrefs; /* refcount */ ++ PRLock* lock; + }; + + /* calculate hash and string length on at once */ +@@ -113,6 +115,8 @@ static poolHash poolHashCreate(int numBuckets) + return ht; + } + ++static const char * rpmstrPoolStrNoLock(rpmstrPool pool, rpmsid sid); ++ + static void poolHashResize(rpmstrPool pool, int numBuckets) + { + poolHash ht = pool->hash; +@@ -120,7 +124,7 @@ static void poolHashResize(rpmstrPool pool, int numBuckets) + + for (int i=0; i<ht->numBuckets; i++) { + if (!ht->buckets[i].keyid) continue; +- unsigned int keyHash = rstrhash(rpmstrPoolStr(pool, ht->buckets[i].keyid)); ++ unsigned int keyHash = rstrhash(rpmstrPoolStrNoLock(pool, ht->buckets[i].keyid)); + for (unsigned int j=0;;j++) { + unsigned int hash = hashbucket(keyHash, j) % numBuckets; + if (!buckets[hash].keyid) { +@@ -149,7 +153,7 @@ static void poolHashAddHEntry(rpmstrPool pool, const char * key, unsigned int ke + ht->buckets[hash].keyid = keyid; + ht->keyCount++; + break; +- } else if (!strcmp(rpmstrPoolStr(pool, ht->buckets[hash].keyid), key)) { ++ } else if (!strcmp(rpmstrPoolStrNoLock(pool, ht->buckets[hash].keyid), key)) { + return; + } + } +@@ -191,7 +195,7 @@ static void poolHashPrintStats(rpmstrPool pool) + int maxcollisions = 0; + + for (i=0; i<ht->numBuckets; i++) { +- unsigned int keyHash = rstrhash(rpmstrPoolStr(pool, ht->buckets[i].keyid)); ++ unsigned int keyHash = rstrhash(rpmstrPoolStrNoLock(pool, ht->buckets[i].keyid)); + for (unsigned int j=0;;j++) { + unsigned int hash = hashbucket(keyHash, i) % ht->numBuckets; + if (hash==i) { +@@ -221,7 +225,7 @@ static void rpmstrPoolRehash(rpmstrPool pool) + + pool->hash = poolHashCreate(sizehint); + for (int i = 1; i <= pool->offs_size; i++) +- poolHashAddEntry(pool, rpmstrPoolStr(pool, i), i); ++ poolHashAddEntry(pool, rpmstrPoolStrNoLock(pool, i), i); + } + + rpmstrPool rpmstrPoolCreate(void) +@@ -240,6 +244,7 @@ rpmstrPool rpmstrPoolCreate(void) + + rpmstrPoolRehash(pool); + pool->nrefs = 1; ++ pool->lock = PR_NewLock(); + return pool; + } + +@@ -257,6 +262,7 @@ rpmstrPool rpmstrPoolFree(rpmstrPool pool) + pool->chunks[i] = _free(pool->chunks[i]); + } + free(pool->chunks); ++ PR_DestroyLock(pool->lock); + free(pool); + } + } +@@ -272,6 +278,8 @@ rpmstrPool rpmstrPoolLink(rpmstrPool pool) + + void rpmstrPoolFreeze(rpmstrPool pool, int keephash) + { ++ if (pool) ++ PR_Lock(pool->lock); + if (pool && !pool->frozen) { + if (!keephash) { + pool->hash = poolHashFree(pool->hash); +@@ -281,15 +289,19 @@ void rpmstrPoolFreeze(rpmstrPool pool, int keephash) + pool->offs_alloced * sizeof(*pool->offs)); + pool->frozen = 1; + } ++ if (pool) ++ PR_Unlock(pool->lock); + } + + void rpmstrPoolUnfreeze(rpmstrPool pool) + { + if (pool) { ++ PR_Lock(pool->lock); + if (pool->hash == NULL) { + rpmstrPoolRehash(pool); + } + pool->frozen = 0; ++ PR_Unlock(pool->lock); + } + } + +@@ -350,7 +362,7 @@ static rpmsid rpmstrPoolGet(rpmstrPool pool, const char * key, size_t keylen, + return 0; + } + +- s = rpmstrPoolStr(pool, ht->buckets[hash].keyid); ++ s = rpmstrPoolStrNoLock(pool, ht->buckets[hash].keyid); + /* pool string could be longer than keylen, require exact matche */ + if (strncmp(s, key, keylen) == 0 && s[keylen] == '\0') + return ht->buckets[hash].keyid; +@@ -373,27 +385,29 @@ static inline rpmsid strn2id(rpmstrPool pool, const char *s, size_t slen, + rpmsid rpmstrPoolIdn(rpmstrPool pool, const char *s, size_t slen, int create) + { + rpmsid sid = 0; +- ++ PR_Lock(pool->lock); + if (s != NULL) { + unsigned int hash = rstrnhash(s, slen); + sid = strn2id(pool, s, slen, hash, create); + } ++ PR_Unlock(pool->lock); + return sid; + } + + rpmsid rpmstrPoolId(rpmstrPool pool, const char *s, int create) + { + rpmsid sid = 0; +- ++ PR_Lock(pool->lock); + if (s != NULL) { + size_t slen; + unsigned int hash = rstrlenhash(s, &slen); + sid = strn2id(pool, s, slen, hash, create); + } ++ PR_Unlock(pool->lock); + return sid; + } + +-const char * rpmstrPoolStr(rpmstrPool pool, rpmsid sid) ++static const char * rpmstrPoolStrNoLock(rpmstrPool pool, rpmsid sid) + { + const char *s = NULL; + if (pool && sid > 0 && sid <= pool->offs_size) +@@ -401,12 +415,23 @@ const char * rpmstrPoolStr(rpmstrPool pool, rpmsid sid) + return s; + } + ++const char * rpmstrPoolStr(rpmstrPool pool, rpmsid sid) ++{ ++ const char *s = NULL; ++ PR_Lock(pool->lock); ++ s = rpmstrPoolStrNoLock(pool, sid); ++ PR_Unlock(pool->lock); ++ return s; ++} ++ + size_t rpmstrPoolStrlen(rpmstrPool pool, rpmsid sid) + { + size_t slen = 0; ++ PR_Lock(pool->lock); + if (pool && sid > 0 && sid <= pool->offs_size) { + slen = strlen(pool->offs[sid]); + } ++ PR_Unlock(pool->lock); + return slen; + } + +@@ -421,5 +446,11 @@ int rpmstrPoolStreq(rpmstrPool poolA, rpmsid sidA, + + rpmsid rpmstrPoolNumStr(rpmstrPool pool) + { +- return (pool != NULL) ? pool->offs_size : 0; ++ rpmsid id = 0; ++ if (pool) { ++ PR_Lock(pool->lock); ++ id = pool->offs_size; ++ PR_Unlock(pool->lock); ++ } ++ return id; + } +-- +2.11.0 + diff --git a/meta/recipes-devtools/rpm/files/0004-build-pack.c-remove-static-local-variables-from-buil.patch b/meta/recipes-devtools/rpm/files/0004-build-pack.c-remove-static-local-variables-from-buil.patch new file mode 100644 index 00000000000..35319b82195 --- /dev/null +++ b/meta/recipes-devtools/rpm/files/0004-build-pack.c-remove-static-local-variables-from-buil.patch @@ -0,0 +1,227 @@ +From 394ab23f7962946bba5351a202869cba71f412eb Mon Sep 17 00:00:00 2001 +From: Alexander Kanavin <alex.kanavin@gmail.com> +Date: Thu, 1 Jun 2017 16:27:57 +0300 +Subject: [PATCH 04/14] build/pack.c: remove static local variables from + buildHost() and getBuildTime() + +Their use is causing difficult to diagnoze data races when building multiple +packages in parallel, and is a bad idea in general, as it also makes it more +difficult to reason about code. + +Upstream-Status: Pending +Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> +--- + build/pack.c | 94 +++++++++++++++++++++++++++++++----------------------------- + 1 file changed, 49 insertions(+), 45 deletions(-) + +diff --git a/build/pack.c b/build/pack.c +index de2566d47..b6a1f7c62 100644 +--- a/build/pack.c ++++ b/build/pack.c +@@ -153,54 +153,47 @@ exit: + return rc; + } + +-static rpm_time_t * getBuildTime(void) ++static rpm_time_t getBuildTime(void) + { +- static rpm_time_t buildTime[1]; ++ rpm_time_t buildTime = 0; + char *srcdate; + time_t epoch; + char *endptr; + +- if (buildTime[0] == 0) { +- srcdate = getenv("SOURCE_DATE_EPOCH"); +- if (srcdate) { +- errno = 0; +- epoch = strtol(srcdate, &endptr, 10); +- if (srcdate == endptr || *endptr || errno != 0) +- rpmlog(RPMLOG_ERR, _("unable to parse SOURCE_DATE_EPOCH\n")); +- else +- buildTime[0] = (int32_t) epoch; +- } else +- buildTime[0] = (int32_t) time(NULL); +- } ++ srcdate = getenv("SOURCE_DATE_EPOCH"); ++ if (srcdate) { ++ errno = 0; ++ epoch = strtol(srcdate, &endptr, 10); ++ if (srcdate == endptr || *endptr || errno != 0) ++ rpmlog(RPMLOG_ERR, _("unable to parse SOURCE_DATE_EPOCH\n")); ++ else ++ buildTime = (int32_t) epoch; ++ } else ++ buildTime = (int32_t) time(NULL); + + return buildTime; + } + +-static const char * buildHost(void) ++static char * buildHost(void) + { +- static char hostname[1024]; +- static int oneshot = 0; ++ char* hostname; + struct hostent *hbn; + char *bhMacro; + +- if (! oneshot) { +- bhMacro = rpmExpand("%{?_buildhost}", NULL); +- if (strcmp(bhMacro, "") != 0 && strlen(bhMacro) < 1024) { +- strcpy(hostname, bhMacro); +- } else { +- if (strcmp(bhMacro, "") != 0) +- rpmlog(RPMLOG_WARNING, _("The _buildhost macro is too long\n")); +- (void) gethostname(hostname, sizeof(hostname)); +- hbn = gethostbyname(hostname); +- if (hbn) +- strcpy(hostname, hbn->h_name); +- else +- rpmlog(RPMLOG_WARNING, +- _("Could not canonicalize hostname: %s\n"), hostname); +- } +- free(bhMacro); +- oneshot = 1; +- } ++ bhMacro = rpmExpand("%{?_buildhost}", NULL); ++ if (strcmp(bhMacro, "") != 0) { ++ rasprintf(&hostname, "%s", bhMacro); ++ } else { ++ hostname = rcalloc(1024, sizeof(*hostname)); ++ (void) gethostname(hostname, 1024); ++ hbn = gethostbyname(hostname); ++ if (hbn) ++ strcpy(hostname, hbn->h_name); ++ else ++ rpmlog(RPMLOG_WARNING, ++ _("Could not canonicalize hostname: %s\n"), hostname); ++ } ++ free(bhMacro); + return(hostname); + } + +@@ -310,7 +303,8 @@ static int haveRichDep(Package pkg) + } + + static rpmRC writeRPM(Package pkg, unsigned char ** pkgidp, +- const char *fileName, char **cookie) ++ const char *fileName, char **cookie, ++ rpm_time_t buildTime, const char* buildHost) + { + FD_t fd = NULL; + char * rpmio_flags = NULL; +@@ -399,7 +393,7 @@ static rpmRC writeRPM(Package pkg, unsigned char ** pkgidp, + + /* Create and add the cookie */ + if (cookie) { +- rasprintf(cookie, "%s %d", buildHost(), (int) (*getBuildTime())); ++ rasprintf(cookie, "%s %d", buildHost, buildTime); + headerPutString(pkg->header, RPMTAG_COOKIE, *cookie); + } + +@@ -548,7 +542,7 @@ static rpmRC checkPackages(char *pkgcheck) + return RPMRC_OK; + } + +-static rpmRC packageBinary(rpmSpec spec, Package pkg, const char *cookie, int cheating, char** filename) ++static rpmRC packageBinary(rpmSpec spec, Package pkg, const char *cookie, int cheating, char** filename, rpm_time_t buildTime, const char* buildHost) + { + const char *errorString; + rpmRC rc = RPMRC_OK; +@@ -567,8 +561,8 @@ static rpmRC packageBinary(rpmSpec spec, Package pkg, const char *cookie, int ch + headerCopyTags(spec->packages->header, pkg->header, copyTags); + + headerPutString(pkg->header, RPMTAG_RPMVERSION, VERSION); +- headerPutString(pkg->header, RPMTAG_BUILDHOST, buildHost()); +- headerPutUint32(pkg->header, RPMTAG_BUILDTIME, getBuildTime(), 1); ++ headerPutString(pkg->header, RPMTAG_BUILDHOST, buildHost); ++ headerPutUint32(pkg->header, RPMTAG_BUILDTIME, &buildTime, 1); + + if (spec->sourcePkgId != NULL) { + headerPutBin(pkg->header, RPMTAG_SOURCEPKGID, spec->sourcePkgId,16); +@@ -610,7 +604,7 @@ static rpmRC packageBinary(rpmSpec spec, Package pkg, const char *cookie, int ch + free(binRpm); + } + +- rc = writeRPM(pkg, NULL, *filename, NULL); ++ rc = writeRPM(pkg, NULL, *filename, NULL, buildTime, buildHost); + if (rc == RPMRC_OK) { + /* Do check each written package if enabled */ + char *pkgcheck = rpmExpand("%{?_build_pkgcheck} ", *filename, NULL); +@@ -629,6 +623,8 @@ struct binaryPackageTaskData + const char *cookie; + int cheating; + char *filename; ++ rpm_time_t buildTime; ++ const char *buildHost; + PRJob *job; + rpmRC result; + PRStatus status; +@@ -638,7 +634,7 @@ struct binaryPackageTaskData + static void runBinaryPackageTask(void* taskdata) + { + struct binaryPackageTaskData *task = taskdata; +- task->result = packageBinary(task->spec, task->pkg, task->cookie, task->cheating, &(task->filename)); ++ task->result = packageBinary(task->spec, task->pkg, task->cookie, task->cheating, &(task->filename), task->buildTime, task->buildHost); + } + + static struct binaryPackageTaskData* runBinaryPackageTasks(rpmSpec spec, const char *cookie, int cheating) +@@ -647,6 +643,8 @@ static struct binaryPackageTaskData* runBinaryPackageTasks(rpmSpec spec, const c + struct binaryPackageTaskData *task = NULL; + struct binaryPackageTaskData *prev = NULL; + PRThreadPool *pool = PR_CreateThreadPool(4, sysconf(_SC_NPROCESSORS_ONLN), 0); ++ rpm_time_t buildTime = getBuildTime(); ++ char *host = buildHost(); + + for (Package pkg = spec->packages; pkg != NULL; pkg = pkg->next) { + task = rcalloc(1, sizeof(*task)); +@@ -654,6 +652,8 @@ static struct binaryPackageTaskData* runBinaryPackageTasks(rpmSpec spec, const c + task->pkg = pkg; + task->cookie = cookie; + task->cheating = cheating; ++ task->buildTime = buildTime; ++ task->buildHost = host; + if (pkg == spec->packages) { + // the first package needs to be processed ahead of others, as they copy + // changelog data from it, and so otherwise data races would happen +@@ -681,6 +681,7 @@ static struct binaryPackageTaskData* runBinaryPackageTasks(rpmSpec spec, const c + } + } + PR_ShutdownThreadPool(pool); ++ free(host); + return tasks; + } + +@@ -731,16 +732,18 @@ rpmRC packageSources(rpmSpec spec, char **cookie) + rpmRC rc; + + /* Add some cruft */ ++ rpm_time_t buildTime = getBuildTime(); ++ char* host = buildHost(); + headerPutString(sourcePkg->header, RPMTAG_RPMVERSION, VERSION); +- headerPutString(sourcePkg->header, RPMTAG_BUILDHOST, buildHost()); +- headerPutUint32(sourcePkg->header, RPMTAG_BUILDTIME, getBuildTime(), 1); ++ headerPutString(sourcePkg->header, RPMTAG_BUILDHOST, (const char*)buildHost); ++ headerPutUint32(sourcePkg->header, RPMTAG_BUILDTIME, &buildTime, 1); + + /* XXX this should be %_srpmdir */ + { char *fn = rpmGetPath("%{_srcrpmdir}/", spec->sourceRpmName,NULL); + char *pkgcheck = rpmExpand("%{?_build_pkgcheck_srpm} ", fn, NULL); + + spec->sourcePkgId = NULL; +- rc = writeRPM(sourcePkg, &spec->sourcePkgId, fn, cookie); ++ rc = writeRPM(sourcePkg, &spec->sourcePkgId, fn, cookie, buildTime, host); + + /* Do check SRPM package if enabled */ + if (rc == RPMRC_OK && pkgcheck[0] != ' ') { +@@ -750,5 +753,6 @@ rpmRC packageSources(rpmSpec spec, char **cookie) + free(pkgcheck); + free(fn); + } ++ free(host); + return rc; + } +-- +2.11.0 + diff --git a/meta/recipes-devtools/rpm/rpm_git.bb b/meta/recipes-devtools/rpm/rpm_git.bb index 2310ee6b09e..b95b4719c19 100644 --- a/meta/recipes-devtools/rpm/rpm_git.bb +++ b/meta/recipes-devtools/rpm/rpm_git.bb @@ -35,6 +35,10 @@ SRC_URI = "git://github.com/rpm-software-management/rpm \ file://0001-Fix-build-with-musl-C-library.patch \ file://0001-Add-a-color-setting-for-mips64_n32-binaries.patch \ file://0001-Add-PYTHON_ABI-when-searching-for-python-libraries.patch \ + file://0001-Split-binary-package-building-into-a-separate-functi.patch \ + file://0002-Run-binary-package-creation-via-thread-pools.patch \ + file://0003-rpmstrpool.c-make-operations-over-string-pools-threa.patch \ + file://0004-build-pack.c-remove-static-local-variables-from-buil.patch \ " PV = "4.13.90+git${SRCPV}" -- 2.11.0 ^ permalink raw reply related [flat|nested] 24+ messages in thread
* Re: [PATCH 2/2] rpm: run binary package generation via thread pools 2017-06-01 15:15 ` [PATCH 2/2] rpm: run binary package generation via thread pools Alexander Kanavin @ 2017-06-01 16:02 ` Alexander Kanavin 2017-06-02 12:44 ` Alexander Kanavin 0 siblings, 1 reply; 24+ messages in thread From: Alexander Kanavin @ 2017-06-01 16:02 UTC (permalink / raw) To: openembedded-core On 06/01/2017 06:15 PM, Alexander Kanavin wrote: > This greatly reduces build times when there is a large amount of small > rpm packages to produce. The patches are rather invasive, > and so will be submitted upstream. And now the pull request is open: https://github.com/rpm-software-management/rpm/pull/226 https://github.com/rpm-software-management/rpm/issues/211 Alex ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 2/2] rpm: run binary package generation via thread pools 2017-06-01 16:02 ` Alexander Kanavin @ 2017-06-02 12:44 ` Alexander Kanavin 0 siblings, 0 replies; 24+ messages in thread From: Alexander Kanavin @ 2017-06-02 12:44 UTC (permalink / raw) To: openembedded-core On 06/01/2017 07:02 PM, Alexander Kanavin wrote: > On 06/01/2017 06:15 PM, Alexander Kanavin wrote: >> This greatly reduces build times when there is a large amount of small >> rpm packages to produce. The patches are rather invasive, >> and so will be submitted upstream. > > And now the pull request is open: > > https://github.com/rpm-software-management/rpm/pull/226 > https://github.com/rpm-software-management/rpm/issues/211 So the upstream's reaction to the pull request is that they're not against this, but they want the multiprocessing to be done with openmp (and not nss/nspr), given that they now actually support openssl as an alternative to nss, and don't want any hard dependencies on netscape stuff. I'll spend some time studying that. Alex ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 0/2] Multi-threaded RPM support 2017-06-01 15:15 [PATCH 0/2] Multi-threaded RPM support Alexander Kanavin 2017-06-01 15:15 ` [PATCH 1/2] package_rpm.bbclass: use multithreaded xz compression Alexander Kanavin 2017-06-01 15:15 ` [PATCH 2/2] rpm: run binary package generation via thread pools Alexander Kanavin @ 2017-06-01 16:52 ` Leonardo Sandoval 2017-06-01 17:13 ` Mark Hatle 2017-06-02 22:20 ` Richard Purdie 2017-06-04 14:15 ` Richard Purdie 4 siblings, 1 reply; 24+ messages in thread From: Leonardo Sandoval @ 2017-06-01 16:52 UTC (permalink / raw) To: Alexander Kanavin; +Cc: openembedded-core On Thu, 2017-06-01 at 18:15 +0300, Alexander Kanavin wrote: > These two patches add multi-threaded features to RPM to speed up the do_package_write_rpm task. > > Specifically: > > 1) Multi-threaded xz compressor is used instead of default single-threaded gz. This has the most > dramatic effect when a recipe produces a smaller number of large-sized packages. > > 2) Packages creation is run in multiple threads via thread pools. This is most beneficial when > a recipe produces a large amount of small packages. > > Some not very scientific benchmarks for time and .rpm sizes > (time is measured for do_package_write_rpm tasks only): > > webkitgtk > before: 9m22s 1550M > after: 1m50s 858M > > glibc-locale > before: 3m2s 125M > after: 40s 56M > > glibc > before: 56s 54M > after: 23s 38M > > perl > before: 1m19s 63M > after: 55s 42M > > python3 > before: 40s 38M > after: 28s 24M it is interesting to see that perl and python does not benefit 'much'. buildstats should give us a better picture per recipe. Leo > > The following changes since commit bd063fa288b49b6e3ea77982d0ccc46610feb1ad: > > tune-mips32*.inc: use consistent comments across all three .inc files (2017-05-30 10:15:20 +0100) > > are available in the git repository at: > > git://git.yoctoproject.org/poky-contrib akanavin/parallel-rpm > http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=akanavin/parallel-rpm > > Alexander Kanavin (2): > package_rpm.bbclass: use multithreaded xz compression > rpm: run binary package generation via thread pools > > meta/classes/package_rpm.bbclass | 2 + > ...y-package-building-into-a-separate-functi.patch | 83 ++++++++ > ...-binary-package-creation-via-thread-pools.patch | 137 +++++++++++++ > ...c-make-operations-over-string-pools-threa.patch | 209 +++++++++++++++++++ > ...c-remove-static-local-variables-from-buil.patch | 227 +++++++++++++++++++++ > meta/recipes-devtools/rpm/rpm_git.bb | 4 + > 6 files changed, 662 insertions(+) > create mode 100644 meta/recipes-devtools/rpm/files/0001-Split-binary-package-building-into-a-separate-functi.patch > create mode 100644 meta/recipes-devtools/rpm/files/0002-Run-binary-package-creation-via-thread-pools.patch > create mode 100644 meta/recipes-devtools/rpm/files/0003-rpmstrpool.c-make-operations-over-string-pools-threa.patch > create mode 100644 meta/recipes-devtools/rpm/files/0004-build-pack.c-remove-static-local-variables-from-buil.patch > > -- > 2.11.0 > ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 0/2] Multi-threaded RPM support 2017-06-01 16:52 ` [PATCH 0/2] Multi-threaded RPM support Leonardo Sandoval @ 2017-06-01 17:13 ` Mark Hatle 2017-06-01 17:37 ` Alexander Kanavin 0 siblings, 1 reply; 24+ messages in thread From: Mark Hatle @ 2017-06-01 17:13 UTC (permalink / raw) To: openembedded-core On 6/1/17 11:52 AM, Leonardo Sandoval wrote: > On Thu, 2017-06-01 at 18:15 +0300, Alexander Kanavin wrote: ... >> perl >> before: 1m19s 63M >> after: 55s 42M >> >> python3 >> before: 40s 38M >> after: 28s 24M > > it is interesting to see that perl and python does not benefit 'much'. > buildstats should give us a better picture per recipe. This may be due to how we process rpmdeps within OpenEmbedded.. but I'm not sure. I'd have expected a bigger time difference with large packages (lots of sub packages) like perl and python3. (kernel-modules would be another to look at, as it can get quite large.) ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 0/2] Multi-threaded RPM support 2017-06-01 17:13 ` Mark Hatle @ 2017-06-01 17:37 ` Alexander Kanavin 2017-06-02 21:41 ` Leonardo Sandoval 0 siblings, 1 reply; 24+ messages in thread From: Alexander Kanavin @ 2017-06-01 17:37 UTC (permalink / raw) To: Mark Hatle, openembedded-core, Leonardo Sandoval On 06/01/2017 08:13 PM, Mark Hatle wrote: >>> perl >>> before: 1m19s 63M >>> after: 55s 42M >>> >>> python3 >>> before: 40s 38M >>> after: 28s 24M >> >> it is interesting to see that perl and python does not benefit 'much'. >> buildstats should give us a better picture per recipe. > > This may be due to how we process rpmdeps within OpenEmbedded.. but I'm not > sure. I'd have expected a bigger time difference with large packages (lots of > sub packages) like perl and python3. The numbers include the overhead of starting bitbake (some 10 seconds). Subtract that (from both 'before' and 'after') and then it's more impressive :) I tested via time bitbake -f -c do_package_write_rpm perl Alex ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 0/2] Multi-threaded RPM support 2017-06-01 17:37 ` Alexander Kanavin @ 2017-06-02 21:41 ` Leonardo Sandoval 0 siblings, 0 replies; 24+ messages in thread From: Leonardo Sandoval @ 2017-06-02 21:41 UTC (permalink / raw) To: Alexander Kanavin; +Cc: openembedded-core Alex, I get this when bitbake core-image-minimal http://errors.yoctoproject.org/Errors/Build/38347/ HEAD is at http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=akanavin/parallel-rpm On Thu, 2017-06-01 at 20:37 +0300, Alexander Kanavin wrote: > On 06/01/2017 08:13 PM, Mark Hatle wrote: > >>> perl > >>> before: 1m19s 63M > >>> after: 55s 42M > >>> > >>> python3 > >>> before: 40s 38M > >>> after: 28s 24M > >> > >> it is interesting to see that perl and python does not benefit 'much'. > >> buildstats should give us a better picture per recipe. > > > > This may be due to how we process rpmdeps within OpenEmbedded.. but I'm not > > sure. I'd have expected a bigger time difference with large packages (lots of > > sub packages) like perl and python3. > > The numbers include the overhead of starting bitbake (some 10 seconds). > Subtract that (from both 'before' and 'after') and then it's more > impressive :) I tested via > > time bitbake -f -c do_package_write_rpm perl > > Alex > ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 0/2] Multi-threaded RPM support 2017-06-01 15:15 [PATCH 0/2] Multi-threaded RPM support Alexander Kanavin ` (2 preceding siblings ...) 2017-06-01 16:52 ` [PATCH 0/2] Multi-threaded RPM support Leonardo Sandoval @ 2017-06-02 22:20 ` Richard Purdie 2017-06-04 14:15 ` Richard Purdie 4 siblings, 0 replies; 24+ messages in thread From: Richard Purdie @ 2017-06-02 22:20 UTC (permalink / raw) To: Alexander Kanavin, openembedded-core I did get a laugh when this showed up in my mail client screen as: "Multi-threaded RP" :) Seriously, I am looking forward to trying and hopefully merging this, should make for some nice speedups. Cheers, RP ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 0/2] Multi-threaded RPM support 2017-06-01 15:15 [PATCH 0/2] Multi-threaded RPM support Alexander Kanavin ` (3 preceding siblings ...) 2017-06-02 22:20 ` Richard Purdie @ 2017-06-04 14:15 ` Richard Purdie 2017-06-05 12:14 ` Alexander Kanavin 4 siblings, 1 reply; 24+ messages in thread From: Richard Purdie @ 2017-06-04 14:15 UTC (permalink / raw) To: Alexander Kanavin, openembedded-core On Thu, 2017-06-01 at 18:15 +0300, Alexander Kanavin wrote: > These two patches add multi-threaded features to RPM to speed up the > do_package_write_rpm task. > > Specifically: > > 1) Multi-threaded xz compressor is used instead of default single- > threaded gz. This has the most > dramatic effect when a recipe produces a smaller number of large- > sized packages. > > 2) Packages creation is run in multiple threads via thread pools. > This is most beneficial when > a recipe produces a large amount of small packages. > > The following changes since commit > bd063fa288b49b6e3ea77982d0ccc46610feb1ad: > > tune-mips32*.inc: use consistent comments across all three .inc > files (2017-05-30 10:15:20 +0100) > > are available in the git repository at: > > git://git.yoctoproject.org/poky-contrib akanavin/parallel-rpm > http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=akanavin/p > arallel-rpm https://autobuilder.yoctoproject.org/main/builders/nightly-x86/builds/1 163/steps/BuildImages/logs/stdio https://autobuilder.yoctoproject.org/main/builders/nightly-x86-64/build s/1203/steps/BuildImages/logs/stdio https://autobuilder.yoctoproject.org/main/builders/nightly-ipk/builds/1 112/steps/BuildImages/logs/stdio I suspect the same issue causing failures in multiple places. Cheers, Richard ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 0/2] Multi-threaded RPM support 2017-06-04 14:15 ` Richard Purdie @ 2017-06-05 12:14 ` Alexander Kanavin 2017-06-05 12:41 ` Alexander Kanavin ` (2 more replies) 0 siblings, 3 replies; 24+ messages in thread From: Alexander Kanavin @ 2017-06-05 12:14 UTC (permalink / raw) To: Richard Purdie, openembedded-core, Leonardo Sandoval On 06/04/2017 05:15 PM, Richard Purdie wrote: > https://autobuilder.yoctoproject.org/main/builders/nightly-x86/builds/1 > 163/steps/BuildImages/logs/stdio > > https://autobuilder.yoctoproject.org/main/builders/nightly-x86-64/build > s/1203/steps/BuildImages/logs/stdio > > https://autobuilder.yoctoproject.org/main/builders/nightly-ipk/builds/1 > 112/steps/BuildImages/logs/stdio > > I suspect the same issue causing failures in multiple places. The patches as they are now require that the C compiler on the host is recent; specifically that it defaults to C11 mode. I'm not sure when GCC made the switch, but it works that way with GCC 6.3.0. In any case, I'll rework the patches to use openmp, as requested by upstream - this may however cause a different set of compatibility problems with older GCC versions. What are the oldest host compiler versions/distro versions that we still need to support in master? Alex ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 0/2] Multi-threaded RPM support 2017-06-05 12:14 ` Alexander Kanavin @ 2017-06-05 12:41 ` Alexander Kanavin 2017-06-05 13:01 ` Burton, Ross 2017-06-05 13:21 ` Joshua Lock 2 siblings, 0 replies; 24+ messages in thread From: Alexander Kanavin @ 2017-06-05 12:41 UTC (permalink / raw) To: Richard Purdie, openembedded-core, Leonardo Sandoval On 06/05/2017 03:14 PM, Alexander Kanavin wrote: > The patches as they are now require that the C compiler on the host is > recent; specifically that it defaults to C11 mode. I'm not sure when GCC > made the switch, but it works that way with GCC 6.3.0. I researched; the change happened in gcc 5 (released april 22 2015). Alex ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 0/2] Multi-threaded RPM support 2017-06-05 12:14 ` Alexander Kanavin 2017-06-05 12:41 ` Alexander Kanavin @ 2017-06-05 13:01 ` Burton, Ross 2017-06-05 13:15 ` Alexander Kanavin 2017-06-05 13:21 ` Joshua Lock 2 siblings, 1 reply; 24+ messages in thread From: Burton, Ross @ 2017-06-05 13:01 UTC (permalink / raw) To: Alexander Kanavin; +Cc: OE-core [-- Attachment #1: Type: text/plain, Size: 368 bytes --] On 5 June 2017 at 13:14, Alexander Kanavin < alexander.kanavin@linux.intel.com> wrote: > The patches as they are now require that the C compiler on the host is > recent; specifically that it defaults to C11 mode. I'm not sure when GCC > made the switch, but it works that way with GCC 6.3.0. > Isn't that just a compiler flag away from being fixed? Ross [-- Attachment #2: Type: text/html, Size: 797 bytes --] ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 0/2] Multi-threaded RPM support 2017-06-05 13:01 ` Burton, Ross @ 2017-06-05 13:15 ` Alexander Kanavin 0 siblings, 0 replies; 24+ messages in thread From: Alexander Kanavin @ 2017-06-05 13:15 UTC (permalink / raw) To: Burton, Ross; +Cc: OE-core On 06/05/2017 04:01 PM, Burton, Ross wrote: > > On 5 June 2017 at 13:14, Alexander Kanavin > <alexander.kanavin@linux.intel.com > <mailto:alexander.kanavin@linux.intel.com>> wrote: > > The patches as they are now require that the C compiler on the host > is recent; specifically that it defaults to C11 mode. I'm not sure > when GCC made the switch, but it works that way with GCC 6.3.0. > > > Isn't that just a compiler flag away from being fixed? Not if the compiler is 4.8 or earlier; it has to support _Atomic (a C11 feature), which only appeared in 4.9. Alex ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 0/2] Multi-threaded RPM support 2017-06-05 12:14 ` Alexander Kanavin 2017-06-05 12:41 ` Alexander Kanavin 2017-06-05 13:01 ` Burton, Ross @ 2017-06-05 13:21 ` Joshua Lock 2017-06-05 13:34 ` Alexander Kanavin 2 siblings, 1 reply; 24+ messages in thread From: Joshua Lock @ 2017-06-05 13:21 UTC (permalink / raw) To: Alexander Kanavin, Richard Purdie, openembedded-core, Leonardo Sandoval On Mon, 2017-06-05 at 15:14 +0300, Alexander Kanavin wrote: > On 06/04/2017 05:15 PM, Richard Purdie wrote: > > > https://autobuilder.yoctoproject.org/main/builders/nightly-x86/buil > > ds/1 > > 163/steps/BuildImages/logs/stdio > > > > https://autobuilder.yoctoproject.org/main/builders/nightly-x86-64/b > > uild > > s/1203/steps/BuildImages/logs/stdio > > > > https://autobuilder.yoctoproject.org/main/builders/nightly-ipk/buil > > ds/1 > > 112/steps/BuildImages/logs/stdio > > > > I suspect the same issue causing failures in multiple places. > > The patches as they are now require that the C compiler on the host > is > recent; specifically that it defaults to C11 mode. I'm not sure when > GCC > made the switch, but it works that way with GCC 6.3.0. > > In any case, I'll rework the patches to use openmp, as requested by > upstream - this may however cause a different set of compatibility > problems with older GCC versions. What are the oldest host compiler > versions/distro versions that we still need to support in master? On the Yocto Autobuilder clusters we have: * centos7 / gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-11) * debian8 / gcc (Debian 4.9.2-10) 4.9.2 * debian-testing / gcc (Debian 6.3.0-14) 6.3.0 20170415 * fedora24 / gcc (GCC) 6.3.1 20161221 (Red Hat 6.3.1-1) * fedora25 / gcc (GCC) 6.3.1 20161221 (Red Hat 6.3.1-1) * opensuse132 / gcc (SUSE Linux) 4.8.3 20140627 [gcc-4_8-branch revision 212064] * opensuse422 / gcc (SUSE Linux) 4.8.5 * ubuntu1604 / gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609 * ubuntu1610 / gcc (Ubuntu 6.2.0-5ubuntu12) 6.2.0 20161005 We are aiming to replace openSUSE 13.2 (as it was discontinued in January of this year) but CentOS 7 has a good long lifetime ahead of it, therefore supporting it and RHEL 7 will require gcc 4.8.5 support. Joshua ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 0/2] Multi-threaded RPM support 2017-06-05 13:21 ` Joshua Lock @ 2017-06-05 13:34 ` Alexander Kanavin 2017-06-05 13:47 ` Davis, Michael 0 siblings, 1 reply; 24+ messages in thread From: Alexander Kanavin @ 2017-06-05 13:34 UTC (permalink / raw) To: Joshua Lock, Richard Purdie, openembedded-core, Leonardo Sandoval On 06/05/2017 04:21 PM, Joshua Lock wrote: > On the Yocto Autobuilder clusters we have: > * centos7 / gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-11) > * debian8 / gcc (Debian 4.9.2-10) 4.9.2 > * debian-testing / gcc (Debian 6.3.0-14) 6.3.0 20170415 > * fedora24 / gcc (GCC) 6.3.1 20161221 (Red Hat 6.3.1-1) > * fedora25 / gcc (GCC) 6.3.1 20161221 (Red Hat 6.3.1-1) > * opensuse132 / gcc (SUSE Linux) 4.8.3 20140627 [gcc-4_8-branch > revision 212064] > * opensuse422 / gcc (SUSE Linux) 4.8.5 > * ubuntu1604 / gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609 > * ubuntu1610 / gcc (Ubuntu 6.2.0-5ubuntu12) 6.2.0 20161005 > > We are aiming to replace openSUSE 13.2 (as it was discontinued in > January of this year) but CentOS 7 has a good long lifetime ahead of > it, therefore supporting it and RHEL 7 will require gcc 4.8.5 support. Thanks; in that case using atomic types and other C11 features is not possible at all for quite some time, and openmp support, while available, is restricted to version 3.1 of the spec (which should be just about enough - earlier versions do not have task support). I actually have gcc 4.8 here (on debian testing, which will become debian 9 in a couple of weeks), even though the default version is 6.3, so I can test 4.8 compatibility locally. Alex ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 0/2] Multi-threaded RPM support 2017-06-05 13:34 ` Alexander Kanavin @ 2017-06-05 13:47 ` Davis, Michael 2017-06-05 14:00 ` Alexander Kanavin 0 siblings, 1 reply; 24+ messages in thread From: Davis, Michael @ 2017-06-05 13:47 UTC (permalink / raw) To: Alexander Kanavin, Joshua Lock, Richard Purdie, openembedded-core, Leonardo Sandoval I just wanted to note that in my quest to get newer chrome to build I have used centos7 with the official devtoolset-6 scl. It was able to build my pyro based distro fine in gcc6. So Centos7 has an officially supported path that may be viable if it comes down to being the last gcc 4.x distro. -----Original Message----- From: openembedded-core-bounces@lists.openembedded.org [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf Of Alexander Kanavin Sent: Monday, June 05, 2017 8:35 AM To: Joshua Lock; Richard Purdie; openembedded-core@lists.openembedded.org; Leonardo Sandoval Subject: Re: [OE-core] [PATCH 0/2] Multi-threaded RPM support On 06/05/2017 04:21 PM, Joshua Lock wrote: > On the Yocto Autobuilder clusters we have: > * centos7 / gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-11) > * debian8 / gcc (Debian 4.9.2-10) 4.9.2 > * debian-testing / gcc (Debian 6.3.0-14) 6.3.0 20170415 > * fedora24 / gcc (GCC) 6.3.1 20161221 (Red Hat 6.3.1-1) > * fedora25 / gcc (GCC) 6.3.1 20161221 (Red Hat 6.3.1-1) > * opensuse132 / gcc (SUSE Linux) 4.8.3 20140627 [gcc-4_8-branch > revision 212064] > * opensuse422 / gcc (SUSE Linux) 4.8.5 > * ubuntu1604 / gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609 > * ubuntu1610 / gcc (Ubuntu 6.2.0-5ubuntu12) 6.2.0 20161005 > > We are aiming to replace openSUSE 13.2 (as it was discontinued in > January of this year) but CentOS 7 has a good long lifetime ahead of > it, therefore supporting it and RHEL 7 will require gcc 4.8.5 support. Thanks; in that case using atomic types and other C11 features is not possible at all for quite some time, and openmp support, while available, is restricted to version 3.1 of the spec (which should be just about enough - earlier versions do not have task support). I actually have gcc 4.8 here (on debian testing, which will become debian 9 in a couple of weeks), even though the default version is 6.3, so I can test 4.8 compatibility locally. Alex -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 0/2] Multi-threaded RPM support 2017-06-05 13:47 ` Davis, Michael @ 2017-06-05 14:00 ` Alexander Kanavin 0 siblings, 0 replies; 24+ messages in thread From: Alexander Kanavin @ 2017-06-05 14:00 UTC (permalink / raw) To: Davis, Michael, Joshua Lock, Richard Purdie, openembedded-core, Leonardo Sandoval On 06/05/2017 04:47 PM, Davis, Michael wrote: > I just wanted to note that in my quest to get newer chrome to build I have used centos7 with the official devtoolset-6 scl. > It was able to build my pyro based distro fine in gcc6. So Centos7 has an officially supported path that may be viable if it comes down to being the last gcc 4.x distro. I guess at some point we'll have to officially do this as well, as the upstream projects will adopt C/C++11 (or even newer) and otherwise become incompatible with 4.x. Alex ^ permalink raw reply [flat|nested] 24+ messages in thread
end of thread, other threads:[~2017-06-05 14:03 UTC | newest] Thread overview: 24+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2017-06-01 15:15 [PATCH 0/2] Multi-threaded RPM support Alexander Kanavin 2017-06-01 15:15 ` [PATCH 1/2] package_rpm.bbclass: use multithreaded xz compression Alexander Kanavin 2017-06-01 17:11 ` Mark Hatle 2017-06-01 17:44 ` Alexander Kanavin 2017-06-01 19:02 ` Andre McCurdy 2017-06-02 11:40 ` Alexander Kanavin 2017-06-02 18:33 ` Andre McCurdy 2017-06-01 15:15 ` [PATCH 2/2] rpm: run binary package generation via thread pools Alexander Kanavin 2017-06-01 16:02 ` Alexander Kanavin 2017-06-02 12:44 ` Alexander Kanavin 2017-06-01 16:52 ` [PATCH 0/2] Multi-threaded RPM support Leonardo Sandoval 2017-06-01 17:13 ` Mark Hatle 2017-06-01 17:37 ` Alexander Kanavin 2017-06-02 21:41 ` Leonardo Sandoval 2017-06-02 22:20 ` Richard Purdie 2017-06-04 14:15 ` Richard Purdie 2017-06-05 12:14 ` Alexander Kanavin 2017-06-05 12:41 ` Alexander Kanavin 2017-06-05 13:01 ` Burton, Ross 2017-06-05 13:15 ` Alexander Kanavin 2017-06-05 13:21 ` Joshua Lock 2017-06-05 13:34 ` Alexander Kanavin 2017-06-05 13:47 ` Davis, Michael 2017-06-05 14:00 ` Alexander Kanavin
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.