* [yocto-autobuilder-helper] [PATCH 1/5] scripts/run-patchmetrics: Only clone metrics if it isn't present
@ 2023-10-23 15:43 Richard Purdie
2023-10-23 15:43 ` [yocto-autobuilder-helper] [PATCH 2/5] scripts/run-patchmetrics: Split out CVE checks Richard Purdie
` (3 more replies)
0 siblings, 4 replies; 5+ messages in thread
From: Richard Purdie @ 2023-10-23 15:43 UTC (permalink / raw)
To: yocto
To prepare for splitting things up, only clone the metrics repo if it isn't present.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
scripts/run-patchmetrics | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/scripts/run-patchmetrics b/scripts/run-patchmetrics
index fc3f214..abe58c7 100755
--- a/scripts/run-patchmetrics
+++ b/scripts/run-patchmetrics
@@ -13,7 +13,9 @@ TIMESTAMP=`date +"%s"`
#
# Patch Metrics
#
-git clone ssh://git@push.yoctoproject.org/yocto-metrics $PARENTDIR/yocto-metrics
+if [ ! -e $PARENTDIR/yocto-metrics ]; then
+ git clone ssh://git@push.yoctoproject.org/yocto-metrics $PARENTDIR/yocto-metrics
+fi
$OURDIR/patchmetrics-update --repo $PARENTDIR --patchscript $PARENTDIR/scripts/contrib/patchreview.py --metadata $TARGETDIR --json $PARENTDIR/yocto-metrics/patch-status.json
git -C $PARENTDIR/yocto-metrics commit -asm "Autobuilder adding new patch stats"
git -C $PARENTDIR/yocto-metrics push
--
2.39.2
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [yocto-autobuilder-helper] [PATCH 2/5] scripts/run-patchmetrics: Split out CVE checks
2023-10-23 15:43 [yocto-autobuilder-helper] [PATCH 1/5] scripts/run-patchmetrics: Only clone metrics if it isn't present Richard Purdie
@ 2023-10-23 15:43 ` Richard Purdie
2023-10-23 15:43 ` [yocto-autobuilder-helper] [PATCH 3/5] metrics: Pass branchname to scripts Richard Purdie
` (2 subsequent siblings)
3 siblings, 0 replies; 5+ messages in thread
From: Richard Purdie @ 2023-10-23 15:43 UTC (permalink / raw)
To: yocto
Split the CVE checks from the patch metrics script
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
config.json | 6 +++++
scripts/run-cvecheck | 54 ++++++++++++++++++++++++++++++++++++++++
scripts/run-patchmetrics | 36 ---------------------------
3 files changed, 60 insertions(+), 36 deletions(-)
create mode 100755 scripts/run-cvecheck
diff --git a/config.json b/config.json
index bebd999..f225148 100644
--- a/config.json
+++ b/config.json
@@ -1208,8 +1208,14 @@
"BB_SERVER_TIMEOUT = '0'"
],
"step1" : {
+ "shortname" : "Generating patch metrics",
"EXTRACMDS" : ["../../yocto-autobuilder-helper/scripts/run-patchmetrics ../ ../meta/ ${HELPERRESULTSDIR}/../../patchmetrics ."]
+ },
+ "step2" : {
+ "shortname" : "Running CVE checks",
+ "EXTRACMDS" : ["../../yocto-autobuilder-helper/scripts/run-cvecheck ../ ../meta/ ${HELPERRESULTSDIR}/../../patchmetrics ."]
}
+
},
"meta-mingw" : {
"NEEDREPOS" : ["poky", "meta-mingw"],
diff --git a/scripts/run-cvecheck b/scripts/run-cvecheck
new file mode 100755
index 0000000..35c796b
--- /dev/null
+++ b/scripts/run-cvecheck
@@ -0,0 +1,54 @@
+#!/bin/bash
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+PARENTDIR=`realpath $1`
+TARGETDIR=`realpath $2`
+RESULTSDIR=`realpath -m $3`
+BUILDDIR=`realpath $4`
+OURDIR=`dirname $0`
+
+TIMESTAMP=`date +"%s"`
+
+#
+# CVE Checks
+#
+if [ ! -e $PARENTDIR/yocto-metrics ]; then
+ git clone ssh://git@push.yoctoproject.org/yocto-metrics $PARENTDIR/yocto-metrics
+fi
+
+if [ ! -d $RESULTSDIR ]; then
+ mkdir $RESULTSDIR
+fi
+
+for branch in master mickledore langdale kirkstone dunfell; do
+ mkdir -p $PARENTDIR/yocto-metrics/cve-check/$branch/
+ git -C $PARENTDIR reset origin/$branch --hard
+ rm conf/local.conf
+ rm conf/bblayers.conf
+ rm -f conf/templateconf.cfg
+ rm tmp/ -rf
+ unset BB_ENV_PASSTHROUGH_ADDITIONS
+ unset BB_ENV_EXTRAWHITE
+ cd ..
+ . oe-init-build-env build
+ bitbake world --runall cve_check -R conf/distro/include/cve-extra-exclusions.inc
+ if [ -e tmp/log/cve/cve-summary.json ]; then
+ git -C $PARENTDIR/yocto-metrics rm cve-check/$branch/*.json
+ mkdir -p $PARENTDIR/yocto-metrics/cve-check/$branch
+ cp tmp/log/cve/cve-summary.json $PARENTDIR/yocto-metrics/cve-check/$branch/$TIMESTAMP.json
+ git -C $PARENTDIR/yocto-metrics add cve-check/$branch/$TIMESTAMP.json
+ git -C $PARENTDIR/yocto-metrics commit -asm "Autobuilder adding new CVE data for branch $branch"
+ git -C $PARENTDIR/yocto-metrics push
+ $OURDIR/cve-report.py tmp/log/cve/cve-summary.json > $RESULTSDIR/cve-status-$branch.txt
+ fi
+done
+
+mkdir -p $PARENTDIR/yocto-metrics/cve-check/
+$OURDIR/cve-generate-chartdata --json $PARENTDIR/yocto-metrics/cve-count-byday.json --resultsdir $PARENTDIR/yocto-metrics/cve-check/
+git -C $PARENTDIR/yocto-metrics add cve-count-byday.json
+git -C $PARENTDIR/yocto-metrics commit -asm "Autobuilder updating CVE counts"
+git -C $PARENTDIR/yocto-metrics push
+
+cp $PARENTDIR/yocto-metrics/cve-count-byday.json $RESULTSDIR
+cp $PARENTDIR/yocto-metrics/cve-count-byday-lastyear.json $RESULTSDIR
diff --git a/scripts/run-patchmetrics b/scripts/run-patchmetrics
index abe58c7..e45d463 100755
--- a/scripts/run-patchmetrics
+++ b/scripts/run-patchmetrics
@@ -27,39 +27,3 @@ fi
$OURDIR/patchmetrics-generate-chartdata --json $PARENTDIR/yocto-metrics/patch-status.json --outputdir $RESULTSDIR
cp $PARENTDIR/yocto-metrics/patch-status.json $RESULTSDIR
cp $PARENTDIR/yocto-metrics/patch-status/* $RESULTSDIR
-
-#
-# CVE Checks
-#
-for branch in master mickledore langdale kirkstone dunfell; do
- mkdir -p $PARENTDIR/yocto-metrics/cve-check/$branch/
- git -C $PARENTDIR reset origin/$branch --hard
- rm conf/local.conf
- rm conf/bblayers.conf
- rm -f conf/templateconf.cfg
- rm tmp/ -rf
- unset BB_ENV_PASSTHROUGH_ADDITIONS
- unset BB_ENV_EXTRAWHITE
- cd ..
- . oe-init-build-env build
- bitbake world --runall cve_check -R conf/distro/include/cve-extra-exclusions.inc
- if [ -e tmp/log/cve/cve-summary.json ]; then
- git -C $PARENTDIR/yocto-metrics rm cve-check/$branch/*.json
- mkdir -p $PARENTDIR/yocto-metrics/cve-check/$branch
- cp tmp/log/cve/cve-summary.json $PARENTDIR/yocto-metrics/cve-check/$branch/$TIMESTAMP.json
- git -C $PARENTDIR/yocto-metrics add cve-check/$branch/$TIMESTAMP.json
- git -C $PARENTDIR/yocto-metrics commit -asm "Autobuilder adding new CVE data for branch $branch"
- git -C $PARENTDIR/yocto-metrics push
- $OURDIR/cve-report.py tmp/log/cve/cve-summary.json > $RESULTSDIR/cve-status-$branch.txt
- fi
-done
-
-mkdir -p $PARENTDIR/yocto-metrics/cve-check/
-$OURDIR/cve-generate-chartdata --json $PARENTDIR/yocto-metrics/cve-count-byday.json --resultsdir $PARENTDIR/yocto-metrics/cve-check/
-git -C $PARENTDIR/yocto-metrics add cve-count-byday.json
-git -C $PARENTDIR/yocto-metrics commit -asm "Autobuilder updating CVE counts"
-git -C $PARENTDIR/yocto-metrics push
-
-cp $PARENTDIR/yocto-metrics/cve-count-byday.json $RESULTSDIR
-cp $PARENTDIR/yocto-metrics/cve-count-byday-lastyear.json $RESULTSDIR
-
--
2.39.2
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [yocto-autobuilder-helper] [PATCH 3/5] metrics: Pass branchname to scripts
2023-10-23 15:43 [yocto-autobuilder-helper] [PATCH 1/5] scripts/run-patchmetrics: Only clone metrics if it isn't present Richard Purdie
2023-10-23 15:43 ` [yocto-autobuilder-helper] [PATCH 2/5] scripts/run-patchmetrics: Split out CVE checks Richard Purdie
@ 2023-10-23 15:43 ` Richard Purdie
2023-10-23 15:43 ` [yocto-autobuilder-helper] [PATCH 4/5] scripts/run-patchmetrics: Only monitor master branch Richard Purdie
2023-10-23 15:43 ` [yocto-autobuilder-helper] [PATCH 5/5] scripts/run-cvecheck: Remove branch iteration Richard Purdie
3 siblings, 0 replies; 5+ messages in thread
From: Richard Purdie @ 2023-10-23 15:43 UTC (permalink / raw)
To: yocto
To prepapre to run the scripts per branch, pass the branchname to the scripts.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
config.json | 4 ++--
scripts/run-cvecheck | 1 +
scripts/run-patchmetrics | 1 +
3 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/config.json b/config.json
index f225148..0c35632 100644
--- a/config.json
+++ b/config.json
@@ -1209,11 +1209,11 @@
],
"step1" : {
"shortname" : "Generating patch metrics",
- "EXTRACMDS" : ["../../yocto-autobuilder-helper/scripts/run-patchmetrics ../ ../meta/ ${HELPERRESULTSDIR}/../../patchmetrics ."]
+ "EXTRACMDS" : ["../../yocto-autobuilder-helper/scripts/run-patchmetrics ../ ../meta/ ${HELPERRESULTSDIR}/../../patchmetrics . ${HELPERBRANCHNAME}"]
},
"step2" : {
"shortname" : "Running CVE checks",
- "EXTRACMDS" : ["../../yocto-autobuilder-helper/scripts/run-cvecheck ../ ../meta/ ${HELPERRESULTSDIR}/../../patchmetrics ."]
+ "EXTRACMDS" : ["../../yocto-autobuilder-helper/scripts/run-cvecheck ../ ../meta/ ${HELPERRESULTSDIR}/../../patchmetrics . ${HELPERBRANCHNAME}"]
}
},
diff --git a/scripts/run-cvecheck b/scripts/run-cvecheck
index 35c796b..d48fd68 100755
--- a/scripts/run-cvecheck
+++ b/scripts/run-cvecheck
@@ -6,6 +6,7 @@ PARENTDIR=`realpath $1`
TARGETDIR=`realpath $2`
RESULTSDIR=`realpath -m $3`
BUILDDIR=`realpath $4`
+BRANCH=$5
OURDIR=`dirname $0`
TIMESTAMP=`date +"%s"`
diff --git a/scripts/run-patchmetrics b/scripts/run-patchmetrics
index e45d463..20e6268 100755
--- a/scripts/run-patchmetrics
+++ b/scripts/run-patchmetrics
@@ -6,6 +6,7 @@ PARENTDIR=`realpath $1`
TARGETDIR=`realpath $2`
RESULTSDIR=`realpath -m $3`
BUILDDIR=`realpath $4`
+BRANCH=$5
OURDIR=`dirname $0`
TIMESTAMP=`date +"%s"`
--
2.39.2
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [yocto-autobuilder-helper] [PATCH 4/5] scripts/run-patchmetrics: Only monitor master branch
2023-10-23 15:43 [yocto-autobuilder-helper] [PATCH 1/5] scripts/run-patchmetrics: Only clone metrics if it isn't present Richard Purdie
2023-10-23 15:43 ` [yocto-autobuilder-helper] [PATCH 2/5] scripts/run-patchmetrics: Split out CVE checks Richard Purdie
2023-10-23 15:43 ` [yocto-autobuilder-helper] [PATCH 3/5] metrics: Pass branchname to scripts Richard Purdie
@ 2023-10-23 15:43 ` Richard Purdie
2023-10-23 15:43 ` [yocto-autobuilder-helper] [PATCH 5/5] scripts/run-cvecheck: Remove branch iteration Richard Purdie
3 siblings, 0 replies; 5+ messages in thread
From: Richard Purdie @ 2023-10-23 15:43 UTC (permalink / raw)
To: yocto
We only monitor the master branch for patch metrics as we can't really make
improvements to release branches.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
scripts/run-patchmetrics | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/scripts/run-patchmetrics b/scripts/run-patchmetrics
index 20e6268..391ac45 100755
--- a/scripts/run-patchmetrics
+++ b/scripts/run-patchmetrics
@@ -11,6 +11,11 @@ OURDIR=`dirname $0`
TIMESTAMP=`date +"%s"`
+# We only monitor patch metrics on the master branch
+if [ "$BRANCH" != "master" ]; then
+ exit 0
+fi
+
#
# Patch Metrics
#
--
2.39.2
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [yocto-autobuilder-helper] [PATCH 5/5] scripts/run-cvecheck: Remove branch iteration
2023-10-23 15:43 [yocto-autobuilder-helper] [PATCH 1/5] scripts/run-patchmetrics: Only clone metrics if it isn't present Richard Purdie
` (2 preceding siblings ...)
2023-10-23 15:43 ` [yocto-autobuilder-helper] [PATCH 4/5] scripts/run-patchmetrics: Only monitor master branch Richard Purdie
@ 2023-10-23 15:43 ` Richard Purdie
3 siblings, 0 replies; 5+ messages in thread
From: Richard Purdie @ 2023-10-23 15:43 UTC (permalink / raw)
To: yocto
Rather than running multiple checkouts, lets move this to the autobuilder
to handle and have it trigger the builds with the right checkouts.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
scripts/run-cvecheck | 52 ++++++++++++++++++++------------------------
1 file changed, 23 insertions(+), 29 deletions(-)
diff --git a/scripts/run-cvecheck b/scripts/run-cvecheck
index d48fd68..6294fe6 100755
--- a/scripts/run-cvecheck
+++ b/scripts/run-cvecheck
@@ -22,34 +22,28 @@ if [ ! -d $RESULTSDIR ]; then
mkdir $RESULTSDIR
fi
-for branch in master mickledore langdale kirkstone dunfell; do
- mkdir -p $PARENTDIR/yocto-metrics/cve-check/$branch/
- git -C $PARENTDIR reset origin/$branch --hard
- rm conf/local.conf
- rm conf/bblayers.conf
- rm -f conf/templateconf.cfg
- rm tmp/ -rf
- unset BB_ENV_PASSTHROUGH_ADDITIONS
- unset BB_ENV_EXTRAWHITE
- cd ..
- . oe-init-build-env build
- bitbake world --runall cve_check -R conf/distro/include/cve-extra-exclusions.inc
- if [ -e tmp/log/cve/cve-summary.json ]; then
- git -C $PARENTDIR/yocto-metrics rm cve-check/$branch/*.json
- mkdir -p $PARENTDIR/yocto-metrics/cve-check/$branch
- cp tmp/log/cve/cve-summary.json $PARENTDIR/yocto-metrics/cve-check/$branch/$TIMESTAMP.json
- git -C $PARENTDIR/yocto-metrics add cve-check/$branch/$TIMESTAMP.json
- git -C $PARENTDIR/yocto-metrics commit -asm "Autobuilder adding new CVE data for branch $branch"
- git -C $PARENTDIR/yocto-metrics push
- $OURDIR/cve-report.py tmp/log/cve/cve-summary.json > $RESULTSDIR/cve-status-$branch.txt
- fi
-done
+mkdir -p $PARENTDIR/yocto-metrics/cve-check/$BRANCH/
+cd ..
+. oe-init-build-env build
+bitbake world --runall cve_check -R conf/distro/include/cve-extra-exclusions.inc
+if [ -e tmp/log/cve/cve-summary.json ]; then
+ git -C $PARENTDIR/yocto-metrics rm cve-check/$BRANCH/*.json
+ mkdir -p $PARENTDIR/yocto-metrics/cve-check/$BRANCH
+ cp tmp/log/cve/cve-summary.json $PARENTDIR/yocto-metrics/cve-check/$BRANCH/$TIMESTAMP.json
+ git -C $PARENTDIR/yocto-metrics add cve-check/$BRANCH/$TIMESTAMP.json
+ git -C $PARENTDIR/yocto-metrics commit -asm "Autobuilder adding new CVE data for branch $BRANCH"
+ git -C $PARENTDIR/yocto-metrics push
+ $OURDIR/cve-report.py tmp/log/cve/cve-summary.json > $RESULTSDIR/cve-status-$BRANCH.txt
+fi
+
+if [ "$BRANCH" = "master" ]; then
+ mkdir -p $PARENTDIR/yocto-metrics/cve-check/
+ $OURDIR/cve-generate-chartdata --json $PARENTDIR/yocto-metrics/cve-count-byday.json --resultsdir $PARENTDIR/yocto-metrics/cve-check/
+ git -C $PARENTDIR/yocto-metrics add cve-count-byday.json
+ git -C $PARENTDIR/yocto-metrics commit -asm "Autobuilder updating CVE counts"
+ git -C $PARENTDIR/yocto-metrics push
-mkdir -p $PARENTDIR/yocto-metrics/cve-check/
-$OURDIR/cve-generate-chartdata --json $PARENTDIR/yocto-metrics/cve-count-byday.json --resultsdir $PARENTDIR/yocto-metrics/cve-check/
-git -C $PARENTDIR/yocto-metrics add cve-count-byday.json
-git -C $PARENTDIR/yocto-metrics commit -asm "Autobuilder updating CVE counts"
-git -C $PARENTDIR/yocto-metrics push
+ cp $PARENTDIR/yocto-metrics/cve-count-byday.json $RESULTSDIR
+ cp $PARENTDIR/yocto-metrics/cve-count-byday-lastyear.json $RESULTSDIR
+fi
-cp $PARENTDIR/yocto-metrics/cve-count-byday.json $RESULTSDIR
-cp $PARENTDIR/yocto-metrics/cve-count-byday-lastyear.json $RESULTSDIR
--
2.39.2
^ permalink raw reply related [flat|nested] 5+ messages in thread
end of thread, other threads:[~2023-10-23 15:44 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-10-23 15:43 [yocto-autobuilder-helper] [PATCH 1/5] scripts/run-patchmetrics: Only clone metrics if it isn't present Richard Purdie
2023-10-23 15:43 ` [yocto-autobuilder-helper] [PATCH 2/5] scripts/run-patchmetrics: Split out CVE checks Richard Purdie
2023-10-23 15:43 ` [yocto-autobuilder-helper] [PATCH 3/5] metrics: Pass branchname to scripts Richard Purdie
2023-10-23 15:43 ` [yocto-autobuilder-helper] [PATCH 4/5] scripts/run-patchmetrics: Only monitor master branch Richard Purdie
2023-10-23 15:43 ` [yocto-autobuilder-helper] [PATCH 5/5] scripts/run-cvecheck: Remove branch iteration Richard Purdie
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.