* [cip-dev] [cip-kernel-sec][RESEND 1/6] check_git_repo: add checks to the local repository
2019-07-10 1:24 [cip-dev] Add support for cip branches and tags Daniel Sangorrin
@ 2019-07-10 1:24 ` Daniel Sangorrin
2019-07-10 1:24 ` [cip-dev] [cip-kernel-sec][RESEND 2/6] prepare_remotes: helper script to prepare local repo Daniel Sangorrin
` (5 subsequent siblings)
6 siblings, 0 replies; 10+ messages in thread
From: Daniel Sangorrin @ 2019-07-10 1:24 UTC (permalink / raw)
To: cip-dev
Add checks to make sure that the local repository exists
and has the configured remotes in place.
Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
scripts/import_stable.py | 1 +
scripts/kernel_sec/branch.py | 21 +++++++++++++++++++++
scripts/report_affected.py | 1 +
scripts/webview.py | 1 +
4 files changed, 24 insertions(+)
diff --git a/scripts/import_stable.py b/scripts/import_stable.py
index 2c88b07..26fcc2f 100755
--- a/scripts/import_stable.py
+++ b/scripts/import_stable.py
@@ -190,4 +190,5 @@ if __name__ == '__main__':
remotes = kernel_sec.branch.get_remotes(args.remote_name,
mainline=args.mainline_remote_name,
stable=args.stable_remote_name)
+ kernel_sec.branch.check_git_repo(args.git_repo, remotes)
main(args.git_repo, remotes, args.debug)
diff --git a/scripts/kernel_sec/branch.py b/scripts/kernel_sec/branch.py
index 10f3339..a138b96 100644
--- a/scripts/kernel_sec/branch.py
+++ b/scripts/kernel_sec/branch.py
@@ -4,10 +4,12 @@
# Public License, Version 3 or later. See http://www.gnu.org/copyleft/gpl.html
# for details.
+import argparse
import io
import os
import re
import subprocess
+import sys
import time
import urllib.error
import urllib.request
@@ -219,3 +221,22 @@ def get_remotes(mappings, mainline=None, stable=None):
if stable:
remotes['stable']['git_name'] = stable
return remotes
+
+
+def check_git_repo(git_repo, remotes):
+ if not os.path.isdir(git_repo):
+ msg = "directory %r not present" % git_repo
+ raise argparse.ArgumentError(None, msg)
+ # .git could be a regular file (worktrees) or a directory
+ if not os.path.exists(os.path.join(git_repo, '.git')):
+ msg = "directory %r is not a git repository" % git_repo
+ raise argparse.ArgumentError(None, msg)
+
+ current_remotes = subprocess.check_output(
+ ['git', 'remote', 'show'], cwd=git_repo).decode(
+ sys.stdout.encoding).strip().split('\n')
+ for key in remotes.keys():
+ remote = remotes[key] # __getitem__ will add git_name
+ if remote['git_name'] not in current_remotes:
+ msg = "remote %r not in git repository" % remote['git_name']
+ raise argparse.ArgumentError(None, msg)
diff --git a/scripts/report_affected.py b/scripts/report_affected.py
index 879c021..d2f1f22 100755
--- a/scripts/report_affected.py
+++ b/scripts/report_affected.py
@@ -100,5 +100,6 @@ if __name__ == '__main__':
remotes = kernel_sec.branch.get_remotes(args.remote_name,
mainline=args.mainline_remote_name,
stable=args.stable_remote_name)
+ kernel_sec.branch.check_git_repo(args.git_repo, remotes)
main(args.git_repo, remotes,
args.only_fixed_upstream, args.include_ignored, *args.branches)
diff --git a/scripts/webview.py b/scripts/webview.py
index a3a643b..52a7b15 100755
--- a/scripts/webview.py
+++ b/scripts/webview.py
@@ -219,6 +219,7 @@ if __name__ == '__main__':
remotes = kernel_sec.branch.get_remotes(args.remote_name,
mainline=args.mainline_remote_name,
stable=args.stable_remote_name)
+ kernel_sec.branch.check_git_repo(args.git_repo, remotes)
conf = {
'/static/style.css': {
--
2.17.1
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [cip-dev] [cip-kernel-sec][RESEND 2/6] prepare_remotes: helper script to prepare local repo
2019-07-10 1:24 [cip-dev] Add support for cip branches and tags Daniel Sangorrin
2019-07-10 1:24 ` [cip-dev] [cip-kernel-sec][RESEND 1/6] check_git_repo: add checks to the local repository Daniel Sangorrin
@ 2019-07-10 1:24 ` Daniel Sangorrin
2019-07-10 1:24 ` [cip-dev] [cip-kernel-sec][RESEND 3/6] report_affected: fix code when branches are specified Daniel Sangorrin
` (4 subsequent siblings)
6 siblings, 0 replies; 10+ messages in thread
From: Daniel Sangorrin @ 2019-07-10 1:24 UTC (permalink / raw)
To: cip-dev
Helper script that prepares the local git repository
with the configured remote branches. Expert developers
and kernel maintainers will probably have their own
worktrees but for new users or a quickstart, this
script should be helpful.
Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
README.md | 5 +++
conf/remotes.yml | 3 ++
scripts/import_stable.py | 8 ++---
scripts/kernel_sec/branch.py | 10 ++++++
scripts/prepare_remotes.py | 62 ++++++++++++++++++++++++++++++++++++
5 files changed, 82 insertions(+), 6 deletions(-)
create mode 100755 scripts/prepare_remotes.py
diff --git a/README.md b/README.md
index 4c5808f..576cc75 100644
--- a/README.md
+++ b/README.md
@@ -23,6 +23,10 @@ this is assumed to be in `../kernel`, with remotes configured in
stable and cip repositories. These can be overridden by command-line options
or configuration (`~/.config/kernel-sec/remotes.yml`).
+* `scripts/prepare_remotes.py` - creates the local git repository
+and adds all configured remotes. You may prefer to skip this script
+and configure the repository by hand.
+
* `scripts/import_debian.py` - import information from Debian's
`kernel_sec` project. It includes all issues that Debian considers
active or that are already tracked here.
@@ -81,6 +85,7 @@ with the keys:
branch from this remote.
* `git_name`: (optional) The name actually used for this git
remote, if it's different from the default.
+* `git_repo_url`: URL of the remote git repository.
## Contributions
diff --git a/conf/remotes.yml b/conf/remotes.yml
index 51c523d..cfaa35f 100644
--- a/conf/remotes.yml
+++ b/conf/remotes.yml
@@ -1,6 +1,9 @@
torvalds:
commit_url_prefix: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=
+ git_repo_url: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
stable:
commit_url_prefix: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit?id=
+ git_repo_url: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
cip:
commit_url_prefix: https://git.kernel.org/pub/scm/linux/kernel/git/cip/linux-cip.git/commit?id=
+ git_repo_url: https://git.kernel.org/pub/scm/linux/kernel/git/cip/linux-cip.git
diff --git a/scripts/import_stable.py b/scripts/import_stable.py
index 26fcc2f..194219d 100755
--- a/scripts/import_stable.py
+++ b/scripts/import_stable.py
@@ -31,11 +31,6 @@ BACKPORT_COMMIT_BOTTOM_RE = re.compile(
.format(**RE_USE))
-def update(git_repo, remote_name):
- subprocess.check_call(['git', 'remote', 'update', remote_name],
- cwd=git_repo)
-
-
def get_backports(git_repo, remotes, branches, debug=False):
backports = {}
@@ -140,7 +135,8 @@ def main(git_repo, remotes, debug=False):
remote_names = set(branch['git_remote'] for branch in branches)
for remote_name in remote_names:
- update(git_repo, remotes[remote_name]['git_name'])
+ kernel_sec.branch.remote_update(
+ git_repo, remotes[remote_name]['git_name'])
backports = get_backports(git_repo, remotes, branches, debug)
c_b_map = kernel_sec.branch.CommitBranchMap(git_repo, remotes, branches)
diff --git a/scripts/kernel_sec/branch.py b/scripts/kernel_sec/branch.py
index a138b96..0023497 100644
--- a/scripts/kernel_sec/branch.py
+++ b/scripts/kernel_sec/branch.py
@@ -223,6 +223,16 @@ def get_remotes(mappings, mainline=None, stable=None):
return remotes
+def remote_update(git_repo, remote_name):
+ subprocess.check_call(['git', 'remote', 'update', remote_name],
+ cwd=git_repo)
+
+
+def remote_add(git_repo, remote_name, remote_url):
+ subprocess.check_call(['git', 'remote', 'add', remote_name, remote_url],
+ cwd=git_repo)
+
+
def check_git_repo(git_repo, remotes):
if not os.path.isdir(git_repo):
msg = "directory %r not present" % git_repo
diff --git a/scripts/prepare_remotes.py b/scripts/prepare_remotes.py
new file mode 100755
index 0000000..59310fc
--- /dev/null
+++ b/scripts/prepare_remotes.py
@@ -0,0 +1,62 @@
+#!/usr/bin/python3
+
+# Copyright 2019 Toshiba corp.
+# Based on import_stable.py by Codethink Ltd.
+#
+# This script is distributed under the terms and conditions of the GNU General
+# Public License, Version 3 or later. See http://www.gnu.org/copyleft/gpl.html
+# for details.
+
+# Helper script that prepares the local git repository with the configured
+# remote branches
+
+import argparse
+import os
+import subprocess
+
+import kernel_sec.branch
+
+
+def main(git_repo, remotes):
+ if os.path.isdir(git_repo):
+ msg = "directory %r already exists" % git_repo
+ raise argparse.ArgumentError(None, msg)
+ else:
+ os.mkdir(git_repo)
+ subprocess.check_call(['git', 'init', '.'], cwd=git_repo)
+
+ for key in remotes.keys():
+ remote = remotes[key] # __getitem__ will add git_name
+ kernel_sec.branch.remote_add(
+ git_repo, remote['git_name'], remote['git_repo_url'])
+ kernel_sec.branch.remote_update(git_repo, remote['git_name'])
+
+ # self-check
+ kernel_sec.branch.check_git_repo(git_repo, remotes)
+
+
+if __name__ == '__main__':
+ parser = argparse.ArgumentParser(
+ description=('Prepare local git repository with configured remotes.'))
+ parser.add_argument('--git-repo',
+ dest='git_repo', default='../kernel',
+ help=('local git repository location '
+ '(default: ../kernel)'),
+ metavar='DIRECTORY')
+ parser.add_argument('--remote-name',
+ dest='remote_name', action='append', default=[],
+ help='git remote name mappings, e.g. stable:korg-stable',
+ metavar='NAME:OTHER-NAME')
+ parser.add_argument('--mainline-remote',
+ dest='mainline_remote_name',
+ help="git remote name to use instead of 'torvalds'",
+ metavar='OTHER-NAME')
+ parser.add_argument('--stable-remote',
+ dest='stable_remote_name',
+ help="git remote name to use instead of 'stable'",
+ metavar='OTHER-NAME')
+ args = parser.parse_args()
+ remotes = kernel_sec.branch.get_remotes(args.remote_name,
+ mainline=args.mainline_remote_name,
+ stable=args.stable_remote_name)
+ main(args.git_repo, remotes)
--
2.17.1
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [cip-dev] [cip-kernel-sec][RESEND 3/6] report_affected: fix code when branches are specified
2019-07-10 1:24 [cip-dev] Add support for cip branches and tags Daniel Sangorrin
2019-07-10 1:24 ` [cip-dev] [cip-kernel-sec][RESEND 1/6] check_git_repo: add checks to the local repository Daniel Sangorrin
2019-07-10 1:24 ` [cip-dev] [cip-kernel-sec][RESEND 2/6] prepare_remotes: helper script to prepare local repo Daniel Sangorrin
@ 2019-07-10 1:24 ` Daniel Sangorrin
2019-07-10 1:24 ` [cip-dev] [cip-kernel-sec][RESEND 4/6] report_affected: add support for reporting on tags Daniel Sangorrin
` (3 subsequent siblings)
6 siblings, 0 replies; 10+ messages in thread
From: Daniel Sangorrin @ 2019-07-10 1:24 UTC (permalink / raw)
To: cip-dev
The previous code could not handle branches with names
other than stable branch names. For example, passing
"linux-4.4.y-cip" as a branch would return an error.
Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
scripts/kernel_sec/branch.py | 8 --------
scripts/report_affected.py | 23 +++++++++++++++++------
2 files changed, 17 insertions(+), 14 deletions(-)
diff --git a/scripts/kernel_sec/branch.py b/scripts/kernel_sec/branch.py
index 0023497..ef88b54 100644
--- a/scripts/kernel_sec/branch.py
+++ b/scripts/kernel_sec/branch.py
@@ -21,9 +21,6 @@ import yaml
from . import version
-_STABLE_BRANCH_RE = re.compile(r'^linux-([\d.]+)\.y$')
-
-
def get_base_ver_stable_branch(base_ver):
branch_name = 'linux-%s.y' % base_ver
return {
@@ -34,11 +31,6 @@ def get_base_ver_stable_branch(base_ver):
}
-def get_stable_branch(branch_name):
- match = _STABLE_BRANCH_RE.match(branch_name)
- return match and get_base_ver_stable_branch(match.group(1))
-
-
def _extract_live_stable_branches(doc):
xhtml_ns = 'http://www.w3.org/1999/xhtml'
ns = {'html': xhtml_ns}
diff --git a/scripts/report_affected.py b/scripts/report_affected.py
index d2f1f22..0ac27f6 100755
--- a/scripts/report_affected.py
+++ b/scripts/report_affected.py
@@ -18,14 +18,25 @@ import kernel_sec.version
def main(git_repo, remotes,
only_fixed_upstream, include_ignored, *branch_names):
+ live_branches = kernel_sec.branch.get_live_branches()
if branch_names:
- # Support stable release strings as shorthand for stable branches
- branches = [kernel_sec.branch.get_base_ver_stable_branch(name)
- if name[0].isdigit()
- else kernel_sec.branch.get_stable_branch(name)
- for name in branch_names]
+ branches = []
+ for branch_name in branch_names:
+ if branch_name[0].isdigit():
+ # 4.4 is mapped to linux-4.4.y
+ name = 'linux-%s.y' % branch_name
+ else:
+ name = branch_name
+
+ for branch in live_branches:
+ if branch['short_name'] == name:
+ branches.append(branch)
+ break
+ else:
+ msg = "Branch %s could not be found" % branch_name
+ raise argparse.ArgumentError(None, msg)
else:
- branches = kernel_sec.branch.get_live_branches()
+ branches = live_branches
if only_fixed_upstream:
branches = [branch for branch in branches
if branch['short_name'] != 'mainline']
--
2.17.1
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [cip-dev] [cip-kernel-sec][RESEND 4/6] report_affected: add support for reporting on tags
2019-07-10 1:24 [cip-dev] Add support for cip branches and tags Daniel Sangorrin
` (2 preceding siblings ...)
2019-07-10 1:24 ` [cip-dev] [cip-kernel-sec][RESEND 3/6] report_affected: fix code when branches are specified Daniel Sangorrin
@ 2019-07-10 1:24 ` Daniel Sangorrin
2019-07-10 14:40 ` Ben Hutchings
2019-07-10 1:24 ` [cip-dev] [cip-kernel-sec][RESEND 5/6] pep8: fix pep8-related errors such as too long lines Daniel Sangorrin
` (2 subsequent siblings)
6 siblings, 1 reply; 10+ messages in thread
From: Daniel Sangorrin @ 2019-07-10 1:24 UTC (permalink / raw)
To: cip-dev
Reporting on tags is useful for product engineers that
have shipped a kernel with a specific tag and need to know
which issues affect their product after some time.
Examples:
$ ./scripts/report_affected.py linux-4.14.y linux-4.4.y:v4.4.107 v4.4.181-cip33
$ cd ../kernel
$ git tag myproduct-v1 0f13d9b4d0efa9e87381717c113df57718bc92d6
$ cd ../cip-kernel-sec
$ ./scripts/report_affected.py linux-4.19.y-cip:myproduct-v1 v4.19.50-cip3
Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
conf/branches.yml | 2 +
scripts/kernel_sec/branch.py | 4 +-
scripts/report_affected.py | 77 +++++++++++++++++++++++++++++++-----
3 files changed, 72 insertions(+), 11 deletions(-)
diff --git a/conf/branches.yml b/conf/branches.yml
index 2ed9db6..8197596 100644
--- a/conf/branches.yml
+++ b/conf/branches.yml
@@ -2,7 +2,9 @@
base_ver: "4.4"
git_remote: cip
git_name: linux-4.4.y-cip
+ tag_regexp: '^v4\.4\.\d+-cip\d+$'
- short_name: linux-4.19.y-cip
base_ver: "4.19"
git_remote: cip
git_name: linux-4.19.y-cip
+ tag_regexp: '^v4\.19\.\d+-cip\d+$'
diff --git a/scripts/kernel_sec/branch.py b/scripts/kernel_sec/branch.py
index ef88b54..96f1cab 100644
--- a/scripts/kernel_sec/branch.py
+++ b/scripts/kernel_sec/branch.py
@@ -141,7 +141,7 @@ def get_sort_key(branch):
return version.get_sort_key(base_ver)
-def _get_commits(git_repo, end, start=None):
+def iter_rev_list(git_repo, end, start=None):
if start:
list_expr = '%s..%s' % (start, end)
else:
@@ -170,7 +170,7 @@ class CommitBranchMap:
branch['git_name'])
else:
end = 'v' + branch['base_ver']
- for commit in _get_commits(git_repo, end, start):
+ for commit in iter_rev_list(git_repo, end, start):
self._commit_sort_key[commit] \
= self._branch_sort_key[branch_name]
start = end
diff --git a/scripts/report_affected.py b/scripts/report_affected.py
index 0ac27f6..7ec4af7 100755
--- a/scripts/report_affected.py
+++ b/scripts/report_affected.py
@@ -9,7 +9,9 @@
# Report issues affecting each stable branch.
import argparse
+import copy
import subprocess
+import re
import kernel_sec.branch
import kernel_sec.issue
@@ -22,15 +24,47 @@ def main(git_repo, remotes,
if branch_names:
branches = []
for branch_name in branch_names:
+ tag = None
if branch_name[0].isdigit():
# 4.4 is mapped to linux-4.4.y
name = 'linux-%s.y' % branch_name
+ elif branch_name[0] == 'v':
+ # an official tag, e.g. v4.4.92-cip11
+ # infer branch from tag (regexp's must be specific)
+ for branch in live_branches:
+ if 'tag_regexp' in branch:
+ # predefined in conf/branches.yml
+ tag_regexp = branch['tag_regexp']
+ elif branch['git_remote'] == 'stable':
+ # stable format, e.g. v4.19.12
+ esc_base_ver = branch['base_ver'].replace('.', '\.')
+ tag_regexp = r'(^v%s$|^v%s\.\d+$)' % (
+ esc_base_ver, esc_base_ver)
+ else:
+ # no tag_regexp defined, or mainline
+ continue
+
+ if re.match(tag_regexp, branch_name):
+ tag = branch_name
+ name = branch['short_name']
+ break
+ else:
+ raise ValueError('Failed to match tag %r' % branch_name)
+ elif ':' in branch_name:
+ # a possibly custom tag, e.g. linux-4.19.y-cip:myproduct-v1
+ name_tuple = tuple(branch_name.split(':'))
+ name = name_tuple[0]
+ tag = name_tuple[1]
else:
name = branch_name
for branch in live_branches:
if branch['short_name'] == name:
- branches.append(branch)
+ # there could be multiple tags for the same branch
+ branch_copy = copy.deepcopy(branch)
+ if tag:
+ branch_copy['tag'] = tag
+ branches.append(branch_copy)
break
else:
msg = "Branch %s could not be found" % branch_name
@@ -45,6 +79,18 @@ def main(git_repo, remotes,
c_b_map = kernel_sec.branch.CommitBranchMap(git_repo, remotes, branches)
+ # cache tag commits and set full_name to show the tag
+ tag_commits = {}
+ for branch in branches:
+ if 'tag' in branch:
+ start = 'v' + branch['base_ver']
+ end = branch['tag']
+ tag_commits[end] = set(
+ kernel_sec.branch.iter_rev_list(git_repo, end, start))
+ branch['full_name'] = ':'.join([branch['short_name'], end])
+ else:
+ branch['full_name'] = branch['short_name']
+
branch_issues = {}
issues = set(kernel_sec.issue.get_list())
@@ -65,15 +111,26 @@ def main(git_repo, remotes,
if not include_ignored and ignore.get(branch_name):
continue
+ # Check if the branch is affected. If not and the issue was fixed
+ # on that branch, then make sure the tag contains that fix
if kernel_sec.issue.affects_branch(
issue, branch, c_b_map.is_commit_in_branch):
- branch_issues.setdefault(branch_name, []).append(cve_id)
+ branch_issues.setdefault(
+ branch['full_name'], []).append(cve_id)
+ elif 'tag' in branch and fixed:
+ if fixed.get(branch_name, 'never') == 'never':
+ continue
+ for commit in fixed[branch_name]:
+ if commit not in tag_commits[branch['tag']]:
+ branch_issues.setdefault(
+ branch['full_name'], []).append(cve_id)
+ break
for branch in branches:
- branch_name = branch['short_name']
- print('%s:' % branch_name,
- *sorted(branch_issues.get(branch_name, []),
- key=kernel_sec.issue.get_id_sort_key))
+ sorted_cve_ids = sorted(
+ branch_issues.get(branch['full_name'], []),
+ key=kernel_sec.issue.get_id_sort_key)
+ print('%s:' % branch['full_name'], *sorted_cve_ids)
if __name__ == '__main__':
@@ -104,9 +161,11 @@ if __name__ == '__main__':
help='include issues that have been marked as ignored')
parser.add_argument('branches',
nargs='*',
- help=('specific branch to report on '
- '(default: all active branches)'),
- metavar='BRANCH')
+ help=('specific branch[:tag] or stable tag to '
+ 'report on (default: all active branches). '
+ 'e.g. linux-4.14.y linux-4.4.y:v4.4.107 '
+ 'v4.4.181-cip33 linux-4.19.y-cip:myproduct-v33'),
+ metavar='[BRANCH[:TAG]|TAG]')
args = parser.parse_args()
remotes = kernel_sec.branch.get_remotes(args.remote_name,
mainline=args.mainline_remote_name,
--
2.17.1
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [cip-dev] [cip-kernel-sec][RESEND 4/6] report_affected: add support for reporting on tags
2019-07-10 1:24 ` [cip-dev] [cip-kernel-sec][RESEND 4/6] report_affected: add support for reporting on tags Daniel Sangorrin
@ 2019-07-10 14:40 ` Ben Hutchings
2019-07-11 4:50 ` daniel.sangorrin at toshiba.co.jp
0 siblings, 1 reply; 10+ messages in thread
From: Ben Hutchings @ 2019-07-10 14:40 UTC (permalink / raw)
To: cip-dev
On Wed, 2019-07-10 at 10:24 +0900, Daniel Sangorrin wrote:
[...]
> --- a/scripts/report_affected.py
> +++ b/scripts/report_affected.py
[...]
> @@ -22,15 +24,47 @@ def main(git_repo, remotes,
> if branch_names:
> branches = []
> for branch_name in branch_names:
> + tag = None
> if branch_name[0].isdigit():
> # 4.4 is mapped to linux-4.4.y
> name = 'linux-%s.y' % branch_name
> + elif branch_name[0] == 'v':
> + # an official tag, e.g. v4.4.92-cip11
> + # infer branch from tag (regexp's must be specific)
> + for branch in live_branches:
> + if 'tag_regexp' in branch:
> + # predefined in conf/branches.yml
> + tag_regexp = branch['tag_regexp']
> + elif branch['git_remote'] == 'stable':
> + # stable format, e.g. v4.19.12
> + esc_base_ver = branch['base_ver'].replace('.', '\.')
This happens to work now, but '\.' is an unrecognised escape sequence
which is deprecated. You presumably meant r'\.', but it might be
clearer to use re.escape().
> + tag_regexp = r'(^v%s$|^v%s\.\d+$)' % (
> + esc_base_ver, esc_base_ver)
I also expected that you would set tag_regexp for stable branches in
the branch module along with all their other fields. Then there's no
need to handle them specially here.
> + else:
> + # no tag_regexp defined, or mainline
> + continue
> +
> + if re.match(tag_regexp, branch_name):
> + tag = branch_name
> + name = branch['short_name']
> + break
> + else:
> + raise ValueError('Failed to match tag %r' % branch_name)
> + elif ':' in branch_name:
> + # a possibly custom tag, e.g. linux-4.19.y-cip:myproduct-v1
> + name_tuple = tuple(branch_name.split(':'))
> + name = name_tuple[0]
> + tag = name_tuple[1]
[...]
You really can do simply:
name, tag = branch_name.split(':', 1)
(Tuple assignment only requires an iterable, not specifically a tuple,
on the right hand side.) So please use that.
Ben.
--
Ben Hutchings, Software Developer Codethink Ltd
https://www.codethink.co.uk/ Dale House, 35 Dale Street
Manchester, M1 2HF, United Kingdom
^ permalink raw reply [flat|nested] 10+ messages in thread
* [cip-dev] [cip-kernel-sec][RESEND 4/6] report_affected: add support for reporting on tags
2019-07-10 14:40 ` Ben Hutchings
@ 2019-07-11 4:50 ` daniel.sangorrin at toshiba.co.jp
0 siblings, 0 replies; 10+ messages in thread
From: daniel.sangorrin at toshiba.co.jp @ 2019-07-11 4:50 UTC (permalink / raw)
To: cip-dev
> From: Ben Hutchings <ben.hutchings@codethink.co.uk>
[...]
> > + if 'tag_regexp' in branch:
> > + # predefined in conf/branches.yml
> > + tag_regexp = branch['tag_regexp']
> > + elif branch['git_remote'] == 'stable':
> > + # stable format, e.g. v4.19.12
> > + esc_base_ver = branch['base_ver'].replace('.', '\.')
>
> This happens to work now, but '\.' is an unrecognised escape sequence
> which is deprecated. You presumably meant r'\.', but it might be
> clearer to use re.escape().
Thanks, you are right. I have used re.escape('.'). I didn't know this function, it's really useful not having to remember how to escape characters (I figured out I could have used \\.).
I think that the YAML strings in conf/branches.yml are being read as if they were raw strings, but if there is a problem with them let me know.
>
> > + tag_regexp = r'(^v%s$|^v%s\.\d+$)' % (
> > + esc_base_ver, esc_base_ver)
>
> I also expected that you would set tag_regexp for stable branches in
> the branch module along with all their other fields. Then there's no
> need to handle them specially here.
Ah sorry about that. I have moved that code to branch.py now.
>
> > + else:
> > + # no tag_regexp defined, or mainline
> > + continue
> > +
> > + if re.match(tag_regexp, branch_name):
> > + tag = branch_name
> > + name = branch['short_name']
> > + break
> > + else:
> > + raise ValueError('Failed to match tag %r' % branch_name)
> > + elif ':' in branch_name:
> > + # a possibly custom tag, e.g. linux-4.19.y-cip:myproduct-v1
> > + name_tuple = tuple(branch_name.split(':'))
> > + name = name_tuple[0]
> > + tag = name_tuple[1]
> [...]
>
> You really can do simply:
>
> name, tag = branch_name.split(':', 1)
>
> (Tuple assignment only requires an iterable, not specifically a tuple,
> on the right hand side.) So please use that.
Nice trick. I fixed the code.
Thanks,
Daniel
>
> Ben.
>
> --
> Ben Hutchings, Software Developer Codethink Ltd
> https://www.codethink.co.uk/ Dale House, 35 Dale Street
> Manchester, M1 2HF, United Kingdom
^ permalink raw reply [flat|nested] 10+ messages in thread
* [cip-dev] [cip-kernel-sec][RESEND 5/6] pep8: fix pep8-related errors such as too long lines
2019-07-10 1:24 [cip-dev] Add support for cip branches and tags Daniel Sangorrin
` (3 preceding siblings ...)
2019-07-10 1:24 ` [cip-dev] [cip-kernel-sec][RESEND 4/6] report_affected: add support for reporting on tags Daniel Sangorrin
@ 2019-07-10 1:24 ` Daniel Sangorrin
2019-07-10 1:24 ` [cip-dev] [cip-kernel-sec][RESEND 6/6] report_affected: add show-description option Daniel Sangorrin
2019-07-10 15:02 ` [cip-dev] Add support for cip branches and tags Ben Hutchings
6 siblings, 0 replies; 10+ messages in thread
From: Daniel Sangorrin @ 2019-07-10 1:24 UTC (permalink / raw)
To: cip-dev
These were distracting when checking new code.
Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
scripts/import_stable.py | 8 +++++---
scripts/kernel_sec/branch.py | 4 ++--
scripts/prepare_remotes.py | 2 +-
scripts/report_affected.py | 2 +-
scripts/webview.py | 2 +-
5 files changed, 10 insertions(+), 8 deletions(-)
diff --git a/scripts/import_stable.py b/scripts/import_stable.py
index 194219d..b4d8e0a 100755
--- a/scripts/import_stable.py
+++ b/scripts/import_stable.py
@@ -64,7 +64,8 @@ def get_backports(git_repo, remotes, branches, debug=False):
backports.setdefault(mainline_commit, {})[branch_name] \
= stable_commit
if line.strip() != '':
- commit_re = BACKPORT_COMMIT_BOTTOM_RE # next line is not top
+ # next line is not top
+ commit_re = BACKPORT_COMMIT_BOTTOM_RE
return backports
@@ -120,7 +121,8 @@ def add_backports(branches, c_b_map, issue_commits, all_backports,
if debug_context:
print('%s/%s: recording commits' %
(debug_context, branch_name))
- issue_commits.setdefault(branch_name, []).extend(branch_commits)
+ issue_commits.setdefault(
+ branch_name, []).extend(branch_commits)
changed = True
else:
if debug_context:
@@ -169,7 +171,7 @@ if __name__ == '__main__':
metavar='DIRECTORY')
parser.add_argument('--remote-name',
dest='remote_name', action='append', default=[],
- help='git remote name mappings, e.g. stable:korg-stable',
+ help='git remote name mappings, e.g. stable:mystable',
metavar='NAME:OTHER-NAME')
parser.add_argument('--mainline-remote',
dest='mainline_remote_name',
diff --git a/scripts/kernel_sec/branch.py b/scripts/kernel_sec/branch.py
index 96f1cab..52f3b42 100644
--- a/scripts/kernel_sec/branch.py
+++ b/scripts/kernel_sec/branch.py
@@ -62,8 +62,8 @@ def _extract_live_stable_branches(doc):
if match:
version = match.group(1)
eol = match.group(2) is not None
- if branch_type not in ['mainline', 'stable', 'longterm', 'linux-next'] \
- or version is None:
+ if branch_type not in ['mainline', 'stable', 'longterm',
+ 'linux-next'] or version is None:
raise ValueError('failed to parse releases row text %r' % row_text)
# Filter out irrelevant branches
diff --git a/scripts/prepare_remotes.py b/scripts/prepare_remotes.py
index 59310fc..ab0db24 100755
--- a/scripts/prepare_remotes.py
+++ b/scripts/prepare_remotes.py
@@ -45,7 +45,7 @@ if __name__ == '__main__':
metavar='DIRECTORY')
parser.add_argument('--remote-name',
dest='remote_name', action='append', default=[],
- help='git remote name mappings, e.g. stable:korg-stable',
+ help='git remote name mappings, e.g. stable:mystable',
metavar='NAME:OTHER-NAME')
parser.add_argument('--mainline-remote',
dest='mainline_remote_name',
diff --git a/scripts/report_affected.py b/scripts/report_affected.py
index 7ec4af7..a97986c 100755
--- a/scripts/report_affected.py
+++ b/scripts/report_affected.py
@@ -143,7 +143,7 @@ if __name__ == '__main__':
metavar='DIRECTORY')
parser.add_argument('--remote-name',
dest='remote_name', action='append', default=[],
- help='git remote name mappings, e.g. stable:korg-stable',
+ help='git remote name mappings, e.g. stable:mystable',
metavar='NAME:OTHER-NAME')
parser.add_argument('--mainline-remote',
dest='mainline_remote_name',
diff --git a/scripts/webview.py b/scripts/webview.py
index 52a7b15..cf54948 100755
--- a/scripts/webview.py
+++ b/scripts/webview.py
@@ -205,7 +205,7 @@ if __name__ == '__main__':
metavar='DIRECTORY')
parser.add_argument('--remote-name',
dest='remote_name', action='append', default=[],
- help='git remote name mappings, e.g. stable:korg-stable',
+ help='git remote name mappings, e.g. stable:mystable',
metavar='NAME:OTHER-NAME')
parser.add_argument('--mainline-remote',
dest='mainline_remote_name',
--
2.17.1
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [cip-dev] [cip-kernel-sec][RESEND 6/6] report_affected: add show-description option
2019-07-10 1:24 [cip-dev] Add support for cip branches and tags Daniel Sangorrin
` (4 preceding siblings ...)
2019-07-10 1:24 ` [cip-dev] [cip-kernel-sec][RESEND 5/6] pep8: fix pep8-related errors such as too long lines Daniel Sangorrin
@ 2019-07-10 1:24 ` Daniel Sangorrin
2019-07-10 15:02 ` [cip-dev] Add support for cip branches and tags Ben Hutchings
6 siblings, 0 replies; 10+ messages in thread
From: Daniel Sangorrin @ 2019-07-10 1:24 UTC (permalink / raw)
To: cip-dev
Rather than looking up each issue file, I would like
to have an overview of what each CVE ID means.
Example:
$ ./scripts/report_affected.py --show-description linux-4.4.y-cip
Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
scripts/report_affected.py | 19 ++++++++++++++-----
1 file changed, 14 insertions(+), 5 deletions(-)
diff --git a/scripts/report_affected.py b/scripts/report_affected.py
index a97986c..b7a2678 100755
--- a/scripts/report_affected.py
+++ b/scripts/report_affected.py
@@ -18,8 +18,8 @@ import kernel_sec.issue
import kernel_sec.version
-def main(git_repo, remotes,
- only_fixed_upstream, include_ignored, *branch_names):
+def main(git_repo, remotes, only_fixed_upstream,
+ include_ignored, show_description, *branch_names):
live_branches = kernel_sec.branch.get_live_branches()
if branch_names:
branches = []
@@ -130,7 +130,13 @@ def main(git_repo, remotes,
sorted_cve_ids = sorted(
branch_issues.get(branch['full_name'], []),
key=kernel_sec.issue.get_id_sort_key)
- print('%s:' % branch['full_name'], *sorted_cve_ids)
+ if show_description:
+ print('%s:' % branch['full_name'])
+ for cve_id in sorted_cve_ids:
+ print(cve_id, '=>',
+ kernel_sec.issue.load(cve_id).get('description', 'None'))
+ else:
+ print('%s:' % branch['full_name'], *sorted_cve_ids)
if __name__ == '__main__':
@@ -159,6 +165,9 @@ if __name__ == '__main__':
parser.add_argument('--include-ignored',
action='store_true',
help='include issues that have been marked as ignored')
+ parser.add_argument('--show-description',
+ action='store_true',
+ help='show the issue description')
parser.add_argument('branches',
nargs='*',
help=('specific branch[:tag] or stable tag to '
@@ -171,5 +180,5 @@ if __name__ == '__main__':
mainline=args.mainline_remote_name,
stable=args.stable_remote_name)
kernel_sec.branch.check_git_repo(args.git_repo, remotes)
- main(args.git_repo, remotes,
- args.only_fixed_upstream, args.include_ignored, *args.branches)
+ main(args.git_repo, remotes, args.only_fixed_upstream,
+ args.include_ignored, args.show_description, *args.branches)
--
2.17.1
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [cip-dev] Add support for cip branches and tags
2019-07-10 1:24 [cip-dev] Add support for cip branches and tags Daniel Sangorrin
` (5 preceding siblings ...)
2019-07-10 1:24 ` [cip-dev] [cip-kernel-sec][RESEND 6/6] report_affected: add show-description option Daniel Sangorrin
@ 2019-07-10 15:02 ` Ben Hutchings
6 siblings, 0 replies; 10+ messages in thread
From: Ben Hutchings @ 2019-07-10 15:02 UTC (permalink / raw)
To: cip-dev
On Wed, 2019-07-10 at 10:24 +0900, Daniel Sangorrin wrote:
> Hello Ben,
>
> Thank you for the detailed review, and sorry for the delay.
> I have modified my patch series taking your comments into account.
> Note: I will reply to your comments separately
>
> I re-send to you the whole series to make sure that patches
> apply properly.
I've applied patches 1, 2, 3, and 5 (as it didn't depend on 4). Please
re-send the remaining two with my requested changes.
Ben.
--
Ben Hutchings, Software Developer Codethink Ltd
https://www.codethink.co.uk/ Dale House, 35 Dale Street
Manchester, M1 2HF, United Kingdom
^ permalink raw reply [flat|nested] 10+ messages in thread