All of lore.kernel.org
 help / color / mirror / Atom feed
* [cip-dev] Add support for cip branches and tags
@ 2019-07-10  1:24 Daniel Sangorrin
  2019-07-10  1:24 ` [cip-dev] [cip-kernel-sec][RESEND 1/6] check_git_repo: add checks to the local repository Daniel Sangorrin
                   ` (6 more replies)
  0 siblings, 7 replies; 10+ messages in thread
From: Daniel Sangorrin @ 2019-07-10  1:24 UTC (permalink / raw)
  To: cip-dev

Hello Ben,

Thank you for the detailed review, and sorry for the delay.
I have modified my patch series taking your comments into account.
Note: I will reply to your comments separately

I re-send to you the whole series to make sure that patches
apply properly.

[cip-kernel-sec][RESEND 1/6] check_git_repo: add checks to the local
[cip-kernel-sec][RESEND 2/6] prepare_remotes: helper script to
[cip-kernel-sec][RESEND 3/6] report_affected: fix code when branches
[cip-kernel-sec][RESEND 4/6] report_affected: add support for
[cip-kernel-sec][RESEND 5/6] pep8: fix pep8-related errors such as
[cip-kernel-sec][RESEND 6/6] report_affected: add show-description

Thanks,
Daniel

^ permalink raw reply	[flat|nested] 10+ messages in thread

* [cip-dev] [cip-kernel-sec][RESEND 1/6] check_git_repo: add checks to the local repository
  2019-07-10  1:24 [cip-dev] Add support for cip branches and tags Daniel Sangorrin
@ 2019-07-10  1:24 ` Daniel Sangorrin
  2019-07-10  1:24 ` [cip-dev] [cip-kernel-sec][RESEND 2/6] prepare_remotes: helper script to prepare local repo Daniel Sangorrin
                   ` (5 subsequent siblings)
  6 siblings, 0 replies; 10+ messages in thread
From: Daniel Sangorrin @ 2019-07-10  1:24 UTC (permalink / raw)
  To: cip-dev

Add checks to make sure that the local repository exists
and has the configured remotes in place.

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 scripts/import_stable.py     |  1 +
 scripts/kernel_sec/branch.py | 21 +++++++++++++++++++++
 scripts/report_affected.py   |  1 +
 scripts/webview.py           |  1 +
 4 files changed, 24 insertions(+)

diff --git a/scripts/import_stable.py b/scripts/import_stable.py
index 2c88b07..26fcc2f 100755
--- a/scripts/import_stable.py
+++ b/scripts/import_stable.py
@@ -190,4 +190,5 @@ if __name__ == '__main__':
     remotes = kernel_sec.branch.get_remotes(args.remote_name,
                                             mainline=args.mainline_remote_name,
                                             stable=args.stable_remote_name)
+    kernel_sec.branch.check_git_repo(args.git_repo, remotes)
     main(args.git_repo, remotes, args.debug)
diff --git a/scripts/kernel_sec/branch.py b/scripts/kernel_sec/branch.py
index 10f3339..a138b96 100644
--- a/scripts/kernel_sec/branch.py
+++ b/scripts/kernel_sec/branch.py
@@ -4,10 +4,12 @@
 # Public License, Version 3 or later. See http://www.gnu.org/copyleft/gpl.html
 # for details.
 
+import argparse
 import io
 import os
 import re
 import subprocess
+import sys
 import time
 import urllib.error
 import urllib.request
@@ -219,3 +221,22 @@ def get_remotes(mappings, mainline=None, stable=None):
     if stable:
         remotes['stable']['git_name'] = stable
     return remotes
+
+
+def check_git_repo(git_repo, remotes):
+    if not os.path.isdir(git_repo):
+        msg = "directory %r not present" % git_repo
+        raise argparse.ArgumentError(None, msg)
+    # .git could be a regular file (worktrees) or a directory
+    if not os.path.exists(os.path.join(git_repo, '.git')):
+        msg = "directory %r is not a git repository" % git_repo
+        raise argparse.ArgumentError(None, msg)
+
+    current_remotes = subprocess.check_output(
+        ['git', 'remote', 'show'], cwd=git_repo).decode(
+            sys.stdout.encoding).strip().split('\n')
+    for key in remotes.keys():
+        remote = remotes[key]  # __getitem__ will add git_name
+        if remote['git_name'] not in current_remotes:
+            msg = "remote %r not in git repository" % remote['git_name']
+            raise argparse.ArgumentError(None, msg)
diff --git a/scripts/report_affected.py b/scripts/report_affected.py
index 879c021..d2f1f22 100755
--- a/scripts/report_affected.py
+++ b/scripts/report_affected.py
@@ -100,5 +100,6 @@ if __name__ == '__main__':
     remotes = kernel_sec.branch.get_remotes(args.remote_name,
                                             mainline=args.mainline_remote_name,
                                             stable=args.stable_remote_name)
+    kernel_sec.branch.check_git_repo(args.git_repo, remotes)
     main(args.git_repo, remotes,
          args.only_fixed_upstream, args.include_ignored, *args.branches)
diff --git a/scripts/webview.py b/scripts/webview.py
index a3a643b..52a7b15 100755
--- a/scripts/webview.py
+++ b/scripts/webview.py
@@ -219,6 +219,7 @@ if __name__ == '__main__':
     remotes = kernel_sec.branch.get_remotes(args.remote_name,
                                             mainline=args.mainline_remote_name,
                                             stable=args.stable_remote_name)
+    kernel_sec.branch.check_git_repo(args.git_repo, remotes)
 
     conf = {
         '/static/style.css': {
-- 
2.17.1

^ permalink raw reply related	[flat|nested] 10+ messages in thread

* [cip-dev] [cip-kernel-sec][RESEND 2/6] prepare_remotes: helper script to prepare local repo
  2019-07-10  1:24 [cip-dev] Add support for cip branches and tags Daniel Sangorrin
  2019-07-10  1:24 ` [cip-dev] [cip-kernel-sec][RESEND 1/6] check_git_repo: add checks to the local repository Daniel Sangorrin
@ 2019-07-10  1:24 ` Daniel Sangorrin
  2019-07-10  1:24 ` [cip-dev] [cip-kernel-sec][RESEND 3/6] report_affected: fix code when branches are specified Daniel Sangorrin
                   ` (4 subsequent siblings)
  6 siblings, 0 replies; 10+ messages in thread
From: Daniel Sangorrin @ 2019-07-10  1:24 UTC (permalink / raw)
  To: cip-dev

Helper script that prepares the local git repository
with the configured remote branches. Expert developers
and kernel maintainers will probably have their own
worktrees but for new users or a quickstart, this
script should be helpful.

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 README.md                    |  5 +++
 conf/remotes.yml             |  3 ++
 scripts/import_stable.py     |  8 ++---
 scripts/kernel_sec/branch.py | 10 ++++++
 scripts/prepare_remotes.py   | 62 ++++++++++++++++++++++++++++++++++++
 5 files changed, 82 insertions(+), 6 deletions(-)
 create mode 100755 scripts/prepare_remotes.py

diff --git a/README.md b/README.md
index 4c5808f..576cc75 100644
--- a/README.md
+++ b/README.md
@@ -23,6 +23,10 @@ this is assumed to be in `../kernel`, with remotes configured in
 stable and cip repositories. These can be overridden by command-line options
 or configuration (`~/.config/kernel-sec/remotes.yml`).
 
+* `scripts/prepare_remotes.py` - creates the local git repository
+and adds all configured remotes. You may prefer to skip this script
+and configure the repository by hand.
+
 * `scripts/import_debian.py` - import information from Debian's
 `kernel_sec` project.  It includes all issues that Debian considers
 active or that are already tracked here.
@@ -81,6 +85,7 @@ with the keys:
   branch from this remote.
 * `git_name`: (optional) The name actually used for this git
   remote, if it's different from the default.
+* `git_repo_url`: URL of the remote git repository.
 
 ## Contributions
 
diff --git a/conf/remotes.yml b/conf/remotes.yml
index 51c523d..cfaa35f 100644
--- a/conf/remotes.yml
+++ b/conf/remotes.yml
@@ -1,6 +1,9 @@
 torvalds:
   commit_url_prefix: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=
+  git_repo_url: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
 stable:
   commit_url_prefix: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit?id=
+  git_repo_url: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
 cip:
   commit_url_prefix: https://git.kernel.org/pub/scm/linux/kernel/git/cip/linux-cip.git/commit?id=
+  git_repo_url: https://git.kernel.org/pub/scm/linux/kernel/git/cip/linux-cip.git
diff --git a/scripts/import_stable.py b/scripts/import_stable.py
index 26fcc2f..194219d 100755
--- a/scripts/import_stable.py
+++ b/scripts/import_stable.py
@@ -31,11 +31,6 @@ BACKPORT_COMMIT_BOTTOM_RE = re.compile(
     .format(**RE_USE))
 
 
-def update(git_repo, remote_name):
-    subprocess.check_call(['git', 'remote', 'update', remote_name],
-                          cwd=git_repo)
-
-
 def get_backports(git_repo, remotes, branches, debug=False):
     backports = {}
 
@@ -140,7 +135,8 @@ def main(git_repo, remotes, debug=False):
     remote_names = set(branch['git_remote'] for branch in branches)
 
     for remote_name in remote_names:
-        update(git_repo, remotes[remote_name]['git_name'])
+        kernel_sec.branch.remote_update(
+            git_repo, remotes[remote_name]['git_name'])
     backports = get_backports(git_repo, remotes, branches, debug)
     c_b_map = kernel_sec.branch.CommitBranchMap(git_repo, remotes, branches)
 
diff --git a/scripts/kernel_sec/branch.py b/scripts/kernel_sec/branch.py
index a138b96..0023497 100644
--- a/scripts/kernel_sec/branch.py
+++ b/scripts/kernel_sec/branch.py
@@ -223,6 +223,16 @@ def get_remotes(mappings, mainline=None, stable=None):
     return remotes
 
 
+def remote_update(git_repo, remote_name):
+    subprocess.check_call(['git', 'remote', 'update', remote_name],
+                          cwd=git_repo)
+
+
+def remote_add(git_repo, remote_name, remote_url):
+    subprocess.check_call(['git', 'remote', 'add', remote_name, remote_url],
+                          cwd=git_repo)
+
+
 def check_git_repo(git_repo, remotes):
     if not os.path.isdir(git_repo):
         msg = "directory %r not present" % git_repo
diff --git a/scripts/prepare_remotes.py b/scripts/prepare_remotes.py
new file mode 100755
index 0000000..59310fc
--- /dev/null
+++ b/scripts/prepare_remotes.py
@@ -0,0 +1,62 @@
+#!/usr/bin/python3
+
+# Copyright 2019 Toshiba corp.
+# Based on import_stable.py by Codethink Ltd.
+#
+# This script is distributed under the terms and conditions of the GNU General
+# Public License, Version 3 or later. See http://www.gnu.org/copyleft/gpl.html
+# for details.
+
+# Helper script that prepares the local git repository with the configured
+# remote branches
+
+import argparse
+import os
+import subprocess
+
+import kernel_sec.branch
+
+
+def main(git_repo, remotes):
+    if os.path.isdir(git_repo):
+        msg = "directory %r already exists" % git_repo
+        raise argparse.ArgumentError(None, msg)
+    else:
+        os.mkdir(git_repo)
+        subprocess.check_call(['git', 'init', '.'], cwd=git_repo)
+
+    for key in remotes.keys():
+        remote = remotes[key]  # __getitem__ will add git_name
+        kernel_sec.branch.remote_add(
+            git_repo, remote['git_name'], remote['git_repo_url'])
+        kernel_sec.branch.remote_update(git_repo, remote['git_name'])
+
+    # self-check
+    kernel_sec.branch.check_git_repo(git_repo, remotes)
+
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser(
+        description=('Prepare local git repository with configured remotes.'))
+    parser.add_argument('--git-repo',
+                        dest='git_repo', default='../kernel',
+                        help=('local git repository location '
+                              '(default: ../kernel)'),
+                        metavar='DIRECTORY')
+    parser.add_argument('--remote-name',
+                        dest='remote_name', action='append', default=[],
+                        help='git remote name mappings, e.g. stable:korg-stable',
+                        metavar='NAME:OTHER-NAME')
+    parser.add_argument('--mainline-remote',
+                        dest='mainline_remote_name',
+                        help="git remote name to use instead of 'torvalds'",
+                        metavar='OTHER-NAME')
+    parser.add_argument('--stable-remote',
+                        dest='stable_remote_name',
+                        help="git remote name to use instead of 'stable'",
+                        metavar='OTHER-NAME')
+    args = parser.parse_args()
+    remotes = kernel_sec.branch.get_remotes(args.remote_name,
+                                            mainline=args.mainline_remote_name,
+                                            stable=args.stable_remote_name)
+    main(args.git_repo, remotes)
-- 
2.17.1

^ permalink raw reply related	[flat|nested] 10+ messages in thread

* [cip-dev] [cip-kernel-sec][RESEND 3/6] report_affected: fix code when branches are specified
  2019-07-10  1:24 [cip-dev] Add support for cip branches and tags Daniel Sangorrin
  2019-07-10  1:24 ` [cip-dev] [cip-kernel-sec][RESEND 1/6] check_git_repo: add checks to the local repository Daniel Sangorrin
  2019-07-10  1:24 ` [cip-dev] [cip-kernel-sec][RESEND 2/6] prepare_remotes: helper script to prepare local repo Daniel Sangorrin
@ 2019-07-10  1:24 ` Daniel Sangorrin
  2019-07-10  1:24 ` [cip-dev] [cip-kernel-sec][RESEND 4/6] report_affected: add support for reporting on tags Daniel Sangorrin
                   ` (3 subsequent siblings)
  6 siblings, 0 replies; 10+ messages in thread
From: Daniel Sangorrin @ 2019-07-10  1:24 UTC (permalink / raw)
  To: cip-dev

The previous code could not handle branches with names
other than stable branch names. For example, passing
"linux-4.4.y-cip" as a branch would return an error.

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 scripts/kernel_sec/branch.py |  8 --------
 scripts/report_affected.py   | 23 +++++++++++++++++------
 2 files changed, 17 insertions(+), 14 deletions(-)

diff --git a/scripts/kernel_sec/branch.py b/scripts/kernel_sec/branch.py
index 0023497..ef88b54 100644
--- a/scripts/kernel_sec/branch.py
+++ b/scripts/kernel_sec/branch.py
@@ -21,9 +21,6 @@ import yaml
 from . import version
 
 
-_STABLE_BRANCH_RE = re.compile(r'^linux-([\d.]+)\.y$')
-
-
 def get_base_ver_stable_branch(base_ver):
     branch_name = 'linux-%s.y' % base_ver
     return {
@@ -34,11 +31,6 @@ def get_base_ver_stable_branch(base_ver):
         }
 
 
-def get_stable_branch(branch_name):
-    match = _STABLE_BRANCH_RE.match(branch_name)
-    return match and get_base_ver_stable_branch(match.group(1))
-
-
 def _extract_live_stable_branches(doc):
     xhtml_ns = 'http://www.w3.org/1999/xhtml'
     ns = {'html': xhtml_ns}
diff --git a/scripts/report_affected.py b/scripts/report_affected.py
index d2f1f22..0ac27f6 100755
--- a/scripts/report_affected.py
+++ b/scripts/report_affected.py
@@ -18,14 +18,25 @@ import kernel_sec.version
 
 def main(git_repo, remotes,
          only_fixed_upstream, include_ignored, *branch_names):
+    live_branches = kernel_sec.branch.get_live_branches()
     if branch_names:
-        # Support stable release strings as shorthand for stable branches
-        branches = [kernel_sec.branch.get_base_ver_stable_branch(name)
-                    if name[0].isdigit()
-                    else kernel_sec.branch.get_stable_branch(name)
-                    for name in branch_names]
+        branches = []
+        for branch_name in branch_names:
+            if branch_name[0].isdigit():
+                # 4.4 is mapped to linux-4.4.y
+                name = 'linux-%s.y' % branch_name
+            else:
+                name = branch_name
+
+            for branch in live_branches:
+                if branch['short_name'] == name:
+                    branches.append(branch)
+                    break
+            else:
+                msg = "Branch %s could not be found" % branch_name
+                raise argparse.ArgumentError(None, msg)
     else:
-        branches = kernel_sec.branch.get_live_branches()
+        branches = live_branches
         if only_fixed_upstream:
             branches = [branch for branch in branches
                         if branch['short_name'] != 'mainline']
-- 
2.17.1

^ permalink raw reply related	[flat|nested] 10+ messages in thread

* [cip-dev] [cip-kernel-sec][RESEND 4/6] report_affected: add support for reporting on tags
  2019-07-10  1:24 [cip-dev] Add support for cip branches and tags Daniel Sangorrin
                   ` (2 preceding siblings ...)
  2019-07-10  1:24 ` [cip-dev] [cip-kernel-sec][RESEND 3/6] report_affected: fix code when branches are specified Daniel Sangorrin
@ 2019-07-10  1:24 ` Daniel Sangorrin
  2019-07-10 14:40   ` Ben Hutchings
  2019-07-10  1:24 ` [cip-dev] [cip-kernel-sec][RESEND 5/6] pep8: fix pep8-related errors such as too long lines Daniel Sangorrin
                   ` (2 subsequent siblings)
  6 siblings, 1 reply; 10+ messages in thread
From: Daniel Sangorrin @ 2019-07-10  1:24 UTC (permalink / raw)
  To: cip-dev

Reporting on tags is useful for product engineers that
have shipped a kernel with a specific tag and need to know
which issues affect their product after some time.

Examples:
$ ./scripts/report_affected.py linux-4.14.y linux-4.4.y:v4.4.107 v4.4.181-cip33
$ cd ../kernel
$ git tag myproduct-v1 0f13d9b4d0efa9e87381717c113df57718bc92d6
$ cd ../cip-kernel-sec
$ ./scripts/report_affected.py linux-4.19.y-cip:myproduct-v1 v4.19.50-cip3

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 conf/branches.yml            |  2 +
 scripts/kernel_sec/branch.py |  4 +-
 scripts/report_affected.py   | 77 +++++++++++++++++++++++++++++++-----
 3 files changed, 72 insertions(+), 11 deletions(-)

diff --git a/conf/branches.yml b/conf/branches.yml
index 2ed9db6..8197596 100644
--- a/conf/branches.yml
+++ b/conf/branches.yml
@@ -2,7 +2,9 @@
   base_ver: "4.4"
   git_remote: cip
   git_name: linux-4.4.y-cip
+  tag_regexp: '^v4\.4\.\d+-cip\d+$'
 - short_name: linux-4.19.y-cip
   base_ver: "4.19"
   git_remote: cip
   git_name: linux-4.19.y-cip
+  tag_regexp: '^v4\.19\.\d+-cip\d+$'
diff --git a/scripts/kernel_sec/branch.py b/scripts/kernel_sec/branch.py
index ef88b54..96f1cab 100644
--- a/scripts/kernel_sec/branch.py
+++ b/scripts/kernel_sec/branch.py
@@ -141,7 +141,7 @@ def get_sort_key(branch):
     return version.get_sort_key(base_ver)
 
 
-def _get_commits(git_repo, end, start=None):
+def iter_rev_list(git_repo, end, start=None):
     if start:
         list_expr = '%s..%s' % (start, end)
     else:
@@ -170,7 +170,7 @@ class CommitBranchMap:
                                  branch['git_name'])
             else:
                 end = 'v' + branch['base_ver']
-            for commit in _get_commits(git_repo, end, start):
+            for commit in iter_rev_list(git_repo, end, start):
                 self._commit_sort_key[commit] \
                     = self._branch_sort_key[branch_name]
             start = end
diff --git a/scripts/report_affected.py b/scripts/report_affected.py
index 0ac27f6..7ec4af7 100755
--- a/scripts/report_affected.py
+++ b/scripts/report_affected.py
@@ -9,7 +9,9 @@
 # Report issues affecting each stable branch.
 
 import argparse
+import copy
 import subprocess
+import re
 
 import kernel_sec.branch
 import kernel_sec.issue
@@ -22,15 +24,47 @@ def main(git_repo, remotes,
     if branch_names:
         branches = []
         for branch_name in branch_names:
+            tag = None
             if branch_name[0].isdigit():
                 # 4.4 is mapped to linux-4.4.y
                 name = 'linux-%s.y' % branch_name
+            elif branch_name[0] == 'v':
+                # an official tag, e.g. v4.4.92-cip11
+                # infer branch from tag (regexp's must be specific)
+                for branch in live_branches:
+                    if 'tag_regexp' in branch:
+                        # predefined in conf/branches.yml
+                        tag_regexp = branch['tag_regexp']
+                    elif branch['git_remote'] == 'stable':
+                        # stable format, e.g. v4.19.12
+                        esc_base_ver = branch['base_ver'].replace('.', '\.')
+                        tag_regexp = r'(^v%s$|^v%s\.\d+$)' % (
+                            esc_base_ver, esc_base_ver)
+                    else:
+                        # no tag_regexp defined, or mainline
+                        continue
+
+                    if re.match(tag_regexp, branch_name):
+                        tag = branch_name
+                        name = branch['short_name']
+                        break
+                else:
+                    raise ValueError('Failed to match tag %r' % branch_name)
+            elif ':' in branch_name:
+                # a possibly custom tag, e.g. linux-4.19.y-cip:myproduct-v1
+                name_tuple = tuple(branch_name.split(':'))
+                name = name_tuple[0]
+                tag = name_tuple[1]
             else:
                 name = branch_name
 
             for branch in live_branches:
                 if branch['short_name'] == name:
-                    branches.append(branch)
+                    # there could be multiple tags for the same branch
+                    branch_copy = copy.deepcopy(branch)
+                    if tag:
+                        branch_copy['tag'] = tag
+                    branches.append(branch_copy)
                     break
             else:
                 msg = "Branch %s could not be found" % branch_name
@@ -45,6 +79,18 @@ def main(git_repo, remotes,
 
     c_b_map = kernel_sec.branch.CommitBranchMap(git_repo, remotes, branches)
 
+    # cache tag commits and set full_name to show the tag
+    tag_commits = {}
+    for branch in branches:
+        if 'tag' in branch:
+            start = 'v' + branch['base_ver']
+            end = branch['tag']
+            tag_commits[end] = set(
+                kernel_sec.branch.iter_rev_list(git_repo, end, start))
+            branch['full_name'] = ':'.join([branch['short_name'], end])
+        else:
+            branch['full_name'] = branch['short_name']
+
     branch_issues = {}
     issues = set(kernel_sec.issue.get_list())
 
@@ -65,15 +111,26 @@ def main(git_repo, remotes,
             if not include_ignored and ignore.get(branch_name):
                 continue
 
+            # Check if the branch is affected. If not and the issue was fixed
+            # on that branch, then make sure the tag contains that fix
             if kernel_sec.issue.affects_branch(
                     issue, branch, c_b_map.is_commit_in_branch):
-                branch_issues.setdefault(branch_name, []).append(cve_id)
+                branch_issues.setdefault(
+                    branch['full_name'], []).append(cve_id)
+            elif 'tag' in branch and fixed:
+                if fixed.get(branch_name, 'never') == 'never':
+                    continue
+                for commit in fixed[branch_name]:
+                    if commit not in tag_commits[branch['tag']]:
+                        branch_issues.setdefault(
+                            branch['full_name'], []).append(cve_id)
+                        break
 
     for branch in branches:
-        branch_name = branch['short_name']
-        print('%s:' % branch_name,
-              *sorted(branch_issues.get(branch_name, []),
-                      key=kernel_sec.issue.get_id_sort_key))
+        sorted_cve_ids = sorted(
+            branch_issues.get(branch['full_name'], []),
+            key=kernel_sec.issue.get_id_sort_key)
+        print('%s:' % branch['full_name'], *sorted_cve_ids)
 
 
 if __name__ == '__main__':
@@ -104,9 +161,11 @@ if __name__ == '__main__':
                         help='include issues that have been marked as ignored')
     parser.add_argument('branches',
                         nargs='*',
-                        help=('specific branch to report on '
-                              '(default: all active branches)'),
-                        metavar='BRANCH')
+                        help=('specific branch[:tag] or stable tag to '
+                              'report on (default: all active branches). '
+                              'e.g. linux-4.14.y linux-4.4.y:v4.4.107 '
+                              'v4.4.181-cip33 linux-4.19.y-cip:myproduct-v33'),
+                        metavar='[BRANCH[:TAG]|TAG]')
     args = parser.parse_args()
     remotes = kernel_sec.branch.get_remotes(args.remote_name,
                                             mainline=args.mainline_remote_name,
-- 
2.17.1

^ permalink raw reply related	[flat|nested] 10+ messages in thread

* [cip-dev] [cip-kernel-sec][RESEND 5/6] pep8: fix pep8-related errors such as too long lines
  2019-07-10  1:24 [cip-dev] Add support for cip branches and tags Daniel Sangorrin
                   ` (3 preceding siblings ...)
  2019-07-10  1:24 ` [cip-dev] [cip-kernel-sec][RESEND 4/6] report_affected: add support for reporting on tags Daniel Sangorrin
@ 2019-07-10  1:24 ` Daniel Sangorrin
  2019-07-10  1:24 ` [cip-dev] [cip-kernel-sec][RESEND 6/6] report_affected: add show-description option Daniel Sangorrin
  2019-07-10 15:02 ` [cip-dev] Add support for cip branches and tags Ben Hutchings
  6 siblings, 0 replies; 10+ messages in thread
From: Daniel Sangorrin @ 2019-07-10  1:24 UTC (permalink / raw)
  To: cip-dev

These were distracting when checking new code.

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 scripts/import_stable.py     | 8 +++++---
 scripts/kernel_sec/branch.py | 4 ++--
 scripts/prepare_remotes.py   | 2 +-
 scripts/report_affected.py   | 2 +-
 scripts/webview.py           | 2 +-
 5 files changed, 10 insertions(+), 8 deletions(-)

diff --git a/scripts/import_stable.py b/scripts/import_stable.py
index 194219d..b4d8e0a 100755
--- a/scripts/import_stable.py
+++ b/scripts/import_stable.py
@@ -64,7 +64,8 @@ def get_backports(git_repo, remotes, branches, debug=False):
                     backports.setdefault(mainline_commit, {})[branch_name] \
                         = stable_commit
                 if line.strip() != '':
-                    commit_re = BACKPORT_COMMIT_BOTTOM_RE  # next line is not top
+                    # next line is not top
+                    commit_re = BACKPORT_COMMIT_BOTTOM_RE
 
     return backports
 
@@ -120,7 +121,8 @@ def add_backports(branches, c_b_map, issue_commits, all_backports,
                 if debug_context:
                     print('%s/%s: recording commits' %
                           (debug_context, branch_name))
-                issue_commits.setdefault(branch_name, []).extend(branch_commits)
+                issue_commits.setdefault(
+                    branch_name, []).extend(branch_commits)
                 changed = True
             else:
                 if debug_context:
@@ -169,7 +171,7 @@ if __name__ == '__main__':
                         metavar='DIRECTORY')
     parser.add_argument('--remote-name',
                         dest='remote_name', action='append', default=[],
-                        help='git remote name mappings, e.g. stable:korg-stable',
+                        help='git remote name mappings, e.g. stable:mystable',
                         metavar='NAME:OTHER-NAME')
     parser.add_argument('--mainline-remote',
                         dest='mainline_remote_name',
diff --git a/scripts/kernel_sec/branch.py b/scripts/kernel_sec/branch.py
index 96f1cab..52f3b42 100644
--- a/scripts/kernel_sec/branch.py
+++ b/scripts/kernel_sec/branch.py
@@ -62,8 +62,8 @@ def _extract_live_stable_branches(doc):
             if match:
                 version = match.group(1)
                 eol = match.group(2) is not None
-        if branch_type not in ['mainline', 'stable', 'longterm', 'linux-next'] \
-           or version is None:
+        if branch_type not in ['mainline', 'stable', 'longterm',
+                               'linux-next'] or version is None:
             raise ValueError('failed to parse releases row text %r' % row_text)
 
         # Filter out irrelevant branches
diff --git a/scripts/prepare_remotes.py b/scripts/prepare_remotes.py
index 59310fc..ab0db24 100755
--- a/scripts/prepare_remotes.py
+++ b/scripts/prepare_remotes.py
@@ -45,7 +45,7 @@ if __name__ == '__main__':
                         metavar='DIRECTORY')
     parser.add_argument('--remote-name',
                         dest='remote_name', action='append', default=[],
-                        help='git remote name mappings, e.g. stable:korg-stable',
+                        help='git remote name mappings, e.g. stable:mystable',
                         metavar='NAME:OTHER-NAME')
     parser.add_argument('--mainline-remote',
                         dest='mainline_remote_name',
diff --git a/scripts/report_affected.py b/scripts/report_affected.py
index 7ec4af7..a97986c 100755
--- a/scripts/report_affected.py
+++ b/scripts/report_affected.py
@@ -143,7 +143,7 @@ if __name__ == '__main__':
                         metavar='DIRECTORY')
     parser.add_argument('--remote-name',
                         dest='remote_name', action='append', default=[],
-                        help='git remote name mappings, e.g. stable:korg-stable',
+                        help='git remote name mappings, e.g. stable:mystable',
                         metavar='NAME:OTHER-NAME')
     parser.add_argument('--mainline-remote',
                         dest='mainline_remote_name',
diff --git a/scripts/webview.py b/scripts/webview.py
index 52a7b15..cf54948 100755
--- a/scripts/webview.py
+++ b/scripts/webview.py
@@ -205,7 +205,7 @@ if __name__ == '__main__':
                         metavar='DIRECTORY')
     parser.add_argument('--remote-name',
                         dest='remote_name', action='append', default=[],
-                        help='git remote name mappings, e.g. stable:korg-stable',
+                        help='git remote name mappings, e.g. stable:mystable',
                         metavar='NAME:OTHER-NAME')
     parser.add_argument('--mainline-remote',
                         dest='mainline_remote_name',
-- 
2.17.1

^ permalink raw reply related	[flat|nested] 10+ messages in thread

* [cip-dev] [cip-kernel-sec][RESEND 6/6] report_affected: add show-description option
  2019-07-10  1:24 [cip-dev] Add support for cip branches and tags Daniel Sangorrin
                   ` (4 preceding siblings ...)
  2019-07-10  1:24 ` [cip-dev] [cip-kernel-sec][RESEND 5/6] pep8: fix pep8-related errors such as too long lines Daniel Sangorrin
@ 2019-07-10  1:24 ` Daniel Sangorrin
  2019-07-10 15:02 ` [cip-dev] Add support for cip branches and tags Ben Hutchings
  6 siblings, 0 replies; 10+ messages in thread
From: Daniel Sangorrin @ 2019-07-10  1:24 UTC (permalink / raw)
  To: cip-dev

Rather than looking up each issue file, I would like
to have an overview of what each CVE ID means.

Example:
$ ./scripts/report_affected.py --show-description linux-4.4.y-cip

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 scripts/report_affected.py | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)

diff --git a/scripts/report_affected.py b/scripts/report_affected.py
index a97986c..b7a2678 100755
--- a/scripts/report_affected.py
+++ b/scripts/report_affected.py
@@ -18,8 +18,8 @@ import kernel_sec.issue
 import kernel_sec.version
 
 
-def main(git_repo, remotes,
-         only_fixed_upstream, include_ignored, *branch_names):
+def main(git_repo, remotes, only_fixed_upstream,
+         include_ignored, show_description, *branch_names):
     live_branches = kernel_sec.branch.get_live_branches()
     if branch_names:
         branches = []
@@ -130,7 +130,13 @@ def main(git_repo, remotes,
         sorted_cve_ids = sorted(
             branch_issues.get(branch['full_name'], []),
             key=kernel_sec.issue.get_id_sort_key)
-        print('%s:' % branch['full_name'], *sorted_cve_ids)
+        if show_description:
+            print('%s:' % branch['full_name'])
+            for cve_id in sorted_cve_ids:
+                print(cve_id, '=>',
+                      kernel_sec.issue.load(cve_id).get('description', 'None'))
+        else:
+            print('%s:' % branch['full_name'], *sorted_cve_ids)
 
 
 if __name__ == '__main__':
@@ -159,6 +165,9 @@ if __name__ == '__main__':
     parser.add_argument('--include-ignored',
                         action='store_true',
                         help='include issues that have been marked as ignored')
+    parser.add_argument('--show-description',
+                        action='store_true',
+                        help='show the issue description')
     parser.add_argument('branches',
                         nargs='*',
                         help=('specific branch[:tag] or stable tag to '
@@ -171,5 +180,5 @@ if __name__ == '__main__':
                                             mainline=args.mainline_remote_name,
                                             stable=args.stable_remote_name)
     kernel_sec.branch.check_git_repo(args.git_repo, remotes)
-    main(args.git_repo, remotes,
-         args.only_fixed_upstream, args.include_ignored, *args.branches)
+    main(args.git_repo, remotes, args.only_fixed_upstream,
+         args.include_ignored, args.show_description, *args.branches)
-- 
2.17.1

^ permalink raw reply related	[flat|nested] 10+ messages in thread

* [cip-dev] [cip-kernel-sec][RESEND 4/6] report_affected: add support for reporting on tags
  2019-07-10  1:24 ` [cip-dev] [cip-kernel-sec][RESEND 4/6] report_affected: add support for reporting on tags Daniel Sangorrin
@ 2019-07-10 14:40   ` Ben Hutchings
  2019-07-11  4:50     ` daniel.sangorrin at toshiba.co.jp
  0 siblings, 1 reply; 10+ messages in thread
From: Ben Hutchings @ 2019-07-10 14:40 UTC (permalink / raw)
  To: cip-dev

On Wed, 2019-07-10 at 10:24 +0900, Daniel Sangorrin wrote:
[...]
> --- a/scripts/report_affected.py
> +++ b/scripts/report_affected.py
[...]
> @@ -22,15 +24,47 @@ def main(git_repo, remotes,
>      if branch_names:
>          branches = []
>          for branch_name in branch_names:
> +            tag = None
>              if branch_name[0].isdigit():
>                  # 4.4 is mapped to linux-4.4.y
>                  name = 'linux-%s.y' % branch_name
> +            elif branch_name[0] == 'v':
> +                # an official tag, e.g. v4.4.92-cip11
> +                # infer branch from tag (regexp's must be specific)
> +                for branch in live_branches:
> +                    if 'tag_regexp' in branch:
> +                        # predefined in conf/branches.yml
> +                        tag_regexp = branch['tag_regexp']
> +                    elif branch['git_remote'] == 'stable':
> +                        # stable format, e.g. v4.19.12
> +                        esc_base_ver = branch['base_ver'].replace('.', '\.')

This happens to work now, but '\.' is an unrecognised escape sequence
which is deprecated.  You presumably meant r'\.', but it might be
clearer to use re.escape().

> +                        tag_regexp = r'(^v%s$|^v%s\.\d+$)' % (
> +                            esc_base_ver, esc_base_ver)

I also expected that you would set tag_regexp for stable branches in
the branch module along with all their other fields.  Then there's no
need to handle them specially here.

> +                    else:
> +                        # no tag_regexp defined, or mainline
> +                        continue
> +
> +                    if re.match(tag_regexp, branch_name):
> +                        tag = branch_name
> +                        name = branch['short_name']
> +                        break
> +                else:
> +                    raise ValueError('Failed to match tag %r' % branch_name)
> +            elif ':' in branch_name:
> +                # a possibly custom tag, e.g. linux-4.19.y-cip:myproduct-v1
> +                name_tuple = tuple(branch_name.split(':'))
> +                name = name_tuple[0]
> +                tag = name_tuple[1]
[...]

You really can do simply:

                name, tag = branch_name.split(':', 1)

(Tuple assignment only requires an iterable, not specifically a tuple,
on the right hand side.)  So please use that.

Ben.

-- 
Ben Hutchings, Software Developer                         Codethink Ltd
https://www.codethink.co.uk/                 Dale House, 35 Dale Street
                                     Manchester, M1 2HF, United Kingdom

^ permalink raw reply	[flat|nested] 10+ messages in thread

* [cip-dev] Add support for cip branches and tags
  2019-07-10  1:24 [cip-dev] Add support for cip branches and tags Daniel Sangorrin
                   ` (5 preceding siblings ...)
  2019-07-10  1:24 ` [cip-dev] [cip-kernel-sec][RESEND 6/6] report_affected: add show-description option Daniel Sangorrin
@ 2019-07-10 15:02 ` Ben Hutchings
  6 siblings, 0 replies; 10+ messages in thread
From: Ben Hutchings @ 2019-07-10 15:02 UTC (permalink / raw)
  To: cip-dev

On Wed, 2019-07-10 at 10:24 +0900, Daniel Sangorrin wrote:
> Hello Ben,
> 
> Thank you for the detailed review, and sorry for the delay.
> I have modified my patch series taking your comments into account.
> Note: I will reply to your comments separately
> 
> I re-send to you the whole series to make sure that patches
> apply properly.

I've applied patches 1, 2, 3, and 5 (as it didn't depend on 4).  Please
re-send the remaining two with my requested changes.

Ben.

-- 
Ben Hutchings, Software Developer                         Codethink Ltd
https://www.codethink.co.uk/                 Dale House, 35 Dale Street
                                     Manchester, M1 2HF, United Kingdom

^ permalink raw reply	[flat|nested] 10+ messages in thread

* [cip-dev] [cip-kernel-sec][RESEND 4/6] report_affected: add support for reporting on tags
  2019-07-10 14:40   ` Ben Hutchings
@ 2019-07-11  4:50     ` daniel.sangorrin at toshiba.co.jp
  0 siblings, 0 replies; 10+ messages in thread
From: daniel.sangorrin at toshiba.co.jp @ 2019-07-11  4:50 UTC (permalink / raw)
  To: cip-dev

> From: Ben Hutchings <ben.hutchings@codethink.co.uk>
[...]
> > +                    if 'tag_regexp' in branch:
> > +                        # predefined in conf/branches.yml
> > +                        tag_regexp = branch['tag_regexp']
> > +                    elif branch['git_remote'] == 'stable':
> > +                        # stable format, e.g. v4.19.12
> > +                        esc_base_ver = branch['base_ver'].replace('.', '\.')
> 
> This happens to work now, but '\.' is an unrecognised escape sequence
> which is deprecated.  You presumably meant r'\.', but it might be
> clearer to use re.escape().

Thanks, you are right. I have used re.escape('.'). I didn't know this function, it's really useful not having to remember how to escape characters (I figured out I could have used \\.).

I think that the YAML strings in conf/branches.yml are being read as if they were raw strings, but if there is a problem with them let me know.

> 
> > +                        tag_regexp = r'(^v%s$|^v%s\.\d+$)' % (
> > +                            esc_base_ver, esc_base_ver)
> 
> I also expected that you would set tag_regexp for stable branches in
> the branch module along with all their other fields.  Then there's no
> need to handle them specially here.

Ah sorry about that. I have moved that code to branch.py now.

> 
> > +                    else:
> > +                        # no tag_regexp defined, or mainline
> > +                        continue
> > +
> > +                    if re.match(tag_regexp, branch_name):
> > +                        tag = branch_name
> > +                        name = branch['short_name']
> > +                        break
> > +                else:
> > +                    raise ValueError('Failed to match tag %r' % branch_name)
> > +            elif ':' in branch_name:
> > +                # a possibly custom tag, e.g. linux-4.19.y-cip:myproduct-v1
> > +                name_tuple = tuple(branch_name.split(':'))
> > +                name = name_tuple[0]
> > +                tag = name_tuple[1]
> [...]
> 
> You really can do simply:
> 
>                 name, tag = branch_name.split(':', 1)
> 
> (Tuple assignment only requires an iterable, not specifically a tuple,
> on the right hand side.)  So please use that.

Nice trick. I fixed the code.

Thanks,
Daniel


> 
> Ben.
> 
> --
> Ben Hutchings, Software Developer                         Codethink Ltd
> https://www.codethink.co.uk/                 Dale House, 35 Dale Street
>                                      Manchester, M1 2HF, United Kingdom

^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2019-07-11  4:50 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-07-10  1:24 [cip-dev] Add support for cip branches and tags Daniel Sangorrin
2019-07-10  1:24 ` [cip-dev] [cip-kernel-sec][RESEND 1/6] check_git_repo: add checks to the local repository Daniel Sangorrin
2019-07-10  1:24 ` [cip-dev] [cip-kernel-sec][RESEND 2/6] prepare_remotes: helper script to prepare local repo Daniel Sangorrin
2019-07-10  1:24 ` [cip-dev] [cip-kernel-sec][RESEND 3/6] report_affected: fix code when branches are specified Daniel Sangorrin
2019-07-10  1:24 ` [cip-dev] [cip-kernel-sec][RESEND 4/6] report_affected: add support for reporting on tags Daniel Sangorrin
2019-07-10 14:40   ` Ben Hutchings
2019-07-11  4:50     ` daniel.sangorrin at toshiba.co.jp
2019-07-10  1:24 ` [cip-dev] [cip-kernel-sec][RESEND 5/6] pep8: fix pep8-related errors such as too long lines Daniel Sangorrin
2019-07-10  1:24 ` [cip-dev] [cip-kernel-sec][RESEND 6/6] report_affected: add show-description option Daniel Sangorrin
2019-07-10 15:02 ` [cip-dev] Add support for cip branches and tags Ben Hutchings

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.