[cip-dev] [cip-kernel-sec 1/2] remotes: automatically add remotes from configuration file

[cip-dev] [cip-kernel-sec 1/2] remotes: automatically add remotes from configuration file

[Daniel Sangorrin]

Currently the user is required to create its own remotes by
hand. This should not be necessary, because the information
is already collected in conf/remotes.yml. For that reason,
if we detect that any remote has not been added to the
local repo then we will add those.

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 README.md                    | 5 +++--
 conf/remotes.yml             | 9 ++++++---
 scripts/import_stable.py     | 9 +++++++++
 scripts/templates/issue.html | 7 ++++---
 4 files changed, 22 insertions(+), 8 deletions(-)

diff --git a/README.md b/README.md
index 4c5808f..dda94a8 100644
--- a/README.md
+++ b/README.md
@@ -77,8 +77,9 @@ These files, if they exist, contain a mapping where the keys
 are default git remote names.  The values are also mappings,
 with the keys:
 
-* `commit_url_prefix`: URL prefix for browsing a commit on a
-  branch from this remote.
+* `git_repo_url`: URL of the remote repository.
+* `commit_url_suffix`: URL suffix that gets appended to `git_repo_url`
+  for browsing a commit on a branch from this remote.
 * `git_name`: (optional) The name actually used for this git
   remote, if it's different from the default.
 
diff --git a/conf/remotes.yml b/conf/remotes.yml
index 51c523d..446bb15 100644
--- a/conf/remotes.yml
+++ b/conf/remotes.yml
@@ -1,6 +1,9 @@
 torvalds:
-  commit_url_prefix: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=
+  git_repo_url: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
+  commit_url_suffix: /commit/?id=
 stable:
-  commit_url_prefix: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit?id=
+  git_repo_url: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
+  commit_url_suffix: /commit?id=
 cip:
-  commit_url_prefix: https://git.kernel.org/pub/scm/linux/kernel/git/cip/linux-cip.git/commit?id=
+  git_repo_url: https://git.kernel.org/pub/scm/linux/kernel/git/cip/linux-cip.git
+  commit_url_suffix: /commit?id=
diff --git a/scripts/import_stable.py b/scripts/import_stable.py
index b3261cf..26e45a9 100755
--- a/scripts/import_stable.py
+++ b/scripts/import_stable.py
@@ -35,6 +35,9 @@ def update(git_repo, remote_name):
     subprocess.check_call(['git', 'remote', 'update', remote_name],
                           cwd=git_repo)
 
+def add(git_repo, remote_name, remote_url):
+    subprocess.check_call(['git', 'remote', 'add', remote_name, remote_url],
+                          cwd=git_repo)
 
 def get_backports(git_repo, remotes, branches, debug=False):
     backports = {}
@@ -140,6 +143,12 @@ def main(git_repo, remotes, debug=False):
     remote_names = set(branch['git_remote'] for branch in branches)
 
     for remote_name in remote_names:
+        import sys
+        current_remotes = subprocess.check_output(['git', 'remote', 'show'],
+            cwd=git_repo).decode(sys.stdout.encoding).strip().split('\n')
+        if remote_name not in current_remotes:
+            add(git_repo, remotes[remote_name]['git_name'],
+                remotes[remote_name]['git_repo_url'])
         update(git_repo, remotes[remote_name]['git_name'])
     backports = get_backports(git_repo, remotes, branches, debug)
     c_b_map = kernel_sec.branch.CommitBranchMap(git_repo, remotes, branches)
diff --git a/scripts/templates/issue.html b/scripts/templates/issue.html
index a3286e1..de42753 100644
--- a/scripts/templates/issue.html
+++ b/scripts/templates/issue.html
@@ -75,7 +75,8 @@
     <th rowspan={{ branches|length }}>Status</th>
     {% for branch, affected in branches %}
     {% set name = branch.short_name %}
-    {% set url_prefix = remotes[branch.git_remote].commit_url_prefix %}
+    {% set git_repo_url = remotes[branch.git_remote].git_repo_url %}
+    {% set url_suffix = remotes[branch.git_remote].commit_url_suffix %}
     <th>
       <a href="/branch/{{ name }}/">{{ name }}</a>
     </th>
@@ -84,7 +85,7 @@
       {% if issue['fixed-by'] and issue['fixed-by'][name] and issue['fixed-by'][name] != 'never' %}
       <span class="good">fixed</span> by
       {% for commit in issue['fixed-by'][name] %}
-      <a href="{{ url_prefix }}{{ commit }}">{{ commit[:12] }}</a>{% if not loop.last %},{% endif %}
+      <a href="{{ git_repo_url }}{{ url_suffix }}{{ commit }}">{{ commit[:12] }}</a>{% if not loop.last %},{% endif %}
       {% endfor %}
       {% else %}
       <span class="good">never affected</span>
@@ -98,7 +99,7 @@
       {% if issue['introduced-by'] and issue['introduced-by'][name] and issue['introduced-by'][name] != 'never' %}
       - introduced by
       {% for commit in issue['introduced-by'][name] %}
-      <a href="{{ url_prefix }}{{ commit }}">{{ commit[:12] }}</a>{% if not loop.last %},{% endif %}
+      <a href="{{ git_repo_url }}{{ url_suffix }}{{ commit }}">{{ commit[:12] }}</a>{% if not loop.last %},{% endif %}
       {% endfor %}
       {% endif %}
       {% endif %}
-- 
2.17.1

[cip-dev] [cip-kernel-sec 2/2] git_repo: add a check for the git_repo directory

[Daniel Sangorrin]

Users are required to prepare a local git repository, by
default "../kernel", however there is no check at the
moment. Alternatively, we could also create a repository
for them (mkdir ../kernel; git init .) but that for
now let's just spit an error.

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 scripts/import_stable.py     |  1 +
 scripts/kernel_sec/branch.py | 11 +++++++++++
 scripts/report_affected.py   |  1 +
 scripts/webview.py           |  1 +
 4 files changed, 14 insertions(+)

diff --git a/scripts/import_stable.py b/scripts/import_stable.py
index 26e45a9..2d9b66b 100755
--- a/scripts/import_stable.py
+++ b/scripts/import_stable.py
@@ -179,6 +179,7 @@ if __name__ == '__main__':
                         dest='git_repo', default='../kernel',
                         help=('git repository from which to read commit logs '
                               '(default: ../kernel)'),
+                        type=kernel_sec.branch.check_git_repo,
                         metavar='DIRECTORY')
     parser.add_argument('--remote-name',
                         dest='remote_name', action='append', default=[],
diff --git a/scripts/kernel_sec/branch.py b/scripts/kernel_sec/branch.py
index 3ede64a..0ea5086 100644
--- a/scripts/kernel_sec/branch.py
+++ b/scripts/kernel_sec/branch.py
@@ -219,3 +219,14 @@ def get_remotes(mappings, mainline=None, stable=None):
     if stable:
         remotes['stable']['git_name'] = stable
     return remotes
+
+
+def check_git_repo(git_repo):
+    import argparse
+    if not os.path.isdir(git_repo):
+        msg = "directory %r not present" % git_repo
+        raise argparse.ArgumentTypeError(msg)
+    if not os.path.isdir(os.path.join(git_repo, '.git')):
+        msg = "directory %r is not a git repository" % git_repo
+        raise argparse.ArgumentTypeError(msg)
+    return git_repo
diff --git a/scripts/report_affected.py b/scripts/report_affected.py
index fac0885..bcb8168 100755
--- a/scripts/report_affected.py
+++ b/scripts/report_affected.py
@@ -72,6 +72,7 @@ if __name__ == '__main__':
                         dest='git_repo', default='../kernel',
                         help=('git repository from which to read commit logs '
                               '(default: ../kernel)'),
+                        type=kernel_sec.branch.check_git_repo,
                         metavar='DIRECTORY')
     parser.add_argument('--remote-name',
                         dest='remote_name', action='append', default=[],
diff --git a/scripts/webview.py b/scripts/webview.py
index 9e16a0c..e334371 100755
--- a/scripts/webview.py
+++ b/scripts/webview.py
@@ -202,6 +202,7 @@ if __name__ == '__main__':
                         dest='git_repo', default='../kernel',
                         help=('git repository from which to read commit logs '
                               '(default: ../kernel)'),
+                        type=kernel_sec.branch.check_git_repo,
                         metavar='DIRECTORY')
     parser.add_argument('--remote-name',
                         dest='remote_name', action='append', default=[],
-- 
2.17.1

[cip-dev] [cip-kernel-sec 1/2] remotes: automatically add remotes from configuration file

[Ben Hutchings]

On Mon, 2019-06-17 at 14:21 +0900, Daniel Sangorrin wrote:
> Currently the user is required to create its own remotes by
> hand. This should not be necessary, because the information
> is already collected in conf/remotes.yml. For that reason,
> if we detect that any remote has not been added to the
> local repo then we will add those.

I'm not yet convinced this is a good idea.

After this change, we would still be assuming that there is a kernel
repository or working tree next to kernel-config.  If we wanted to be
really helpful we could create a new repository if it's not there.

But realistically, a developer using this probably already has a kernel
git repository, just not in the expected place.  Isn't it likely that
the developer also already has the remotes we want, but under different
names?

Aside from that, I have some specific issues with the implementation:

> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
> ?README.md????????????????????| 5 +++--
> ?conf/remotes.yml?????????????| 9 ++++++---
> ?scripts/import_stable.py?????| 9 +++++++++
> ?scripts/templates/issue.html | 7 ++++---
> ?4 files changed, 22 insertions(+), 8 deletions(-)
> 
> diff --git a/README.md b/README.md
> index 4c5808f..dda94a8 100644
> --- a/README.md
> +++ b/README.md
> @@ -77,8 +77,9 @@ These files, if they exist, contain a mapping where the keys
> ?are default git remote names.??The values are also mappings,
> ?with the keys:
> ?
> -* `commit_url_prefix`: URL prefix for browsing a commit on a
> -??branch from this remote.
> +* `git_repo_url`: URL of the remote repository.
> +* `commit_url_suffix`: URL suffix that gets appended to `git_repo_url`
> +??for browsing a commit on a branch from this remote.

Although it's now common to have a single https: URL that works as both
a git remote and a web view, there are plenty of git hosts that don't
work that way.

So these two things should be kept independent.

[...]
> --- a/scripts/import_stable.py
> +++ b/scripts/import_stable.py
> @@ -35,6 +35,9 @@ def update(git_repo, remote_name):
> ?????subprocess.check_call(['git', 'remote', 'update', remote_name],
> ???????????????????????????cwd=git_repo)
> ?
> +def add(git_repo, remote_name, remote_url):
> +????subprocess.check_call(['git', 'remote', 'add', remote_name, remote_url],
> +??????????????????????????cwd=git_repo)
> ?
> ?def get_backports(git_repo, remotes, branches, debug=False):
> ?????backports = {}

There should be two blank lines between top-level definitions,
according to PEP-8.

> @@ -140,6 +143,12 @@ def main(git_repo, remotes, debug=False):
>      remote_names = set(branch['git_remote'] for branch in branches)
>  
>      for remote_name in remote_names:
> +        import sys

This import belongs at the top level with the other imports.

> +        current_remotes = subprocess.check_output(['git', 'remote', 'show'],
> +            cwd=git_repo).decode(sys.stdout.encoding).strip().split('\n')

This doesn't belong inside the loop.

> +        if remote_name not in current_remotes:
> +            add(git_repo, remotes[remote_name]['git_name'],
> +                remotes[remote_name]['git_repo_url'])
>          update(git_repo, remotes[remote_name]['git_name'])
>      backports = get_backports(git_repo, remotes, branches, debug)
>      c_b_map = kernel_sec.branch.CommitBranchMap(git_repo, remotes, branches)

Ben.

-- 
Ben Hutchings, Software Developer                ?        Codethink Ltd
https://www.codethink.co.uk/                 Dale House, 35 Dale Street
                                     Manchester, M1 2HF, United Kingdom

[cip-dev] [cip-kernel-sec 1/2] remotes: automatically add remotes from configuration file

[daniel.sangorrin at toshiba.co.jp]

> From: Ben Hutchings <ben.hutchings@codethink.co.uk>
> On Mon, 2019-06-17 at 14:21 +0900, Daniel Sangorrin wrote:
> > Currently the user is required to create its own remotes by
> > hand. This should not be necessary, because the information
> > is already collected in conf/remotes.yml. For that reason,
> > if we detect that any remote has not been added to the
> > local repo then we will add those.
> 
> I'm not yet convinced this is a good idea.

I sent a new patch that takes a different approach. The new patch will
only check that the necessary remotes are there. I also prepared a
new separate script, for new users to prepare the local repository.

> After this change, we would still be assuming that there is a kernel
> repository or working tree next to kernel-config.  If we wanted to be
> really helpful we could create a new repository if it's not there.

I didn't understand the kernel-config thing. Maybe I am missing something important.

> But realistically, a developer using this probably already has a kernel
> git repository, just not in the expected place.  Isn't it likely that
> the developer also already has the remotes we want, but under different
> names?
> 
> Aside from that, I have some specific issues with the implementation:

Thanks the new patch has taken those issues into account.

Kind regards,
Daniel


> 
> > Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> > ---
> > ?README.md????????????????????| 5 +++--
> > ?conf/remotes.yml?????????????| 9 ++++++---
> > ?scripts/import_stable.py?????| 9 +++++++++
> > ?scripts/templates/issue.html | 7 ++++---
> > ?4 files changed, 22 insertions(+), 8 deletions(-)
> >
> > diff --git a/README.md b/README.md
> > index 4c5808f..dda94a8 100644
> > --- a/README.md
> > +++ b/README.md
> > @@ -77,8 +77,9 @@ These files, if they exist, contain a mapping where the keys
> > ?are default git remote names.??The values are also mappings,
> > ?with the keys:
> >
> > -* `commit_url_prefix`: URL prefix for browsing a commit on a
> > -??branch from this remote.
> > +* `git_repo_url`: URL of the remote repository.
> > +* `commit_url_suffix`: URL suffix that gets appended to `git_repo_url`
> > +??for browsing a commit on a branch from this remote.
> 
> Although it's now common to have a single https: URL that works as both
> a git remote and a web view, there are plenty of git hosts that don't
> work that way.
> 
> So these two things should be kept independent.
> 
> [...]
> > --- a/scripts/import_stable.py
> > +++ b/scripts/import_stable.py
> > @@ -35,6 +35,9 @@ def update(git_repo, remote_name):
> > ?????subprocess.check_call(['git', 'remote', 'update', remote_name],
> > ???????????????????????????cwd=git_repo)
> >
> > +def add(git_repo, remote_name, remote_url):
> > +????subprocess.check_call(['git', 'remote', 'add', remote_name, remote_url],
> > +??????????????????????????cwd=git_repo)
> >
> > ?def get_backports(git_repo, remotes, branches, debug=False):
> > ?????backports = {}
> 
> There should be two blank lines between top-level definitions,
> according to PEP-8.
> 
> > @@ -140,6 +143,12 @@ def main(git_repo, remotes, debug=False):
> >      remote_names = set(branch['git_remote'] for branch in branches)
> >
> >      for remote_name in remote_names:
> > +        import sys
> 
> This import belongs at the top level with the other imports.
> 
> > +        current_remotes = subprocess.check_output(['git', 'remote', 'show'],
> > +            cwd=git_repo).decode(sys.stdout.encoding).strip().split('\n')
> 
> This doesn't belong inside the loop.
> 
> > +        if remote_name not in current_remotes:
> > +            add(git_repo, remotes[remote_name]['git_name'],
> > +                remotes[remote_name]['git_repo_url'])
> >          update(git_repo, remotes[remote_name]['git_name'])
> >      backports = get_backports(git_repo, remotes, branches, debug)
> >      c_b_map = kernel_sec.branch.CommitBranchMap(git_repo, remotes, branches)
> 
> Ben.
> 
> --
> Ben Hutchings, Software Developer                ?        Codethink Ltd
> https://www.codethink.co.uk/                 Dale House, 35 Dale Street
>                                      Manchester, M1 2HF, United Kingdom

[cip-dev] [cip-kernel-sec 1/2] remotes: automatically add remotes from configuration file

[Ben Hutchings]

On Tue, 2019-06-18 at 03:54 +0000, daniel.sangorrin at toshiba.co.jp wrote:
> > > > From: Ben Hutchings <ben.hutchings@codethink.co.uk>
> > On Mon, 2019-06-17 at 14:21 +0900, Daniel Sangorrin wrote:
> > > Currently the user is required to create its own remotes by
> > > hand. This should not be necessary, because the information
> > > is already collected in conf/remotes.yml. For that reason,
> > > if we detect that any remote has not been added to the
> > > local repo then we will add those.
> > 
> > I'm not yet convinced this is a good idea.
> 
> I sent a new patch that takes a different approach. The new patch will
> only check that the necessary remotes are there. I also prepared a
> new separate script, for new users to prepare the local repository.
> 
> > After this change, we would still be assuming that there is a kernel
> > repository or working tree next to kernel-config.??If we wanted to be
> > really helpful we could create a new repository if it's not there.
> 
> I didn't understand the kernel-config thing. Maybe I am missing something important.
[...]

Sorry, I meant "next to kernel-sec".

Ben.

-- 
Ben Hutchings, Software Developer                ?        Codethink Ltd
https://www.codethink.co.uk/                 Dale House, 35 Dale Street
                                     Manchester, M1 2HF, United Kingdom