[PATCH v2] Add TRACKING.IMPORTS to xen.git to more easily manage imported files that need to be kept in sync with an upstream

[PATCH v2] Add TRACKING.IMPORTS to xen.git to more easily manage imported files that need to be kept in sync with an upstream

[Lars Kurth]

Following the recent discussion, we had on IRC and the action I had in 
the March community call, this file provides a file format that 
enables writing an automated test to check whether files are out of sync. 

Once the file format is agree, I will write a test or script.

I also need some more correct test data, aka entries in the file from
committers looking after the following files
[Jan]
xen/arch/x86/cpu/mwait-idle.c 
[Stefano, Julien - this has to be finalized]
xen/drivers/passthrough/arm/smmu.c
xen/arch/arm/vgic/*
xen/include/asm-arm/div64.h
xen/drivers/char/meson-uart.c
xen/arch/arm/arm32/lib/*
xen/arch/arm/arm64/lib/*
xen/arch/arm/arm64/cache.S
xen/arch/arm/arm64/bpi.S
xen/include/asm-arm/system.h
xen/arch/arm/arm64/insn.c
[Others?]
xen/common/rbtree.c

Note that in some cases Linux has diverged and some Linux files have disappeared. 
Julien also raised the point, that in some cases only a subset of code from Linux Xen files was applied or that only some functions get moved across to Xen.

I believe that is entirely OK. The workflow would be in most cases that:
- We use a Linux (source) commit as a benchmark and record the commit ID
- If there is a change in Linux the test will fail
- The committer looks at the diff and either
  - Decides to ignore it and bumps the commit ID in this file
  - Decides the change is needed, integrates it into Xen and then 
    bumps the commit ID in this file

Changes since v1
* Require a colon after repo:, file:, ... keywords
* Replace manual:|auto: with file: as there auto: use-case was invalid
* Added more verbose description of format

Signed-off-by: Lars Kurth <lars.kurth@citrix.com>
CC: committers@xenproject.org

---
 TRACKING.FILES | 50 ++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 50 insertions(+)
 create mode 100644 TRACKING.FILES

diff --git a/TRACKING.FILES b/TRACKING.FILES
new file mode 100644
index 0000000000..3afb468ed7
--- /dev/null
+++ b/TRACKING.FILES
@@ -0,0 +1,50 @@
+# This file contains information about source files that have been
+# copied from other sources and need to be tracked
+#
+# The file may contain lines starting with ...
+# --------------------------------------------
+# version: of file format
+# repo: repository definition
+# file: a mapping to track files
+#
+# Note that repo: entries must come *before* file: entries
+#
+# Repository Definitions are of the following format
+# --------------------------------------------------
+# repo: name-of-source-repo git|svn https-url-of-source-repo
+#
+# name-of-source-repo:
+#   Name of source repository. The name will be used as reference in file:
+#   statements
+#
+# git|svn:
+#   Type ofsource repository
+#
+# https-url-of-source-repo:
+#   URL of source repository
+#
+# For example:
+#   repo: linux-torvalds git https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
+#
+# Mappings to track files are of the following format
+# ---------------------------------------------------
+# file: xen-file name-of-original-repo original-file commit-id
+#
+# xen-file:
+#   Xen file that needs to be tracked
+#
+# name-of-original-repo:
+#   A reference to a source repository defined by *repo* keyword
+#
+# original-file:
+#   File in original-repo from which we regularly want to merge changes
+#   into xen-file
+#
+# commit id:
+#   Last commit id of original-file that was deemed to be ok
+#   and either imported into the tree or rejected
+#
+# For example:
+#   file: xen/drivers/passthrough/arm/smmu.c linux-torvalds linux/drivers/iommu/arm-smmu.c b77cf11f094136
+
+version: 1
-- 
2.13.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [PATCH v2] Add TRACKING.IMPORTS to xen.git to more easily manage imported files that need to be kept in sync with an upstream

[Lars Kurth]

Following the recent discussion, we had on IRC and the action I had in 
the March community call, this file provides a file format that 
enables writing an automated test to check whether files are out of sync. 

Once the file format is agree, I will write a test or script.

I also need some more correct test data, aka entries in the file from
committers looking after the following files
[Jan]
xen/arch/x86/cpu/mwait-idle.c 
[Stefano, Julien - this has to be finalized]
xen/drivers/passthrough/arm/smmu.c
xen/arch/arm/vgic/*
xen/include/asm-arm/div64.h
xen/drivers/char/meson-uart.c
xen/arch/arm/arm32/lib/*
xen/arch/arm/arm64/lib/*
xen/arch/arm/arm64/cache.S
xen/arch/arm/arm64/bpi.S
xen/include/asm-arm/system.h
xen/arch/arm/arm64/insn.c
[Others?]
xen/common/rbtree.c

Note that in some cases Linux has diverged and some Linux files have disappeared. 
Julien also raised the point, that in some cases only a subset of code from Linux Xen files was applied or that only some functions get moved across to Xen.

I believe that is entirely OK. The workflow would be in most cases that:
- We use a Linux (source) commit as a benchmark and record the commit ID
- If there is a change in Linux the test will fail
- The committer looks at the diff and either
  - Decides to ignore it and bumps the commit ID in this file
  - Decides the change is needed, integrates it into Xen and then 
    bumps the commit ID in this file

Changes since v1
* Require a colon after repo:, file:, ... keywords
* Replace manual:|auto: with file: as there auto: use-case was invalid
* Added more verbose description of format

Signed-off-by: Lars Kurth <lars.kurth@citrix.com>
CC: committers@xenproject.org

---
 TRACKING.FILES | 50 ++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 50 insertions(+)
 create mode 100644 TRACKING.FILES

diff --git a/TRACKING.FILES b/TRACKING.FILES
new file mode 100644
index 0000000000..3afb468ed7
--- /dev/null
+++ b/TRACKING.FILES
@@ -0,0 +1,50 @@
+# This file contains information about source files that have been
+# copied from other sources and need to be tracked
+#
+# The file may contain lines starting with ...
+# --------------------------------------------
+# version: of file format
+# repo: repository definition
+# file: a mapping to track files
+#
+# Note that repo: entries must come *before* file: entries
+#
+# Repository Definitions are of the following format
+# --------------------------------------------------
+# repo: name-of-source-repo git|svn https-url-of-source-repo
+#
+# name-of-source-repo:
+#   Name of source repository. The name will be used as reference in file:
+#   statements
+#
+# git|svn:
+#   Type ofsource repository
+#
+# https-url-of-source-repo:
+#   URL of source repository
+#
+# For example:
+#   repo: linux-torvalds git https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
+#
+# Mappings to track files are of the following format
+# ---------------------------------------------------
+# file: xen-file name-of-original-repo original-file commit-id
+#
+# xen-file:
+#   Xen file that needs to be tracked
+#
+# name-of-original-repo:
+#   A reference to a source repository defined by *repo* keyword
+#
+# original-file:
+#   File in original-repo from which we regularly want to merge changes
+#   into xen-file
+#
+# commit id:
+#   Last commit id of original-file that was deemed to be ok
+#   and either imported into the tree or rejected
+#
+# For example:
+#   file: xen/drivers/passthrough/arm/smmu.c linux-torvalds linux/drivers/iommu/arm-smmu.c b77cf11f094136
+
+version: 1
-- 
2.13.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH v2] Add TRACKING.IMPORTS to xen.git to more easily manage imported files that need to be kept in sync with an upstream

[Jan Beulich]

>>> On 24.05.19 at 03:36, <lars.kurth@citrix.com> wrote:
> --- /dev/null
> +++ b/TRACKING.FILES
> @@ -0,0 +1,50 @@
> +# This file contains information about source files that have been
> +# copied from other sources and need to be tracked
> +#
> +# The file may contain lines starting with ...
> +# --------------------------------------------
> +# version: of file format
> +# repo: repository definition
> +# file: a mapping to track files
> +#
> +# Note that repo: entries must come *before* file: entries
> +#
> +# Repository Definitions are of the following format
> +# --------------------------------------------------
> +# repo: name-of-source-repo git|svn https-url-of-source-repo
> +#
> +# name-of-source-repo:
> +#   Name of source repository. The name will be used as reference in file:
> +#   statements

May I suggest another formatting change, as the colon uses now
have different meaning:

# repo: <name-of-source-repo> <type-of-source-repo> <url-of-source-repo>
#
# <name-of-source-repo>
#   Name of source repository. The name will be used as reference in file:
#   statements

> +# git|svn:
> +#   Type ofsource repository

Nit: Missing blank.

> +# https-url-of-source-repo:
> +#   URL of source repository

Why https? Any form of URL should be fine here.

> +# For example:
> +#   repo: linux-torvalds git https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git 

Didn't we agree on examples moving into the commit message,
or the post-commit-message area, as they'll become redundant
(and eventually stale) once we gain actual content here?

Jan



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH v2] Add TRACKING.IMPORTS to xen.git to more easily manage imported files that need to be kept in sync with an upstream

[Jan Beulich]

>>> On 24.05.19 at 03:36, <lars.kurth@citrix.com> wrote:
> --- /dev/null
> +++ b/TRACKING.FILES
> @@ -0,0 +1,50 @@
> +# This file contains information about source files that have been
> +# copied from other sources and need to be tracked
> +#
> +# The file may contain lines starting with ...
> +# --------------------------------------------
> +# version: of file format
> +# repo: repository definition
> +# file: a mapping to track files
> +#
> +# Note that repo: entries must come *before* file: entries
> +#
> +# Repository Definitions are of the following format
> +# --------------------------------------------------
> +# repo: name-of-source-repo git|svn https-url-of-source-repo
> +#
> +# name-of-source-repo:
> +#   Name of source repository. The name will be used as reference in file:
> +#   statements

May I suggest another formatting change, as the colon uses now
have different meaning:

# repo: <name-of-source-repo> <type-of-source-repo> <url-of-source-repo>
#
# <name-of-source-repo>
#   Name of source repository. The name will be used as reference in file:
#   statements

> +# git|svn:
> +#   Type ofsource repository

Nit: Missing blank.

> +# https-url-of-source-repo:
> +#   URL of source repository

Why https? Any form of URL should be fine here.

> +# For example:
> +#   repo: linux-torvalds git https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git 

Didn't we agree on examples moving into the commit message,
or the post-commit-message area, as they'll become redundant
(and eventually stale) once we gain actual content here?

Jan



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH v2] Add TRACKING.IMPORTS to xen.git to more easily manage imported files that need to be kept in sync with an upstream

[Lars Kurth]

On 24/05/2019, 05:24, "Jan Beulich" <JBeulich@suse.com> wrote:

    >>> On 24.05.19 at 03:36, <lars.kurth@citrix.com> wrote:
    > --- /dev/null
    > +++ b/TRACKING.FILES
    > @@ -0,0 +1,50 @@
    > +# This file contains information about source files that have been
    > +# copied from other sources and need to be tracked
    > +#
    > +# The file may contain lines starting with ...
    > +# --------------------------------------------
    > +# version: of file format
    > +# repo: repository definition
    > +# file: a mapping to track files
    > +#
    > +# Note that repo: entries must come *before* file: entries
    > +#
    > +# Repository Definitions are of the following format
    > +# --------------------------------------------------
    > +# repo: name-of-source-repo git|svn https-url-of-source-repo
    > +#
    > +# name-of-source-repo:
    > +#   Name of source repository. The name will be used as reference in file:
    > +#   statements
    
    May I suggest another formatting change, as the colon uses now
    have different meaning:
    
    # repo: <name-of-source-repo> <type-of-source-repo> <url-of-source-repo>
    #
    # <name-of-source-repo>
    #   Name of source repository. The name will be used as reference in file:
    #   statements
    
    > +# git|svn:
    > +#   Type ofsource repository
    
    Nit: Missing blank.
    
    > +# https-url-of-source-repo:
    > +#   URL of source repository
    
    Why https? Any form of URL should be fine here.
    
Sure. I think Ian suggested originally.

    > +# For example:
    > +#   repo: linux-torvalds git https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git 
    
    Didn't we agree on examples moving into the commit message,
    or the post-commit-message area, as they'll become redundant
    (and eventually stale) once we gain actual content here?
    
Ah yes, I had forgotten about this

Lars    
    
    

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH v2] Add TRACKING.IMPORTS to xen.git to more easily manage imported files that need to be kept in sync with an upstream

[Lars Kurth]

On 24/05/2019, 05:24, "Jan Beulich" <JBeulich@suse.com> wrote:

    >>> On 24.05.19 at 03:36, <lars.kurth@citrix.com> wrote:
    > --- /dev/null
    > +++ b/TRACKING.FILES
    > @@ -0,0 +1,50 @@
    > +# This file contains information about source files that have been
    > +# copied from other sources and need to be tracked
    > +#
    > +# The file may contain lines starting with ...
    > +# --------------------------------------------
    > +# version: of file format
    > +# repo: repository definition
    > +# file: a mapping to track files
    > +#
    > +# Note that repo: entries must come *before* file: entries
    > +#
    > +# Repository Definitions are of the following format
    > +# --------------------------------------------------
    > +# repo: name-of-source-repo git|svn https-url-of-source-repo
    > +#
    > +# name-of-source-repo:
    > +#   Name of source repository. The name will be used as reference in file:
    > +#   statements
    
    May I suggest another formatting change, as the colon uses now
    have different meaning:
    
    # repo: <name-of-source-repo> <type-of-source-repo> <url-of-source-repo>
    #
    # <name-of-source-repo>
    #   Name of source repository. The name will be used as reference in file:
    #   statements
    
    > +# git|svn:
    > +#   Type ofsource repository
    
    Nit: Missing blank.
    
    > +# https-url-of-source-repo:
    > +#   URL of source repository
    
    Why https? Any form of URL should be fine here.
    
Sure. I think Ian suggested originally.

    > +# For example:
    > +#   repo: linux-torvalds git https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git 
    
    Didn't we agree on examples moving into the commit message,
    or the post-commit-message area, as they'll become redundant
    (and eventually stale) once we gain actual content here?
    
Ah yes, I had forgotten about this

Lars    
    
    

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel