Golang Xen packages and the golang packaging system

Golang Xen packages and the golang packaging system

[George Dunlap]

So currently, our build system will install the xenlight package into $PREFIX/share/gocode/src/golang.xenproject.org/xenlight.  However, it actually takes a bit of wrestling to get golang to use this location, and makes it difficult to use shared code.  It would be nice if people could simply add `golang.xenproject.org/xenlight` to their dependencies, and have `go get` just DTRT.

This basically involves two things:

1. Creating a publicly-accessible suitable git repo containing the golang package(s)

2. Causing `curl golang.xenproject.org/$PKGNAME` to return some HTML with the following rune:

<meta name="go-import" content=“golang.xenproject.org git $URL”>

Where $URL points to the tree from #1.

We should probably also have some more human-friendly content in case someone wanders there with a web browser.

“Suitable git tree” means:
- That it contains just the bindings.  
- Any ‘released’ Xen version has a tag of the vA.B.C format

I think ideally we’d like the ‘master’ branch of this git tree to contain the bindings from the xenbits.xenproject.org/xen.git master branch, but that’s somewhat negotiable.

So what we’d need to do is have a process / hook somewhere which would, on push to xenbits.xenproject.org/xen.git ’s master branch:
 - Generate the bindings from the source code
 - Copy these bindings into the git repo, replacing the old bindings entirely (i.e., deleting files which don’t exist any more, adding new files)
 - Running ‘git commit’, probably with information about the commit from which this code has been generated
 - Check to see if there is a new RELEASE-X.Y.Z tag and generate an appropriate tag
 - Push to the git repo in step #1 above

This script should live in xen.git I guess.

I guess maybe it would make sense to have one git tree per package, and put them at xenbits.xenproject.org/golang/$PKGNAME? (e.g., xenbits.xenproject.org/golang/xenlight.git?)  At any rate, something that would allow us to leverage our current githttp instance on xenbits.

One question I have from the above is how the xen.git RELEASE-X.Y.Z should correspond to the vA.B.C in the golang package repo.

The obvious answer, of course, is (A, B, C) = (X, Y, Z); that is, xen.git tag RELEASE-4.14.0 should create a golang package tag of v4.14.0.

The issue with this is that golang assumes you’re using semantic versioning; so a `go get -u` would normally feel justified in upgrading from v4.14.x to v4.15.x.

A couple of possible responses:

1. Declare that OK.  That would mean not only that we have to have v4.15.x be compatible with golang source code written against v4.14; it would also mean that v4.15.x needs to be able to *compile* against libxl version 4.14.  Which might be a good idea, but we’d want to think carefully before making that kind of commitment.

2. Declare that people need to use `go get -u=patch` when updating, and/or use go.mod &c to manually restrict the version of go to use to the currently-installed Xen version

3. Map (A, B, C) = (Y, Z, 0).  (i.e., RELEASE-4.14.5 would make tag v14.5.0 .)  `go get` wouldn’t update automatically, but it might be confusing which version *should* be used; particularly if we ever roll over a major version for Xen.

Any other possibilities?

I think I’d start out with #2, and then consider moving to #1 at some point in the future.

Thoughts?

 -George

Re: Golang Xen packages and the golang packaging system

[Nick Rosbrook]

> One question I have from the above is how the xen.git RELEASE-X.Y.Z should correspond to the vA.B.C in the golang package repo.
>
> The obvious answer, of course, is (A, B, C) = (X, Y, Z); that is, xen.git tag RELEASE-4.14.0 should create a golang package tag of v4.14.0.
>
> The issue with this is that golang assumes you’re using semantic versioning; so a `go get -u` would normally feel justified in upgrading from v4.14.x to v4.15.x.
>
> A couple of possible responses:
>
> 1. Declare that OK.  That would mean not only that we have to have v4.15.x be compatible with golang source code written against v4.14; it would also mean that v4.15.x needs to be able to *compile* against libxl version 4.14.  Which might be a good idea, but we’d want to think carefully before making that kind of commitment.
>
> 2. Declare that people need to use `go get -u=patch` when updating, and/or use go.mod &c to manually restrict the version of go to use to the currently-installed Xen version
>
> 3. Map (A, B, C) = (Y, Z, 0).  (i.e., RELEASE-4.14.5 would make tag v14.5.0 .)  `go get` wouldn’t update automatically, but it might be confusing which version *should* be used; particularly if we ever roll over a major version for Xen.
>
> Any other possibilities?
>
> I think I’d start out with #2, and then consider moving to #1 at some point in the future.
>
> Thoughts?

We should also consider aligning with Go module versioning
conventions. For example, right now the package is unstable, so
according to convention we should be in v0.1.x. A "v0" indicates to
the Go ecosystem that, at this stage, we will likely make breaking
changes to the package API. So, tagging v4.14.0 is a bit confusing
since this indicates that we are on the 4th major version of the
package, and that it's stable. See [1] and [2] for more on these
conventions.

However, things are obviously complicated by the fact that the
xenlight package depends on libxl, and following convention might make
that relationship less clear and difficult to track. But, if we stray
from convention we at least need to be clear about it and have a good
explanation why.

That said, unless we can come up with a good way to follow convention
*and* keep the libxl version sorted, I think #2 makes the most sense
right now.

-NR

[1] https://blog.golang.org/publishing-go-modules
[2] https://blog.golang.org/v2-go-modules

Re: Golang Xen packages and the golang packaging system

[George Dunlap]

> On Apr 22, 2020, at 7:55 PM, Nick Rosbrook <rosbrookn@gmail.com> wrote:
> 
>> One question I have from the above is how the xen.git RELEASE-X.Y.Z should correspond to the vA.B.C in the golang package repo.
>> 
>> The obvious answer, of course, is (A, B, C) = (X, Y, Z); that is, xen.git tag RELEASE-4.14.0 should create a golang package tag of v4.14.0.
>> 
>> The issue with this is that golang assumes you’re using semantic versioning; so a `go get -u` would normally feel justified in upgrading from v4.14.x to v4.15.x.
>> 
>> A couple of possible responses:
>> 
>> 1. Declare that OK.  That would mean not only that we have to have v4.15.x be compatible with golang source code written against v4.14; it would also mean that v4.15.x needs to be able to *compile* against libxl version 4.14.  Which might be a good idea, but we’d want to think carefully before making that kind of commitment.
>> 
>> 2. Declare that people need to use `go get -u=patch` when updating, and/or use go.mod &c to manually restrict the version of go to use to the currently-installed Xen version
>> 
>> 3. Map (A, B, C) = (Y, Z, 0).  (i.e., RELEASE-4.14.5 would make tag v14.5.0 .)  `go get` wouldn’t update automatically, but it might be confusing which version *should* be used; particularly if we ever roll over a major version for Xen.
>> 
>> Any other possibilities?
>> 
>> I think I’d start out with #2, and then consider moving to #1 at some point in the future.
>> 
>> Thoughts?
> 
> We should also consider aligning with Go module versioning
> conventions. For example, right now the package is unstable, so
> according to convention we should be in v0.1.x. A "v0" indicates to
> the Go ecosystem that, at this stage, we will likely make breaking
> changes to the package API. So, tagging v4.14.0 is a bit confusing
> since this indicates that we are on the 4th major version of the
> package, and that it's stable. See [1] and [2] for more on these
> conventions.
> 
> However, things are obviously complicated by the fact that the
> xenlight package depends on libxl, and following convention might make
> that relationship less clear and difficult to track. But, if we stray
> from convention we at least need to be clear about it and have a good
> explanation why.
> 
> That said, unless we can come up with a good way to follow convention
> *and* keep the libxl version sorted, I think #2 makes the most sense
> right now.

So the way we have things right now, I don’t think #1 is actually possible: If you have Xen 4.14 installed, and you update to a 4.15 version of the generated bindings, there will be loads of code trying to marshal structural elements that don’t exist yet.

Unless we can think of a good way to have `go build` detect at build-time what’s available and compile those out, we’re always going to need a 1-1 connection between golang xenlight package and the installed Xen version (unfortunately).

Hmm… or, could we actually use the reflect package to detect it at runtime?  That’s annoying because you’re still stuck using the headers compiled in at *compile* time.

The other thing to point out is that we can always start with 0.x and then switch to 4.14 if we want to; but going the other way is going to be more difficult.

I guess we could call it v0.4.14 for now.  If we manage to make it backwards-and-forwards-compatible, we can rename it v1.0; if we decide to give up on that idea, we could rename them v4.14, v4.15, &c.

 -George

Re: Golang Xen packages and the golang packaging system

[Ian Jackson]

George Dunlap writes ("Golang Xen packages and the golang packaging  system"):
> So currently, our build system will install the xenlight package into $PREFIX/share/gocode/src/golang.xenproject.org/xenlight.  However, it actually takes a bit of wrestling to get golang to use this location, and makes it difficult to use shared code.  It would be nice if people could simply add `golang.xenproject.org/xenlight` to their dependencies, and have `go get` just DTRT.
> 
> This basically involves two things:
> 
> 1. Creating a publicly-accessible suitable git repo containing the golang package(s)
> 
> 2. Causing `curl golang.xenproject.org/$PKGNAME` to return some HTML with the following rune:
> 
> <meta name="go-import" content=“golang.xenproject.org git $URL”>
> 
> Where $URL points to the tree from #1.
> 
> We should probably also have some more human-friendly content in case someone wanders there with a web browser.
> 
> “Suitable git tree” means:
> - That it contains just the bindings.  
...
> So what we’d need to do is have a process / hook somewhere which would, on push to xenbits.xenproject.org/xen.git ’s master branch:
>  - Generate the bindings from the source code
>  - Copy these bindings into the git repo, replacing the old bindings entirely (i.e., deleting files which don’t exist any more, adding new files)
>  - Running ‘git commit’, probably with information about the commit from which this code has been generated
>  - Check to see if there is a new RELEASE-X.Y.Z tag and generate an appropriate tag
>  - Push to the git repo in step #1 above

This is quite unpleasant.  In particular, it makes a git tree out of
output files.  What will we do when someone sends us patches to the
bindings ?

Can we not instead provide some metadata at the top level of xen.git
which tells golang how to run enough of our build system to build the
needed .go files ?

Ian.

Re: Golang Xen packages and the golang packaging system

[Ian Jackson]

Ian Jackson writes ("Re: Golang Xen packages and the golang packaging  system"):
> This is quite unpleasant.  In particular, it makes a git tree out of
> output files.  What will we do when someone sends us patches to the
> bindings ?

Also, anyone who redistributes your proposed golang package is
violating our licence unless they ship a copy of xen.git[1] too, since
the golang package is not source code.

[1] Technically, a copy of the relevant parts will do.

Ian.

Re: Golang Xen packages and the golang packaging system

[George Dunlap]

> On Apr 23, 2020, at 12:27 PM, Ian Jackson <ian.jackson@citrix.com> wrote:
> 
> Ian Jackson writes ("Re: Golang Xen packages and the golang packaging  system"):
>> This is quite unpleasant.  In particular, it makes a git tree out of
>> output files.  What will we do when someone sends us patches to the
>> bindings ?
> 
> Also, anyone who redistributes your proposed golang package is
> violating our licence unless they ship a copy of xen.git[1] too, since
> the golang package is not source code.
> 
> [1] Technically, a copy of the relevant parts will do.

The “relevant parts” would primarily be gengotypes.py, right?  Oh, and I guess libxl_test.idl and friends.  libxl_test.idl isn’t included in the distribution either.

I’m not an expert in the golang build system, but they generally seem to be trying to keep the functionality simple (which of course, means if you want to do anything non-basic, it’s incredibly complicated or completely impossible).

There’s a command, `go generate`, which we could use to run gengotypes.py to generate the appropriate files.  But I’m not sure how to use that in a practical way for this sort of package: it might end up that people wanting to use the package would need to manually clone it, then manually run `go generate` before manually building the package.

Checking in the generated files means that someone can simply add `golang.xenproject.org/xenlight` as a dependency (perhaps with a specific version tag, like v4.14), and everything Just Works.

Nick may have some ideas on how to use the golang build system more effectively.

 -George

Re: Golang Xen packages and the golang packaging system

[George Dunlap]

> On Apr 23, 2020, at 12:49 PM, George Dunlap <george.dunlap@citrix.com> wrote:
> 
> 
> 
>> On Apr 23, 2020, at 12:27 PM, Ian Jackson <ian.jackson@citrix.com> wrote:
>> 
>> Ian Jackson writes ("Re: Golang Xen packages and the golang packaging  system"):
>>> This is quite unpleasant.  In particular, it makes a git tree out of
>>> output files.  What will we do when someone sends us patches to the
>>> bindings ?
>> 
>> Also, anyone who redistributes your proposed golang package is
>> violating our licence unless they ship a copy of xen.git[1] too, since
>> the golang package is not source code.
>> 
>> [1] Technically, a copy of the relevant parts will do.
> 
> The “relevant parts” would primarily be gengotypes.py, right?  Oh, and I guess libxl_test.idl and friends.  libxl_test.idl isn’t included in the distribution either.
> 
> I’m not an expert in the golang build system, but they generally seem to be trying to keep the functionality simple (which of course, means if you want to do anything non-basic, it’s incredibly complicated or completely impossible).
> 
> There’s a command, `go generate`, which we could use to run gengotypes.py to generate the appropriate files.  But I’m not sure how to use that in a practical way for this sort of package: it might end up that people wanting to use the package would need to manually clone it, then manually run `go generate` before manually building the package.
> 
> Checking in the generated files means that someone can simply add `golang.xenproject.org/xenlight` as a dependency (perhaps with a specific version tag, like v4.14), and everything Just Works.
> 
> Nick may have some ideas on how to use the golang build system more effectively.

So, the following seems to work quite well actually:

mkdir vendor
ln -s vendor/golang.xenproject.org /usr/share/gocode/src/golang.xenproject.org
echo “golang.xenproject.org/xenlight” >> vendor/modules.txt
go build -mod=vendor

Using the above method, (say) redctl.git would build exactly the same on Xen 4.14 as on Xen 4.15 (assuming redctl wasn’t using anything not available in 4.14).

I’m inclined to say we should start with just telling people to do that, and look at doing something else if we discover that’s not suitable for some reason.

 -George

Re: Golang Xen packages and the golang packaging system

[Nick Rosbrook]

On Thu, Apr 23, 2020 at 1:22 PM George Dunlap <George.Dunlap@citrix.com> wrote:
>
>
> > On Apr 23, 2020, at 12:49 PM, George Dunlap <george.dunlap@citrix.com> wrote:
> >
> >
> >
> >> On Apr 23, 2020, at 12:27 PM, Ian Jackson <ian.jackson@citrix.com> wrote:
> >>
> >> Ian Jackson writes ("Re: Golang Xen packages and the golang packaging  system"):
> >>> This is quite unpleasant.  In particular, it makes a git tree out of
> >>> output files.  What will we do when someone sends us patches to the
> >>> bindings ?
> >>
> >> Also, anyone who redistributes your proposed golang package is
> >> violating our licence unless they ship a copy of xen.git[1] too, since
> >> the golang package is not source code.
> >>
> >> [1] Technically, a copy of the relevant parts will do.
> >
> > The “relevant parts” would primarily be gengotypes.py, right?  Oh, and I guess libxl_test.idl and friends.  libxl_test.idl isn’t included in the distribution either.
> >
> > I’m not an expert in the golang build system, but they generally seem to be trying to keep the functionality simple (which of course, means if you want to do anything non-basic, it’s incredibly complicated or completely impossible).
> >
> > There’s a command, `go generate`, which we could use to run gengotypes.py to generate the appropriate files.  But I’m not sure how to use that in a practical way for this sort of package: it might end up that people wanting to use the package would need to manually clone it, then manually run `go generate` before manually building the package.
> >
> > Checking in the generated files means that someone can simply add `golang.xenproject.org/xenlight` as a dependency (perhaps with a specific version tag, like v4.14), and everything Just Works.
> >
> > Nick may have some ideas on how to use the golang build system more effectively.
>
> So, the following seems to work quite well actually:
>
> mkdir vendor
> ln -s vendor/golang.xenproject.org /usr/share/gocode/src/golang.xenproject.org
> echo “golang.xenproject.org/xenlight” >> vendor/modules.txt
> go build -mod=vendor
>
> Using the above method, (say) redctl.git would build exactly the same on Xen 4.14 as on Xen 4.15 (assuming redctl wasn’t using anything not available in 4.14).
>
> I’m inclined to say we should start with just telling people to do that, and look at doing something else if we discover that’s not suitable for some reason.

If it's not viable to create another repo for the xenlight package, I
think we should should just initialize the go module, i.e. go.mod, at
xen.git/tools/golang. The downside is that tags cannot be independent
from the rest of xen.git, so users need to have `require <module
path>/xenlight@RELEASE-4.14.0` in their go.mod, but at least its `go
get`-able. And, this does not fetch the entire git tree.

This would also mean that we actually track the generated code (which
isn't really a big deal IMO, it's expected that people track their
generated gRPC code, for example).

I spent some time today trying to get `go get <vanity URL>/xenlight`
to work, after defining the go.mod (with <vanity URL>/xenlight as the
import path) in tools/golang. I *think* it's do-able with a meta-tag
like: `<meta name="go-import" content="<vanity URL>/xenlight mod
https://proxy.golang.org">`, but I was having a hard time getting the
package to show up on pkg.go.dev or proxy.golang.org. I'm not totally
familiar with the module proxies yet, but it seems like it could be a
way to detach ourselves from the "package is defined at the root of a
git repo" problem.

-NR

Re: Golang Xen packages and the golang packaging system

[George Dunlap]

> On Apr 24, 2020, at 5:04 AM, Nick Rosbrook <rosbrookn@gmail.com> wrote:
> 
> On Thu, Apr 23, 2020 at 1:22 PM George Dunlap <George.Dunlap@citrix.com> wrote:
>> 
>> 
>>> On Apr 23, 2020, at 12:49 PM, George Dunlap <george.dunlap@citrix.com> wrote:
>>> 
>>> 
>>> 
>>>> On Apr 23, 2020, at 12:27 PM, Ian Jackson <ian.jackson@citrix.com> wrote:
>>>> 
>>>> Ian Jackson writes ("Re: Golang Xen packages and the golang packaging system"):
>>>>> This is quite unpleasant.  In particular, it makes a git tree out of
>>>>> output files.  What will we do when someone sends us patches to the
>>>>> bindings ?
>>>> 
>>>> Also, anyone who redistributes your proposed golang package is
>>>> violating our licence unless they ship a copy of xen.git[1] too, since
>>>> the golang package is not source code.
>>>> 
>>>> [1] Technically, a copy of the relevant parts will do.
>>> 
>>> The “relevant parts” would primarily be gengotypes.py, right?  Oh, and I guess libxl_test.idl and friends.  libxl_test.idl isn’t included in the distribution either.
>>> 
>>> I’m not an expert in the golang build system, but they generally seem to be trying to keep the functionality simple (which of course, means if you want to do anything non-basic, it’s incredibly complicated or completely impossible).
>>> 
>>> There’s a command, `go generate`, which we could use to run gengotypes.py to generate the appropriate files.  But I’m not sure how to use that in a practical way for this sort of package: it might end up that people wanting to use the package would need to manually clone it, then manually run `go generate` before manually building the package.
>>> 
>>> Checking in the generated files means that someone can simply add `golang.xenproject.org/xenlight` as a dependency (perhaps with a specific version tag, like v4.14), and everything Just Works.
>>> 
>>> Nick may have some ideas on how to use the golang build system more effectively.
>> 
>> So, the following seems to work quite well actually:
>> 
>> mkdir vendor
>> ln -s vendor/golang.xenproject.org /usr/share/gocode/src/golang.xenproject.org
>> echo “golang.xenproject.org/xenlight” >> vendor/modules.txt
>> go build -mod=vendor
>> 
>> Using the above method, (say) redctl.git would build exactly the same on Xen 4.14 as on Xen 4.15 (assuming redctl wasn’t using anything not available in 4.14).
>> 
>> I’m inclined to say we should start with just telling people to do that, and look at doing something else if we discover that’s not suitable for some reason.
> 
> If it's not viable to create another repo for the xenlight package, I
> think we should should just initialize the go module, i.e. go.mod, at
> xen.git/tools/golang. The downside is that tags cannot be independent
> from the rest of xen.git, so users need to have `require <module
> path>/xenlight@RELEASE-4.14.0` in their go.mod, but at least its `go
> get`-able. And, this does not fetch the entire git tree.
> 
> This would also mean that we actually track the generated code (which
> isn't really a big deal IMO, it's expected that people track their
> generated gRPC code, for example).

Yes, I was playing with this yesterday and it seems to work OK.

The thing I didn’t necessarily like about this was that suppose you had a public project that used the xenlight bindings, and you upgraded to Xen 4.15, but some of your users hadn’t.  If you updated this to RELEASE-4.15.0, then all your downstreams would stop working, even if you weren’t using any functionality specific to Xen 4.15.

But I suppose what that would really mean is that:
1) We should make sure that xenlight@RELEASE-$V works on > $V as well
2) Projects depending on the bindings should use the oldest version of the Xen bindings suitable for their use case.

Both of those are probably reasonable.

Another issue that happens with checking in generated code is that the idl changes and nobody re-generates the code.  We’d probably want an osstest check that would refuse to push from staging -> master if re-running the code generator produced a different output.  (But that has its own annoyances: it seems that different versions of python sort things in different orders, so I often have to throw away spurious changes to the generated files because our two versions of python seem to order some things differently.)

BTW the separate repo isn’t off the table.  But there were some things other Ian pointed out:

1. The GPL requires that you provide the “preferred form for modification” to all the code.  I’m not sure this has been adjudicated in court, but there’s a strong argument that *generated* code doesn’t match that criteria: that to satisfy the GPL you’d need to include libxl_types.idl, idl.py, gengotypes.py, and a Makefile suitable for tying them all together.  (Not that the generation needs to be run with `go build`, but that ideally the infrastructure would be there so that it *could* be run.)

2. Ian was concerned with how someone using the bindings would submit a patch upstream.  Suppose someone cloned our “bindings” repo, made some changes so that it worked for them, then wanted to submit the patch upstream.  How would they do that?

I suppose even if we do a “deep link” into our actual repo, it won’t necessarily be obvious to people how to modify things and submit patches.  We’d probably want a how-to.

Anyway, do you want to submit a patch adding a `go.mod` in the appropriate place?  I’ve always had a hard time figuring out how go.mod actually works; there seems to be no *manual*, only *howtos*.

 -George

Re: Golang Xen packages and the golang packaging system

[Nick Rosbrook]

On Fri, Apr 24, 2020 at 7:26 AM George Dunlap <George.Dunlap@citrix.com> wrote:
>
>
>
> > On Apr 24, 2020, at 5:04 AM, Nick Rosbrook <rosbrookn@gmail.com> wrote:
> >
> > On Thu, Apr 23, 2020 at 1:22 PM George Dunlap <George.Dunlap@citrix.com> wrote:
> >>
> >>
> >>> On Apr 23, 2020, at 12:49 PM, George Dunlap <george.dunlap@citrix.com> wrote:
> >>>
> >>>
> >>>
> >>>> On Apr 23, 2020, at 12:27 PM, Ian Jackson <ian.jackson@citrix.com> wrote:
> >>>>
> >>>> Ian Jackson writes ("Re: Golang Xen packages and the golang packaging system"):
> >>>>> This is quite unpleasant.  In particular, it makes a git tree out of
> >>>>> output files.  What will we do when someone sends us patches to the
> >>>>> bindings ?
> >>>>
> >>>> Also, anyone who redistributes your proposed golang package is
> >>>> violating our licence unless they ship a copy of xen.git[1] too, since
> >>>> the golang package is not source code.
> >>>>
> >>>> [1] Technically, a copy of the relevant parts will do.
> >>>
> >>> The “relevant parts” would primarily be gengotypes.py, right?  Oh, and I guess libxl_test.idl and friends.  libxl_test.idl isn’t included in the distribution either.
> >>>
> >>> I’m not an expert in the golang build system, but they generally seem to be trying to keep the functionality simple (which of course, means if you want to do anything non-basic, it’s incredibly complicated or completely impossible).
> >>>
> >>> There’s a command, `go generate`, which we could use to run gengotypes.py to generate the appropriate files.  But I’m not sure how to use that in a practical way for this sort of package: it might end up that people wanting to use the package would need to manually clone it, then manually run `go generate` before manually building the package.
> >>>
> >>> Checking in the generated files means that someone can simply add `golang.xenproject.org/xenlight` as a dependency (perhaps with a specific version tag, like v4.14), and everything Just Works.
> >>>
> >>> Nick may have some ideas on how to use the golang build system more effectively.
> >>
> >> So, the following seems to work quite well actually:
> >>
> >> mkdir vendor
> >> ln -s vendor/golang.xenproject.org /usr/share/gocode/src/golang.xenproject.org
> >> echo “golang.xenproject.org/xenlight” >> vendor/modules.txt
> >> go build -mod=vendor
> >>
> >> Using the above method, (say) redctl.git would build exactly the same on Xen 4.14 as on Xen 4.15 (assuming redctl wasn’t using anything not available in 4.14).
> >>
> >> I’m inclined to say we should start with just telling people to do that, and look at doing something else if we discover that’s not suitable for some reason.
> >
> > If it's not viable to create another repo for the xenlight package, I
> > think we should should just initialize the go module, i.e. go.mod, at
> > xen.git/tools/golang. The downside is that tags cannot be independent
> > from the rest of xen.git, so users need to have `require <module
> > path>/xenlight@RELEASE-4.14.0` in their go.mod, but at least its `go
> > get`-able. And, this does not fetch the entire git tree.
> >
> > This would also mean that we actually track the generated code (which
> > isn't really a big deal IMO, it's expected that people track their
> > generated gRPC code, for example).
>
> Yes, I was playing with this yesterday and it seems to work OK.
>
> The thing I didn’t necessarily like about this was that suppose you had a public project that used the xenlight bindings, and you upgraded to Xen 4.15, but some of your users hadn’t.  If you updated this to RELEASE-4.15.0, then all your downstreams would stop working, even if you weren’t using any functionality specific to Xen 4.15.

The go.mod is really just a way of specifying dependencies. Definig a
module for xenlight does not take away the ability of downstreams to
do `go build -mod=vendor`, where their preferred version of xenlight
is vendored.

> But I suppose what that would really mean is that:
> 1) We should make sure that xenlight@RELEASE-$V works on > $V as well
> 2) Projects depending on the bindings should use the oldest version of the Xen bindings suitable for their use case.
>
> Both of those are probably reasonable.

I agree.

> Another issue that happens with checking in generated code is that the idl changes and nobody re-generates the code.  We’d probably want an osstest check that would refuse to push from staging -> master if re-running the code generator produced a different output.  (But that has its own annoyances: it seems that different versions of python sort things in different orders, so I often have to throw away spurious changes to the generated files because our two versions of python seem to order some things differently.)

Yeah, I have noticed that on my machine as well. I guess we could just
sort the cases by value or alphabetically before writing the switch
statements. Have you seen the re-ordering happen in other places too?

[...]

> Anyway, do you want to submit a patch adding a `go.mod` in the appropriate place?  I’ve always had a hard time figuring out how go.mod actually works; there seems to be no *manual*, only *howtos*.

Yes, I can do that. I'm going to keep tinkering with the remote import
path and module proxy so that users could still just do `import
"golang.xenproject.org/xenlight"`, but if I can't get something like
that working I'll define the module "canonically."

Thanks,
-NR

Re: Golang Xen packages and the golang packaging system

[Nick Rosbrook]

> BTW the separate repo isn’t off the table.  But there were some things other Ian pointed out:

After trying (and failing) to get a go module with a remote import
path like `golang.xenproject.org/xenlight` defined in xen.git, I would
like to circle back to the separate repo.

In theory, modules are not supposed to be tightly coupled with vcs,
but in practice they are. It seems like you cannot define a module
with a remote import path without pointing to the repo *root*. I tried
many ways, but every attempt resulted in `go get enr0n.net/xenlight`
downloading the root of xen.git as the "module", while the actual
module in tools/golang/xenlight was excluded since it was seen as a
nested go module. If you want to get around these issues, I think you
would need to define your own module proxy. Or, we would need to just
give up on the "vanity" URL and use
`xenbits.xen.org/git-http/xen.git/tools/golang/xenlight` as the import
path.

For reference, I did get a test module `enr0n.net/xenlight` that
points to github.com/enr0n/xenlight working. That part is trivial if
you are able to point to the repo root.

Overall, between fighting with Go modules, tagging versions, and
making sure code is re-generated on IDL changes, it seems to me that
there are more negatives with putting the module in xen.git than there
are with putting the module in its own repo.

> 1. The GPL requires that you provide the “preferred form for modification” to all the code.  I’m not sure this has been adjudicated in court, but there’s a strong argument that *generated* code doesn’t match that criteria: that to satisfy the GPL you’d need to include libxl_types.idl, idl.py, gengotypes.py, and a Makefile suitable for tying them all together.  (Not that the generation needs to be run with `go build`, but that ideally the infrastructure would be there so that it *could* be run.)

Is there anything involved in solving this issue besides making sure
those files are copied to the repo in addition to the generated go
files? Or is there some concern in doing so?

> 2. Ian was concerned with how someone using the bindings would submit a patch upstream.  Suppose someone cloned our “bindings” repo, made some changes so that it worked for them, then wanted to submit the patch upstream.  How would they do that?

I think we could mostly solve this with a good README explaining what
to do. It's not uncommon to see (a) generated Go code with // DO NOT
EDIT at the top, and (b) repos (mirrors) on github that do not accept
PRs. Or am I oversimplifying?

Thanks,
-NR