Migrating custom qemu.org infrastructure to GitLab

Migrating custom qemu.org infrastructure to GitLab

[Stefan Hajnoczi]

Dear QEMU community,
QEMU currently has a static website, wiki, git repo hosting, and
special-purpose cronjobs/containers running in VMs. There is currently
no system administrator looking after our infrastructure so the most
urgent tasks fall onto me, the remainder are ignored/postponed. The
current situation exposes qemu.org to the risk of downtime and
security issues.

Another limitation is that each piece of infrastructure is managed
separately and one-time contributors cannot easily propose changes
because they do not have access. It would be much better to use our
existing code review process so that anyone can make changes to
infrastructure by sending a patch.

GitLab's Continuous Integration (CI) system provides a powerful way to
perform actions defined in yaml files in qemu.git. This includes
running scripts, builds, publishing build artifacts, etc. We have
already begun using it for automated builds and tests:
https://gitlab.com/qemu-project/qemu/-/blob/master/.gitlab-ci.yml

GitLab also offers git repo hosting, wikis, issue tracking, and other
features. It is possible to log in using GitHub, Google, or Twitter
single sign-on if you do not want to create another account. As more
open source projects use GitLab it becomes easier for one-time
contributors who will already be familiar with the tools from other
projects.

Here is a full list of GitLab's features:
https://about.gitlab.com/features/

GitLab offers the gold/ultimate tier for free to open source projects:
https://about.gitlab.com/solutions/open-source/

GitLab itself is open source and can be self-hosted if we decide to
leave in the future.

With this in mind I propose moving qemu.org infrastructure to GitLab
incrementally. This needs to be done carefully to avoid disruption and
only where GitLab meets the requirements. The QEMU project will
continue to have access to cloud hosting for running custom
infrastructure or adding runners to GitLab CI to improve CI
performance.

The following infrastructure components can be considered for GitLab migration:

1. qemu-web.git static site generation. GitLab CI/CD can build the
static website on each qemu-web.git commit and publish the HTML
artifacts.

2. wiki.qemu.org is a MediaWiki instance. Account creation is a hurdle
to one-time or new contributors. It is unclear whether GitLab's wiki
is expressive enough for a lossless conversion of the existing QEMU
wiki. Any volunteers interested in evaluating the wiki migration would
be appreciated.

3. Git repo hosting is a core feature of GitLab and we already have a
qemu.git mirror. Hosting the repos on GitLab reduces the need for
qemu.org ssh access.

4. The QEMU release process can be moved to CI/CD so that publishing
stable releases and release candidates is less dependent on one
committer's machine or scripts.

5. Issue tracking. Launchpad more or less works, but the login always
bothers me. If we move git repo hosting then it makes sense to do
issue tracking on GitLab too.

There is a snowball effect where the experience is improved the more
GitLab features we use, so I hope that most of these migrations will
be possible.

Next steps:
 * If you have an interest in one or more of these infrastructure
components, please join the discussion.
 * If there are no volunteers for an infrastructure component I'll
slowly work my way through evaluating GitLab and propose migrations

Stefan

Re: Migrating custom qemu.org infrastructure to GitLab

[Daniel P. Berrangé]

On Wed, Jul 08, 2020 at 10:52:38AM +0100, Stefan Hajnoczi wrote:
> Dear QEMU community,
> QEMU currently has a static website, wiki, git repo hosting, and
> special-purpose cronjobs/containers running in VMs. There is currently
> no system administrator looking after our infrastructure so the most
> urgent tasks fall onto me, the remainder are ignored/postponed. The
> current situation exposes qemu.org to the risk of downtime and
> security issues.

I would further say that maintaining infrastructure is not a good use
of our contributors' time. It is better to have our contributors spend
time delivering features that are beneficial to our users, rather than
playing at sysadmin.

> Another limitation is that each piece of infrastructure is managed
> separately and one-time contributors cannot easily propose changes
> because they do not have access. It would be much better to use our
> existing code review process so that anyone can make changes to
> infrastructure by sending a patch.

When I was thinking about this for libvirt I came to better understand
the appeal of the gitforge for project infrastructure. I've written
about it in detail for libvirt:

   https://www.redhat.com/archives/libvir-list/2020-March/msg00009.html

The distributed SCM model of git democratized access to source control
features across contributors. Drive-by contributors have the same level
of SCM features as the main project contributors, instead of being
second class citizens as they are with a non-distributed SCM.

With the hosted Git forges, this concept covers all aspects of project
infrastructure. If your project is using GitLab for all its infra
needs, then when a user forks the repo they get access to all the
same features as the main project. The fork is no longer a second
class citizen lacking access to certain parts of the infra. I find
this is especially compelling in the area of CI, containers and web
hosting.

Consider if qemu-web.git is hosted on gitlab, using GitLab CI to
generate the static site content, and GitLab pages to host the
website. If a user wants to contribute to qemu-web, they can fork
the repo and be confident that the CI jobs in their fork are going
to work in the exact same way as in the master repo. They can use
GitLab Pages to browse the generated content for their fork to
see exactly how it will look.

This eliminates the big pain point in qemu-web contribution. Many
times I was tripped up with the problem of qemu-web.git requiring
a version of Jekyll that is incompatible with the version that
ships on my distro.

> GitLab's Continuous Integration (CI) system provides a powerful way to
> perform actions defined in yaml files in qemu.git. This includes
> running scripts, builds, publishing build artifacts, etc. We have
> already begun using it for automated builds and tests:
> https://gitlab.com/qemu-project/qemu/-/blob/master/.gitlab-ci.yml

The CI integration has probably been the single best thing about
libvirt's switch to GitLab.

> GitLab also offers git repo hosting, wikis, issue tracking, and other
> features. It is possible to log in using GitHub, Google, or Twitter
> single sign-on if you do not want to create another account. As more
> open source projects use GitLab it becomes easier for one-time
> contributors who will already be familiar with the tools from other
> projects.

> Here is a full list of GitLab's features:
> https://about.gitlab.com/features/
> 
> GitLab offers the gold/ultimate tier for free to open source projects:
> https://about.gitlab.com/solutions/open-source/
> 
> GitLab itself is open source and can be self-hosted if we decide to
> leave in the future.

NB self-hosting would reduce features compared to gitlab.com unless
you paid for the enterprise edition. You'd also have to bring your
own infra for running CI jobs which quickly gets very expensive
as freedesktop.org admins have learnt the hard way. The CI resources
provided by gitlab.com are incredibly generous.

> With this in mind I propose moving qemu.org infrastructure to GitLab
> incrementally. This needs to be done carefully to avoid disruption and
> only where GitLab meets the requirements. The QEMU project will
> continue to have access to cloud hosting for running custom
> infrastructure or adding runners to GitLab CI to improve CI
> performance.
> 
> The following infrastructure components can be considered for GitLab migration:
> 
> 1. qemu-web.git static site generation. GitLab CI/CD can build the
> static website on each qemu-web.git commit and publish the HTML
> artifacts.

This is an easy win. it should be especially simple for qemu, compared to
libvirt, since qemu.org already used separate sub-domains for other content
like git & downloads.

> 2. wiki.qemu.org is a MediaWiki instance. Account creation is a hurdle
> to one-time or new contributors. It is unclear whether GitLab's wiki
> is expressive enough for a lossless conversion of the existing QEMU
> wiki. Any volunteers interested in evaluating the wiki migration would
> be appreciated.

Yeah, this is a potentially big piece of work. We didn't finish this
in libvirt either. Looking at the libvirt mediawiki though, I decided
not todo a straight export/import of all content.

A bunch of content was horribly out of date and needed deleting as
it was irrelevant to anyone, or outright misleading wrt current state
of the world.

A lot of content on the wiki really only existed in the wiki, because
contributing to our libvirt.org website was harder than it needed to
be. We switched the main website over to use RST instead of HTML so
authoring content is much closer to what you'd experience with the
wiki markup. The wiki became a burden for contributors as due to
spam attacks we can't allow anonymous access, and need manual account
creation as with QEMU wiki.

Every page on the main website now has a link "edit this page"
that takes you straight to the source file in the gitlab.com UI,
where you can interactively edit the text in your fork and then
submit a merge request with the result.

The wiki also suffered from the content not having any level of review.
While the intentions of many contributors were good, the results often
left alot to be desired. 

So overall we decided that while GitLab does offer a wiki, we are not
intending to use it. Instead we're going to aim to turn the subset of
wiki content that is genuinely useful into pages on the main website.

IOW, I'd encourage whoever tackles the wiki to look at the content
with a critical eye and decide whether it really belongs in the wiki
at all. Considering:

  https://wiki.qemu.org/index.php?title=Special:AllPages

The Planning / Changes pages for each release are probably reasonable
as wiki pages.

The Interships/ProjectIdeas pages would possibly be better off as
GitLab Issue tickets, tagged with a "interships" label. That way
when a project starts you can have the issue to track the work
right from the start.

The Documentation/Platforms content arguably should be part of
the main qemu.git docs.

Many of the feature pages are probably better as part of the formal
QEMU documentation too.

> 3. Git repo hosting is a core feature of GitLab and we already have a
> qemu.git mirror. Hosting the repos on GitLab reduces the need for
> qemu.org ssh access.

With libvirt we had long used both gitlab.com/libvirt and github.com/libvirt
as read-only mirrors of the master libvirt.org/git/ repos.  When we formally
adopted gitlab as primary infrastructure, we simply flipped the mirroring
processes around. So now libvirt.org/git is a read-only mirror, and gitlab
is the master.  Gitlab has built-in support to push to github.com as a
mirror which is useful.

So turning git.qemu.org into a readonly mirror would ease the transition
to avoid breaking any scripts / workflows relying on that URL in the short
term.

> 4. The QEMU release process can be moved to CI/CD so that publishing
> stable releases and release candidates is less dependent on one
> committer's machine or scripts.
> 
> 5. Issue tracking. Launchpad more or less works, but the login always
> bothers me. If we move git repo hosting then it makes sense to do
> issue tracking on GitLab too.

The big thing that always bothers me about launchpad is how easy it
is to get confused between issues for QEMU upstream and issues for
legacy releases in Ubuntu distro. This is the curse of using a distro
issue tracker - libvirt had the same problem with its use of Red Hat
bugzilla.

Integrated issue tracking alongside the primary git repo makes a lot
of sense. Once you declare gitlab.com the primary repo, contributors
will immediately look to gitlab for the issue tracking too whether
you tell them to or not. 

The gitlab issue tracker is fairly simple especially compared to the
complexity horror of red hat bugzilla, so it was a massive win for
libvirt to switch away from bugzilla for issues. Probably not such
a big difference compared to launchpad in terms of complexity. The
little things though like magic git commit messages to close issues
are nice though.

There is a question of what todo with existing bugs in launchpad.

Essentially three choices

 1. Move all the open bugs to gitlab
 2. Move some relevant bugs to gitlab, but close outdated ones
 3. Leave existing launchpad bugs but don't allow new ones filed

So far for libvirt we did option 3 there, with a little bit of
option 2.

> There is a snowball effect where the experience is improved the more
> GitLab features we use, so I hope that most of these migrations will
> be possible.
> 
> Next steps:
>  * If you have an interest in one or more of these infrastructure
> components, please join the discussion.
>  * If there are no volunteers for an infrastructure component I'll
> slowly work my way through evaluating GitLab and propose migrations

The experiance of moving infra to gitlab.com has been quite positive
for libvirt, and me in particular as the single point of failure for
much of libvirt's old infrastructure. I think it will be beneficial
for QEMU too, for many of the reasons I mention above and in the
email link I quoted.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Migrating custom qemu.org infrastructure to GitLab

[Paolo Bonzini]

On 08/07/20 12:53, Daniel P. Berrangé wrote:
> Consider if qemu-web.git is hosted on gitlab, using GitLab CI to
> generate the static site content, and GitLab pages to host the
> website. If a user wants to contribute to qemu-web, they can fork
> the repo and be confident that the CI jobs in their fork are going
> to work in the exact same way as in the master repo. They can use
> GitLab Pages to browse the generated content for their fork to
> see exactly how it will look.
> 
> This eliminates the big pain point in qemu-web contribution. Many
> times I was tripped up with the problem of qemu-web.git requiring
> a version of Jekyll that is incompatible with the version that
> ships on my distro.

I would have no issue with using a pull request workflow for qemu-web.
The lack of "git range-diff" functionality for gitlab is an absolute
showstopper for using it in QEMU, though.

>> GitLab's Continuous Integration (CI) system provides a powerful way to
>> perform actions defined in yaml files in qemu.git. This includes
>> running scripts, builds, publishing build artifacts, etc. We have
>> already begun using it for automated builds and tests:
>> https://gitlab.com/qemu-project/qemu/-/blob/master/.gitlab-ci.yml
> 
> The CI integration has probably been the single best thing about
> libvirt's switch to GitLab.

How do you handle non-x86 platforms?  Has there been any progress in
gitlab runner support for s390 and PPC?

> The Documentation/Platforms content arguably should be part of
> the main qemu.git docs.
> 
> Many of the feature pages are probably better as part of the formal
> QEMU documentation too.

Yes, definitely; but someone has to do the work.  At least the obsolete
features are clearly marked as so.

Re: Migrating custom qemu.org infrastructure to GitLab

[Daniel P. Berrangé]

On Wed, Jul 08, 2020 at 01:17:11PM +0200, Paolo Bonzini wrote:
> On 08/07/20 12:53, Daniel P. Berrangé wrote:
> > Consider if qemu-web.git is hosted on gitlab, using GitLab CI to
> > generate the static site content, and GitLab pages to host the
> > website. If a user wants to contribute to qemu-web, they can fork
> > the repo and be confident that the CI jobs in their fork are going
> > to work in the exact same way as in the master repo. They can use
> > GitLab Pages to browse the generated content for their fork to
> > see exactly how it will look.
> > 
> > This eliminates the big pain point in qemu-web contribution. Many
> > times I was tripped up with the problem of qemu-web.git requiring
> > a version of Jekyll that is incompatible with the version that
> > ships on my distro.
> 
> I would have no issue with using a pull request workflow for qemu-web.
> The lack of "git range-diff" functionality for gitlab is an absolute
> showstopper for using it in QEMU, though.

FWIW, I tried to avoid mentioning merge requests, as I didn't want
to derail the discussion & believe use of gitlab for infrastructure
is worthwhile even if you never use merge requests.  I do tend to
think that merge requests would be a reasonable thing to use for
qemu-web in particular.

> >> GitLab's Continuous Integration (CI) system provides a powerful way to
> >> perform actions defined in yaml files in qemu.git. This includes
> >> running scripts, builds, publishing build artifacts, etc. We have
> >> already begun using it for automated builds and tests:
> >> https://gitlab.com/qemu-project/qemu/-/blob/master/.gitlab-ci.yml
> > 
> > The CI integration has probably been the single best thing about
> > libvirt's switch to GitLab.
> 
> How do you handle non-x86 platforms?  Has there been any progress in
> gitlab runner support for s390 and PPC?

We don't do any native non-x86 stuff right now.  We do all non-x86
builds using cross-compilers. Obviously that means we don't get any
testing coverage, only build coverage.

We did update our lcitool to be able to build VMs to serve as gitlab
runners, principally to provide us FreeBSD runners. After doing all
the work though, we discovered a way to integrate Cirrus CI into
GitLab CI, using a shim called "cirrus-run".  So now we have macOS
and FreeBSD jobs in GitLab CI, that use a shim to call out to Cirrus
CI todo the actual work:

From this pipeline:

  https://gitlab.com/libvirt/libvirt/-/pipelines/164291392

These two jobs actually ran on Cirrus CI:

  https://gitlab.com/libvirt/libvirt/-/jobs/628816093
  https://gitlab.com/libvirt/libvirt/-/jobs/628816086

We don't get a nice build log output in gitlab but that's just a
limitation of the current cirrus-run impl - we could easily enhance
it to pull the buld log out of Cirrus CI, so it is immediately
visible in GitLab logs.

IOW, the cirrus-run hack shows that even if you can't get a native runner
integration for GitLab CI, you can always use a shim to call out to pretty
much any external system you desire. So that provides a backup plan for
non-x86 CI support.

> > The Documentation/Platforms content arguably should be part of
> > the main qemu.git docs.
> > 
> > Many of the feature pages are probably better as part of the formal
> > QEMU documentation too.
> 
> Yes, definitely; but someone has to do the work.  At least the obsolete
> features are clearly marked as so.

Even if no one has time todo the work now, if we can spend a bit of time
to categorize current wiki pages and document a desired "plan", then we
can publicise it as a "bite sized task" and potentially motivate future
contributors to do some of the work as a useful way in to the QEMU world.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Migrating custom qemu.org infrastructure to GitLab

[Thomas Huth]

On 08/07/2020 12.53, Daniel P. Berrangé wrote:
> On Wed, Jul 08, 2020 at 10:52:38AM +0100, Stefan Hajnoczi wrote:
[...]
>> With this in mind I propose moving qemu.org infrastructure to GitLab
>> incrementally. [...]

FWIW, I think moving the QEMU infrastructure zoo to GitLab is a very
good idea!

Daniel already mentioned most of the things that I had in mind after
reading your mail (well, actually he mentioned way more things that I
had in mind), but let me add some sentences below anyway...

>> 2. wiki.qemu.org is a MediaWiki instance. Account creation is a hurdle
>> to one-time or new contributors. It is unclear whether GitLab's wiki
>> is expressive enough for a lossless conversion of the existing QEMU
>> wiki. Any volunteers interested in evaluating the wiki migration would
>> be appreciated.
> 
> Yeah, this is a potentially big piece of work. We didn't finish this
> in libvirt either. Looking at the libvirt mediawiki though, I decided
> not todo a straight export/import of all content.
> A bunch of content was horribly out of date and needed deleting as
> it was irrelevant to anyone, or outright misleading wrt current state
> of the world.

Having done some clean-up work in the QEMU wiki in the past already, I
can only agree with Daniel. I don't think that we want a 1:1 conversion
of the existing Wiki, we should rather start from scratch and only
convert the important pages. Or should we really keep stuff like
https://wiki.qemu.org/Documentation/KQemu around?

Maybe we should also rather convert the important content into parts of
the main website instead, just like Daniel mentioned it.

Whichever way we choose, I volunteer to help here if my spare time permits.

>> 5. Issue tracking. Launchpad more or less works, but the login always
>> bothers me. If we move git repo hosting then it makes sense to do
>> issue tracking on GitLab too.
> 
> The big thing that always bothers me about launchpad is how easy it
> is to get confused between issues for QEMU upstream and issues for
> legacy releases in Ubuntu distro.

+1000 !

I was already thinking of suggesting to move the bug tracker to either
gitlab or github or anywhere else during next KVM forum, since it is
IMHO a real pain.

I've seen so many bugs that users tried to open against the downstream
Ubuntu QEMU package but ended up in the upstream tracker instead. Apart
from that, the Launchpad UI is partly really horrible in my eyes (for
example you never know which action will trigger an immediate change and
which needs to be confirmed by pressing a button). Additional many
developers don't have a Launchpad account, so bugs can not be assigned
properly and you just have to pray that people see the notification
e-mails on the mailing list.

> There is a question of what todo with existing bugs in launchpad.
> 
> Essentially three choices
> 
>  1. Move all the open bugs to gitlab
>  2. Move some relevant bugs to gitlab, but close outdated ones
>  3. Leave existing launchpad bugs but don't allow new ones filed

I think we could set most (outdated) bugs simply to "incomplete" with a
message saying that the reporter should open a new bug on Gitlab if
necessary. Then after 60 days, the "incomplete" bugs will expire (i.e.
auto-close).

 Thomas

Re: Migrating custom qemu.org infrastructure to GitLab

[Philippe Mathieu-Daudé]

On 7/8/20 1:48 PM, Thomas Huth wrote:
> On 08/07/2020 12.53, Daniel P. Berrangé wrote:
>> On Wed, Jul 08, 2020 at 10:52:38AM +0100, Stefan Hajnoczi wrote:
> [...]
>>> With this in mind I propose moving qemu.org infrastructure to GitLab
>>> incrementally. [...]
> 
> FWIW, I think moving the QEMU infrastructure zoo to GitLab is a very
> good idea!
> 
> Daniel already mentioned most of the things that I had in mind after
> reading your mail (well, actually he mentioned way more things that I
> had in mind), but let me add some sentences below anyway...

Same comment ;)

I find sometime confusing the see which GitLab features are restricted
to the paid version and which are available for open source projects.

>>> 5. Issue tracking. Launchpad more or less works, but the login always
>>> bothers me. If we move git repo hosting then it makes sense to do
>>> issue tracking on GitLab too.
>>
>> The big thing that always bothers me about launchpad is how easy it
>> is to get confused between issues for QEMU upstream and issues for
>> legacy releases in Ubuntu distro.
> 
> +1000 !
> 
> I was already thinking of suggesting to move the bug tracker to either
> gitlab or github or anywhere else during next KVM forum, since it is
> IMHO a real pain.
> 
> I've seen so many bugs that users tried to open against the downstream
> Ubuntu QEMU package but ended up in the upstream tracker instead. Apart
> from that, the Launchpad UI is partly really horrible in my eyes (for
> example you never know which action will trigger an immediate change and
> which needs to be confirmed by pressing a button). Additional many
> developers don't have a Launchpad account, so bugs can not be assigned
> properly and you just have to pray that people see the notification
> e-mails on the mailing list.
> 
>> There is a question of what todo with existing bugs in launchpad.
>>
>> Essentially three choices
>>
>>  1. Move all the open bugs to gitlab
>>  2. Move some relevant bugs to gitlab, but close outdated ones
>>  3. Leave existing launchpad bugs but don't allow new ones filed
> 
> I think we could set most (outdated) bugs simply to "incomplete" with a
> message saying that the reporter should open a new bug on Gitlab if
> necessary. Then after 60 days, the "incomplete" bugs will expire (i.e.
> auto-close).

Some users hide their email on launchpad, so we would be hard to simply
re-import their bug on gitlab. Now if you ask them to import it, it is
easier. 60 days seem enough to react.

Something that always bugged me on launchpad is you can not Cc other
people on a bug if they don't have a launchpad account. I haven't
checked if GitLab allows that (Bugzilla does).

We should do some experiments first, because I saw various ways to use
the GitLab ticket tags, and none convinced me it is practical.

Should anyone add any tag? Should we restrict to a set of useful tags?

Current launchpad tags:

    35 arm
    21 linux-user
    17 qemu
    16 ppc
    15 windows
    13 testcase
    11 tcg
    9 mips
    9 usb
    9 qemu-img
    6 i386
    6 feature-request
    2 sh4
    2 patch
    1 s390x
    1 sparc

The patterns I see:

- user-mode vs softmmu
  (linux-user)

- one tag per architecture
  (arm / ppc / mips / i386 / sh4 / s390x / sparc)

- host OS
  (windows)

- accelerator
  (tcg)

- subsystem
  (usb)

- tools
  (qemu-img)

- feature requests

- testcase

I suppose tags are hints to maintainers, so keeping something similar to
the MAINTAINERS file separation could be useful.

Note the GitLab roles:
https://docs.gitlab.com/ee/development/permissions.html

    No access (0)
    Guest (10)
    Reporter (20)
    Developer (30)
    Maintainer (40)
    Owner (50)

Re: Migrating custom qemu.org infrastructure to GitLab

[Daniel P. Berrangé]

On Wed, Jul 08, 2020 at 03:19:08PM +0200, Philippe Mathieu-Daudé wrote:
> On 7/8/20 1:48 PM, Thomas Huth wrote:
> > On 08/07/2020 12.53, Daniel P. Berrangé wrote:
> >> On Wed, Jul 08, 2020 at 10:52:38AM +0100, Stefan Hajnoczi wrote:
> > [...]
> >>> With this in mind I propose moving qemu.org infrastructure to GitLab
> >>> incrementally. [...]
> > 
> > FWIW, I think moving the QEMU infrastructure zoo to GitLab is a very
> > good idea!
> > 
> > Daniel already mentioned most of the things that I had in mind after
> > reading your mail (well, actually he mentioned way more things that I
> > had in mind), but let me add some sentences below anyway...
> 
> Same comment ;)
> 
> I find sometime confusing the see which GitLab features are restricted
> to the paid version and which are available for open source projects.
> 
> >>> 5. Issue tracking. Launchpad more or less works, but the login always
> >>> bothers me. If we move git repo hosting then it makes sense to do
> >>> issue tracking on GitLab too.
> >>
> >> The big thing that always bothers me about launchpad is how easy it
> >> is to get confused between issues for QEMU upstream and issues for
> >> legacy releases in Ubuntu distro.
> > 
> > +1000 !
> > 
> > I was already thinking of suggesting to move the bug tracker to either
> > gitlab or github or anywhere else during next KVM forum, since it is
> > IMHO a real pain.
> > 
> > I've seen so many bugs that users tried to open against the downstream
> > Ubuntu QEMU package but ended up in the upstream tracker instead. Apart
> > from that, the Launchpad UI is partly really horrible in my eyes (for
> > example you never know which action will trigger an immediate change and
> > which needs to be confirmed by pressing a button). Additional many
> > developers don't have a Launchpad account, so bugs can not be assigned
> > properly and you just have to pray that people see the notification
> > e-mails on the mailing list.
> > 
> >> There is a question of what todo with existing bugs in launchpad.
> >>
> >> Essentially three choices
> >>
> >>  1. Move all the open bugs to gitlab
> >>  2. Move some relevant bugs to gitlab, but close outdated ones
> >>  3. Leave existing launchpad bugs but don't allow new ones filed
> > 
> > I think we could set most (outdated) bugs simply to "incomplete" with a
> > message saying that the reporter should open a new bug on Gitlab if
> > necessary. Then after 60 days, the "incomplete" bugs will expire (i.e.
> > auto-close).
> 
> Some users hide their email on launchpad, so we would be hard to simply
> re-import their bug on gitlab. Now if you ask them to import it, it is
> easier. 60 days seem enough to react.
> 
> Something that always bugged me on launchpad is you can not Cc other
> people on a bug if they don't have a launchpad account. I haven't
> checked if GitLab allows that (Bugzilla does).

GitLab doesn't expose anyone's email address. Any interaction with other
users is exclusively via their GitLab user name. So yes, you need an
account to be added to notifications for an issue.

> We should do some experiments first, because I saw various ways to use
> the GitLab ticket tags, and none convinced me it is practical.

Why is that ?  I find the tagging to be one of the things i really
like coming over from the bugzilla world. It is useful for doing an
initial triage of bugs in particular, to sort them into logical buckets.

I think that's particularly useful with our subsystem maintainer model,
as it will let us direct bugs towards specific maintainers.

In libvirt we had some generic labels for all projects

  https://gitlab.com/groups/libvirt/-/labels

And then further project specific labels

  https://gitlab.com/libvirt/libvirt/-/labels

> Should anyone add any tag? Should we restrict to a set of useful tags?

I believe only admins can define the tags, you can't add arbitrary
tags to a project as a user.

> I suppose tags are hints to maintainers, so keeping something similar to
> the MAINTAINERS file separation could be useful.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Migrating custom qemu.org infrastructure to GitLab

[Philippe Mathieu-Daudé]

On 7/8/20 3:32 PM, Daniel P. Berrangé wrote:
> On Wed, Jul 08, 2020 at 03:19:08PM +0200, Philippe Mathieu-Daudé wrote:
>> On 7/8/20 1:48 PM, Thomas Huth wrote:
>>> On 08/07/2020 12.53, Daniel P. Berrangé wrote:
>>>> On Wed, Jul 08, 2020 at 10:52:38AM +0100, Stefan Hajnoczi wrote:
>>> [...]
>>>>> With this in mind I propose moving qemu.org infrastructure to GitLab
>>>>> incrementally. [...]
>>>
>>> FWIW, I think moving the QEMU infrastructure zoo to GitLab is a very
>>> good idea!
>>>
>>> Daniel already mentioned most of the things that I had in mind after
>>> reading your mail (well, actually he mentioned way more things that I
>>> had in mind), but let me add some sentences below anyway...
>>
>> Same comment ;)
>>
>> I find sometime confusing the see which GitLab features are restricted
>> to the paid version and which are available for open source projects.
>>
>>>>> 5. Issue tracking. Launchpad more or less works, but the login always
>>>>> bothers me. If we move git repo hosting then it makes sense to do
>>>>> issue tracking on GitLab too.
>>>>
>>>> The big thing that always bothers me about launchpad is how easy it
>>>> is to get confused between issues for QEMU upstream and issues for
>>>> legacy releases in Ubuntu distro.
>>>
>>> +1000 !
>>>
>>> I was already thinking of suggesting to move the bug tracker to either
>>> gitlab or github or anywhere else during next KVM forum, since it is
>>> IMHO a real pain.
>>>
>>> I've seen so many bugs that users tried to open against the downstream
>>> Ubuntu QEMU package but ended up in the upstream tracker instead. Apart
>>> from that, the Launchpad UI is partly really horrible in my eyes (for
>>> example you never know which action will trigger an immediate change and
>>> which needs to be confirmed by pressing a button). Additional many
>>> developers don't have a Launchpad account, so bugs can not be assigned
>>> properly and you just have to pray that people see the notification
>>> e-mails on the mailing list.
>>>
>>>> There is a question of what todo with existing bugs in launchpad.
>>>>
>>>> Essentially three choices
>>>>
>>>>  1. Move all the open bugs to gitlab
>>>>  2. Move some relevant bugs to gitlab, but close outdated ones
>>>>  3. Leave existing launchpad bugs but don't allow new ones filed
>>>
>>> I think we could set most (outdated) bugs simply to "incomplete" with a
>>> message saying that the reporter should open a new bug on Gitlab if
>>> necessary. Then after 60 days, the "incomplete" bugs will expire (i.e.
>>> auto-close).
>>
>> Some users hide their email on launchpad, so we would be hard to simply
>> re-import their bug on gitlab. Now if you ask them to import it, it is
>> easier. 60 days seem enough to react.
>>
>> Something that always bugged me on launchpad is you can not Cc other
>> people on a bug if they don't have a launchpad account. I haven't
>> checked if GitLab allows that (Bugzilla does).
> 
> GitLab doesn't expose anyone's email address. Any interaction with other
> users is exclusively via their GitLab user name. So yes, you need an
> account to be added to notifications for an issue.
> 
>> We should do some experiments first, because I saw various ways to use
>> the GitLab ticket tags, and none convinced me it is practical.
> 
> Why is that ?  I find the tagging to be one of the things i really
> like coming over from the bugzilla world. It is useful for doing an
> initial triage of bugs in particular, to sort them into logical buckets.
> 
> I think that's particularly useful with our subsystem maintainer model,
> as it will let us direct bugs towards specific maintainers.
> 
> In libvirt we had some generic labels for all projects
> 
>   https://gitlab.com/groups/libvirt/-/labels
> 
> And then further project specific labels
> 
>   https://gitlab.com/libvirt/libvirt/-/labels

Excellent, this is exactly what we need, so I'm not worried anymore :)

> 
>> Should anyone add any tag? Should we restrict to a set of useful tags?
> 
> I believe only admins can define the tags, you can't add arbitrary
> tags to a project as a user.

We are good then (users can still suggest pertinent tags in when opening
an issue).

> 
>> I suppose tags are hints to maintainers, so keeping something similar to
>> the MAINTAINERS file separation could be useful.
> 
> Regards,
> Daniel
> 

Re: Migrating custom qemu.org infrastructure to GitLab

[Stefan Hajnoczi]

On Wed, Jul 8, 2020 at 10:52 AM Stefan Hajnoczi <stefanha@gmail.com> wrote:

Great, thanks for all the responses!

Since others are interested in qemu-web.git, the wiki, and the bug
tracker, I will investigate git repo hosting (mirrors).

I'll send an update once I have more experience with GitLab and a
proposal for how to perform the switch. If anyone objects we can
discuss their concerns.

Stefan

Re: Migrating custom qemu.org infrastructure to GitLab

[Gerd Hoffmann]

  Hi,

> > 2. wiki.qemu.org is a MediaWiki instance. Account creation is a hurdle
> > to one-time or new contributors. It is unclear whether GitLab's wiki
> > is expressive enough for a lossless conversion of the existing QEMU
> > wiki. Any volunteers interested in evaluating the wiki migration would
> > be appreciated.
> 
> Yeah, this is a potentially big piece of work. We didn't finish this
> in libvirt either. Looking at the libvirt mediawiki though, I decided
> not todo a straight export/import of all content.

FYI: gitlab wiki is basically just a git repo with markdown pages +
renderer + gui editor.  You can also update the wiki using git clone +
edit + git commit + git push.

take care,
  Gerd

Re: Migrating custom qemu.org infrastructure to GitLab

[Thomas Huth]

On 09/07/2020 12.16, Gerd Hoffmann wrote:
>   Hi,
> 
>>> 2. wiki.qemu.org is a MediaWiki instance. Account creation is a hurdle
>>> to one-time or new contributors. It is unclear whether GitLab's wiki
>>> is expressive enough for a lossless conversion of the existing QEMU
>>> wiki. Any volunteers interested in evaluating the wiki migration would
>>> be appreciated.
>>
>> Yeah, this is a potentially big piece of work. We didn't finish this
>> in libvirt either. Looking at the libvirt mediawiki though, I decided
>> not todo a straight export/import of all content.
> 
> FYI: gitlab wiki is basically just a git repo with markdown pages +
> renderer + gui editor.  You can also update the wiki using git clone +
> edit + git commit + git push.

FWIW, seems like we could use the "pandoc" tool to convert Mediawiki
(our old Wiki) to Markdown (Gitlab wiki). I've done a quick test and
converted https://wiki.qemu.org/Contribute/MailingLists into
https://gitlab.com/huth/qemu/-/wikis/Contribute/MailingLists with some
few clicks.

But the longer I look at most Wiki pages, the more I think that we
should convert the important pages rather into a part of qemu-web
instead. I'll have a closer look and will suggest some patches when time
permits...

 Thomas

Re: Migrating custom qemu.org infrastructure to GitLab

[Paolo Bonzini]

On 09/07/20 12:22, Thomas Huth wrote:
> FWIW, seems like we could use the "pandoc" tool to convert Mediawiki
> (our old Wiki) to Markdown (Gitlab wiki). I've done a quick test and
> converted https://wiki.qemu.org/Contribute/MailingLists into
> https://gitlab.com/huth/qemu/-/wikis/Contribute/MailingLists with some
> few clicks.
> 
> But the longer I look at most Wiki pages, the more I think that we
> should convert the important pages rather into a part of qemu-web
> instead. I'll have a closer look and will suggest some patches when time
> permits...

The wiki was cleaned up more or less at the same time as the
qemu-web.git repo was created (actually as a prerequisite), it's
actually not in a bad shape.  The idea was that the wiki kept:

- stuff that really belonged in documentation (such as completed
features and developer information)

- stuff that needs to be edited quickly (such as feature pages or or
internship ideas)

- developer-targeted information that doesn't belong in documentation
(such as CI status), even if it's linked from qemu.org (e.g.
https://www.qemu.org/contribute/

while qemu-web got the more user-targeted information.  This is because
updating qemu-web is a bit slower, requiring review and all that.

We can certainly move some wiki pages to qemu-web, like we did for
"report a bug" in the past and like Alex did recently for the
Conservancy page.  But I think there aren't that many left, most of them
are in the first category above and should be moved to docs/devel (for
example https://wiki.qemu.org/Contribute/SubmitAPatch).

Once we have docs CI on GitLab we can easily link to them from
qemu-web.git, so setting up docs CI is probably a good first step
towards relying more on GitLab and also cleaning up the wiki.

Paolo

Re: Migrating custom qemu.org infrastructure to GitLab

[Thomas Huth]

On 09/07/2020 12.33, Paolo Bonzini wrote:
> On 09/07/20 12:22, Thomas Huth wrote:
>> FWIW, seems like we could use the "pandoc" tool to convert Mediawiki
>> (our old Wiki) to Markdown (Gitlab wiki). I've done a quick test and
>> converted https://wiki.qemu.org/Contribute/MailingLists into
>> https://gitlab.com/huth/qemu/-/wikis/Contribute/MailingLists with some
>> few clicks.
>>
>> But the longer I look at most Wiki pages, the more I think that we
>> should convert the important pages rather into a part of qemu-web
>> instead. I'll have a closer look and will suggest some patches when time
>> permits...
> 
> The wiki was cleaned up more or less at the same time as the
> qemu-web.git repo was created (actually as a prerequisite), it's
> actually not in a bad shape.

Agreed, there were definitely worse times in the past.

> We can certainly move some wiki pages to qemu-web, like we did for
> "report a bug" in the past and like Alex did recently for the
> Conservancy page.  But I think there aren't that many left, most of them
> are in the first category above and should be moved to docs/devel (for
> example https://wiki.qemu.org/Contribute/SubmitAPatch).

I was looking at pages like
https://wiki.qemu.org/Contribute/MailingLists and
https://wiki.qemu.org/License ... but yes, the first one should be
likely moved to docs/devel, too, and the second one is maybe redundant
anyway, since we have this information in the LICENSE file already (so
we could also link to
https://git.qemu.org/?p=qemu.git;a=blob_plain;f=LICENSE instead).

 Thomas

Delete some Wiki pages (was: Migrating custom qemu.org infrastructure to GitLab)

[Thomas Huth]

On 09/07/2020 12.33, Paolo Bonzini wrote:
> On 09/07/20 12:22, Thomas Huth wrote:
>> FWIW, seems like we could use the "pandoc" tool to convert Mediawiki
>> (our old Wiki) to Markdown (Gitlab wiki). I've done a quick test and
>> converted https://wiki.qemu.org/Contribute/MailingLists into
>> https://gitlab.com/huth/qemu/-/wikis/Contribute/MailingLists with some
>> few clicks.
>>
>> But the longer I look at most Wiki pages, the more I think that we
>> should convert the important pages rather into a part of qemu-web
>> instead. I'll have a closer look and will suggest some patches when time
>> permits...
> 
> The wiki was cleaned up more or less at the same time as the
> qemu-web.git repo was created (actually as a prerequisite), it's
> actually not in a bad shape.

There are still quite a bit of pages that either contain hardly any
usable information or are completely outdated. Paolo, Stefan, may I ask
you to check whether the following pages could be deleted? I don't have
access rights to delete a page, so if you agree that they are mostly
useless, please go ahead and delete them:

- https://wiki.qemu.org/Features/40p :
  only few infos here, and 40p is merged already

- https://wiki.qemu.org/Features/PRePCleanup :
  prep machine was removed some time ago, 40p is the replacement

- https://wiki.qemu.org/Features/BeBox
  old project from 2010, never got merged AFAIK

- https://wiki.qemu.org/Features/Documentation/interop
  that has been superseded by:
  https://www.qemu.org/docs/master/interop/index.html

- https://wiki.qemu.org/Features/Documentation/specs
  that has been superseded by:
  https://www.qemu.org/docs/master/specs/index.html

- https://wiki.qemu.org/Features/DriveRefactoring
  Out of date, we've got -blockdev now (and not -blkdev)

- https://wiki.qemu.org/Features/LegacyRemoval
  Has been replaced by the "deprecated features" section in our docs

- https://wiki.qemu.org/Features/Machines/Edison
  old project from 2015, never took off AFAIK

- https://wiki.qemu.org/Features/LibvirtWiresharkDissector
  seems to be a libvirt proposal - IMHO should not be in the QEMU wiki

- https://wiki.qemu.org/Features/Tegra2
  old project, AFAIK never got merged

- https://wiki.qemu.org/Features/Version3.0
  Old suggestions for QEMU version 3.0 ... we're close to 5.1 already

- https://wiki.qemu.org/KeySigningParty2013
  https://wiki.qemu.org/KeySigningParty2014
  https://wiki.qemu.org/KeySigningParty2015
  Only some few old information here, useless nowadays?

- https://wiki.qemu.org/Features/network_reentrant
  Old ideas from 2013 ... I think vhost-net superseded this?

- https://wiki.qemu.org/Planning/Relicensing
  I think this has been completed. The page looks very outdated now.

- https://wiki.qemu.org/SecurityProcess
  Should be replaced with a redirect

 Thomas

Re: Delete some Wiki pages (was: Migrating custom qemu.org infrastructure to GitLab)

[Daniel P. Berrangé]

On Thu, Jul 09, 2020 at 03:10:44PM +0200, Thomas Huth wrote:
> On 09/07/2020 12.33, Paolo Bonzini wrote:
> > On 09/07/20 12:22, Thomas Huth wrote:
> >> FWIW, seems like we could use the "pandoc" tool to convert Mediawiki
> >> (our old Wiki) to Markdown (Gitlab wiki). I've done a quick test and
> >> converted https://wiki.qemu.org/Contribute/MailingLists into
> >> https://gitlab.com/huth/qemu/-/wikis/Contribute/MailingLists with some
> >> few clicks.
> >>
> >> But the longer I look at most Wiki pages, the more I think that we
> >> should convert the important pages rather into a part of qemu-web
> >> instead. I'll have a closer look and will suggest some patches when time
> >> permits...
> > 
> > The wiki was cleaned up more or less at the same time as the
> > qemu-web.git repo was created (actually as a prerequisite), it's
> > actually not in a bad shape.
> 
> There are still quite a bit of pages that either contain hardly any
> usable information or are completely outdated. Paolo, Stefan, may I ask
> you to check whether the following pages could be deleted? I don't have
> access rights to delete a page, so if you agree that they are mostly
> useless, please go ahead and delete them:

> - https://wiki.qemu.org/Features/LibvirtWiresharkDissector
>   seems to be a libvirt proposal - IMHO should not be in the QEMU wiki

This was a GSoC project and at that time libvirt GSoC projects were
under the QEMU umbrella. Libvirt now runs GSoC itself.

Either way, there's no particularly important content here, but it is
the second hit in google for "libvirt wireshark" ! I'm fine with it
being removed in general unless we want to keep a record of old GSoc
projects ?

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Delete some Wiki pages (was: Migrating custom qemu.org infrastructure to GitLab)

[Paolo Bonzini]

On 09/07/20 15:10, Thomas Huth wrote:
> - https://wiki.qemu.org/Features/40p :
> - https://wiki.qemu.org/Features/PRePCleanup :
> - https://wiki.qemu.org/Features/BeBox
> - https://wiki.qemu.org/Features/DriveRefactoring
> - https://wiki.qemu.org/Features/LegacyRemoval
> - https://wiki.qemu.org/Features/Machines/Edison
> - https://wiki.qemu.org/Features/Tegra2

Can you make them either "Completed feature pages" or "Obsolete feature
pages"?  We can certainly drop the obsolete features if we migrate to
the GitLab wiki.

> - https://wiki.qemu.org/Features/Documentation/interop
>   that has been superseded by:
>   https://www.qemu.org/docs/master/interop/index.html
> 
> - https://wiki.qemu.org/Features/Documentation/specs
>   that has been superseded by:
>   https://www.qemu.org/docs/master/specs/index.html

Need to check if there's anything else left to do for those two manuals.
 Probably the rST conversion of QMP documentation.  I would leave them
around, they are linked from the main Features/Documentation page.

> - https://wiki.qemu.org/Features/LibvirtWiresharkDissector
>   seems to be a libvirt proposal - IMHO should not be in the QEMU wiki

Probably an internship idea.

> - https://wiki.qemu.org/Features/Version3.0
>   Old suggestions for QEMU version 3.0 ... we're close to 5.1 already

This is more like ideas for deprecation.

> - https://wiki.qemu.org/KeySigningParty2013
>   https://wiki.qemu.org/KeySigningParty2014
>   https://wiki.qemu.org/KeySigningParty2015
>   Only some few old information here, useless nowadays?
> 
> - https://wiki.qemu.org/Features/network_reentrant
>   Old ideas from 2013 ... I think vhost-net superseded this?

It's in [[Category:Obsolete feature pages]] for a reason. :)

> - https://wiki.qemu.org/Planning/Relicensing
>   I think this has been completed. The page looks very outdated now.

No, it's not completed and it's probably never going to be.

> - https://wiki.qemu.org/SecurityProcess
>   Should be replaced with a redirect

Done.

Paolo

Re: Migrating custom qemu.org infrastructure to GitLab

[Philippe Mathieu-Daudé]

On 7/8/20 11:52 AM, Stefan Hajnoczi wrote:
> 
> There is a snowball effect where the experience is improved the more
> GitLab features we use, so I hope that most of these migrations will
> be possible.

I've been looking at other features we might be interested in.

1/ Calendar

I was looking for a calendar option to register the release dates
and the Tuesday QEMU/KVM external call.
They recommend to pair with GSuite instead:
https://about.gitlab.com/handbook/tools-and-tips/#google-calendar
Or calendly: https://calendly.com/

2/ Due dates

We can set 'due date' to issues/merge requests, useful to not
miss soft-freeze or CVE embargo expiration:
https://docs.gitlab.com/ee/user/project/issues/due_dates.html

3/ Milestones as releases

https://docs.gitlab.com/ee/user/project/milestones/#milestones-as-releases

4/ d3js graphs

https://docs.gitlab.com/ee/development/new_fe_guide/development/components.html#graphs
So we can now have more interesting graphs in the wiki/doc such:
https://observablehq.com/@philmd/qemu-aarch64-softmmu-qom-tree

5/ CI testing metrics
https://docs.gitlab.com/ee/ci/metrics_reports.html#how-it-works

6/ Sentry error tracking

Maybe to gather the CI errors?
https://docs.gitlab.com/ee/user/project/operations/error_tracking.html#sentry-error-tracking

7/ License Compliance

https://docs.gitlab.com/ee/user/compliance/license_compliance/

8/ Connecting with other bug tracking services:

In case we want to link BZ:
https://docs.gitlab.com/ee/user/project/integrations/bugzilla.html

Or eventually launchpad:
https://docs.gitlab.com/ee/user/project/integrations/custom_issue_tracker.html


From these points I'm interested in investigating

- CI testing metrics
- d3js graphs
- License Compliance

Regards,

Phil.

Re: Migrating custom qemu.org infrastructure to GitLab

[Philippe Mathieu-Daudé]

On 7/8/20 11:52 AM, Stefan Hajnoczi wrote:
>
> 3. Git repo hosting is a core feature of GitLab and we already have a
> qemu.git mirror. Hosting the repos on GitLab reduces the need for
> qemu.org ssh access.

I also noted cloning from gitlab.com is way faster than git.qemu.org,
I suppose because of huge CDN behind.

Re: Migrating custom qemu.org infrastructure to GitLab

[Philippe Mathieu-Daudé]

On 7/10/20 4:04 PM, Philippe Mathieu-Daudé wrote:
> On 7/8/20 11:52 AM, Stefan Hajnoczi wrote:
>>
>> There is a snowball effect where the experience is improved the more
>> GitLab features we use, so I hope that most of these migrations will
>> be possible.
> 
> I've been looking at other features we might be interested in.
[...]

> 7/ License Compliance
> 
> https://docs.gitlab.com/ee/user/compliance/license_compliance/

WRT IRC feedback QEMU licensing is a lost cause, I can not tell,
but I think if GitLab has scripts to take care of it, it is an
improvement over what we have now:

  a) Users will have a clearer idea of the permitted licenses

  b) New contributions will be checked for correct license by
     a machine, so reviewer can focus on other topics.

> 8/ Connecting with other bug tracking services:
> 
> In case we want to link BZ:
> https://docs.gitlab.com/ee/user/project/integrations/bugzilla.html
> 
> Or eventually launchpad:
> https://docs.gitlab.com/ee/user/project/integrations/custom_issue_tracker.html
> 
> 
> From these points I'm interested in investigating
> 
> - CI testing metrics
> - d3js graphs
> - License Compliance
> 
> Regards,
> 
> Phil.
> 

Re: Migrating custom qemu.org infrastructure to GitLab

[Peter Maydell]

On Mon, 13 Jul 2020 at 09:39, Philippe Mathieu-Daudé <philmd@redhat.com> wrote:
>
> On 7/10/20 4:04 PM, Philippe Mathieu-Daudé wrote:
>
> > 7/ License Compliance
> >
> > https://docs.gitlab.com/ee/user/compliance/license_compliance/
>
> WRT IRC feedback QEMU licensing is a lost cause, I can not tell,
> but I think if GitLab has scripts to take care of it, it is an
> improvement over what we have now:

The docs say it only supports C programs that use the Conan
package manager. We don't, we just use configure to find and
link against our external dependencies.

>   b) New contributions will be checked for correct license by
>      a machine, so reviewer can focus on other topics.

The documentation doesn't say anything about it handling license
checking on contributions to the project itself. It looks like
it's mainly intended for "I have a javascript project that
pulls in 5000 tiny dependencies from npm, I want to know that
none of them is accidentally using a license that's not compatible
with the project's license". That's not a problem QEMU has.

thanks
-- PMM

Migrate Wiki to Gitlab? (was: Migrating custom qemu.org infrastructure to GitLab)

[Thomas Huth]

On 09/07/2020 12.22, Thomas Huth wrote:
> On 09/07/2020 12.16, Gerd Hoffmann wrote:
>>   Hi,
>>
>>>> 2. wiki.qemu.org is a MediaWiki instance. Account creation is a hurdle
>>>> to one-time or new contributors. It is unclear whether GitLab's wiki
>>>> is expressive enough for a lossless conversion of the existing QEMU
>>>> wiki. Any volunteers interested in evaluating the wiki migration would
>>>> be appreciated.
>>>
>>> Yeah, this is a potentially big piece of work. We didn't finish this
>>> in libvirt either. Looking at the libvirt mediawiki though, I decided
>>> not todo a straight export/import of all content.
>>
>> FYI: gitlab wiki is basically just a git repo with markdown pages +
>> renderer + gui editor.  You can also update the wiki using git clone +
>> edit + git commit + git push.
> 
> FWIW, seems like we could use the "pandoc" tool to convert Mediawiki
> (our old Wiki) to Markdown (Gitlab wiki). I've done a quick test and
> converted https://wiki.qemu.org/Contribute/MailingLists into
> https://gitlab.com/huth/qemu/-/wikis/Contribute/MailingLists with some
> few clicks.

Revisiting this topic after a couple of weeks, I think there is one more
thing to consider: If I've got that right, your account has to be a member
of the corresponding project to be able to edit a page in the Wiki in
gitlab. So unless we want to add lots of persons to the qemu-project as
members (which we likely do not really want, do we?), it's maybe better to
keep the separate MediaWiki instance with the separate user accounts, I guess?

 Thomas

Re: Migrate Wiki to Gitlab? (was: Migrating custom qemu.org infrastructure to GitLab)

[Daniel P. Berrangé]

On Fri, Oct 23, 2020 at 07:59:00AM +0200, Thomas Huth wrote:
> On 09/07/2020 12.22, Thomas Huth wrote:
> > On 09/07/2020 12.16, Gerd Hoffmann wrote:
> >>   Hi,
> >>
> >>>> 2. wiki.qemu.org is a MediaWiki instance. Account creation is a hurdle
> >>>> to one-time or new contributors. It is unclear whether GitLab's wiki
> >>>> is expressive enough for a lossless conversion of the existing QEMU
> >>>> wiki. Any volunteers interested in evaluating the wiki migration would
> >>>> be appreciated.
> >>>
> >>> Yeah, this is a potentially big piece of work. We didn't finish this
> >>> in libvirt either. Looking at the libvirt mediawiki though, I decided
> >>> not todo a straight export/import of all content.
> >>
> >> FYI: gitlab wiki is basically just a git repo with markdown pages +
> >> renderer + gui editor.  You can also update the wiki using git clone +
> >> edit + git commit + git push.
> > 
> > FWIW, seems like we could use the "pandoc" tool to convert Mediawiki
> > (our old Wiki) to Markdown (Gitlab wiki). I've done a quick test and
> > converted https://wiki.qemu.org/Contribute/MailingLists into
> > https://gitlab.com/huth/qemu/-/wikis/Contribute/MailingLists with some
> > few clicks.
> 
> Revisiting this topic after a couple of weeks, I think there is one more
> thing to consider: If I've got that right, your account has to be a member
> of the corresponding project to be able to edit a page in the Wiki in
> gitlab. So unless we want to add lots of persons to the qemu-project as
> members (which we likely do not really want, do we?), it's maybe better to
> keep the separate MediaWiki instance with the separate user accounts, I guess?

Yep, manual says it requires "Developer" permissions which is certainly
not something we want to give out widely, as it grants way too much
privilege.

I'd still suggest we look at folding the wiki into either the website or
the main repo documentation as appropriate for the docs in question.
Having 3 separate sources of documentation just feels wrong to me, and
we largely lost the "drive by" contributor for the wiki since we had
to lock new account creation - only a handful of people actually ask us
to create accounts for them.

The qemu-web.git could be made fairly low overhead for contributors by
accepting contributions via merge requests, which would let people just
interatively edit the web pages and submit the changes all from the web
UI. Not quite as seemless as a wiki, but not far off.

We could even demarcate a subset of the website with a /wiki URL and just
stuff all current wiki content in there as RST. Then have a much looser
review policy for merging changes in that subtree. Basically allow anything
as long as it is well formatted and not grossly incorrect. Could even 100%
automate approval under that subdir if you want the true wiki experiance
of having no content review.


The main downside with the main qemu.git documentation is that it takes
a long time to get docs through the contribution workflow and you have
to deal with the qemu-devel firehose. Use of RST has improved the authoring
part, but review/merge is still a bit tedious. There is no single docs
maintainer, and parts of docs are completely lacking maintainer entries.
So it can be pot-luck whether your docs contribution gets reviewed and
merged in reasonable time.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|