[LSF/MM/BPF TOPIC] vmtest: Reusable virtual machine testing infrastructure

[LSF/MM/BPF TOPIC] vmtest: Reusable virtual machine testing infrastructure

[Daniel Xu]

=== Introduction ===

Testing is paradoxically one of BPF's great strengths as well as one of it's
current weaknesses. Fortunately, this weakness is not too far from being
corrected.

BPF_PROG_RUN is somewhat of a double edged sword. On the one hand, you can run
reproducibly run progs in near-production context. On the other hand, since BPF
is so deeply intertwined with the running kernel, you must make the kernel you
run tests on as close to your production kernel as possible to get full testing
benefits.

This is going to be more of an issue going forward through the growth of kfuncs
because kfuncs do not possess a stable ABI [5]. Proper testing should be
encouraged at a community-wide level in order to avoid accidental surprises and
potential loss of faith in BPF "stability".

Most successful kernel-dependent projects deploy some form of
virtual-machine-based testing [1][2][3][4] to solve the above issues. However,
there are two problems with this:

1. VM-based testing is not quite common knowledge yet and remains somewhat of
   a dark art to successfully implement.

2. Multiple implementations of what is essentially the same thing is somewhat
   of a drain on resources.

(These are not necessarily bad things -- it is useful and necessary to explore a
problem space before settling on best practices)

vmtest [0] aims to solve both problems.

=== Goals ===

I'd like to do a short presentation on the design and ideas behind vmtest. I'd
also like to show a quick demo. It shouldn't take very long. I'll probably
also share what I'd like to implement next. I don't know what that's going
to be at time of writing b/c I'm probably going to get to it before LSFMMBPF.

For the rest of the time I'd like to discuss what the community would like to
see in vmtest. And to hear what it'd take to see adoption from other projects.
Obviously no one can be required to adopt vmtest but I think it'll save everyone
a good deal of effort if done correctly.

Thanks,
Daniel


[0]: https://github.com/danobi/vmtest
[1]: https://github.com/cilium/ebpf/blob/master/run-tests.sh
[2]: https://github.com/osandov/drgn/tree/main/vmtest
[3]: https://github.com/libbpf/ci/blob/master/run-qemu/run.sh
[4]: https://github.com/torvalds/linux/blob/master/tools/testing/selftests/bpf/vmtest.sh
[5]: https://github.com/torvalds/linux/commit/16c294a6aad86

Re: [Lsf-pc] [LSF/MM/BPF TOPIC] vmtest: Reusable virtual machine testing infrastructure

[Josef Bacik]

On Mon, Feb 27, 2023 at 6:02 PM Daniel Xu <dxu@dxuuu.xyz> wrote:
>
> === Introduction ===
>
> Testing is paradoxically one of BPF's great strengths as well as one of it's
> current weaknesses. Fortunately, this weakness is not too far from being
> corrected.
>
> BPF_PROG_RUN is somewhat of a double edged sword. On the one hand, you can run
> reproducibly run progs in near-production context. On the other hand, since BPF
> is so deeply intertwined with the running kernel, you must make the kernel you
> run tests on as close to your production kernel as possible to get full testing
> benefits.
>
> This is going to be more of an issue going forward through the growth of kfuncs
> because kfuncs do not possess a stable ABI [5]. Proper testing should be
> encouraged at a community-wide level in order to avoid accidental surprises and
> potential loss of faith in BPF "stability".
>
> Most successful kernel-dependent projects deploy some form of
> virtual-machine-based testing [1][2][3][4] to solve the above issues. However,
> there are two problems with this:
>
> 1. VM-based testing is not quite common knowledge yet and remains somewhat of
>    a dark art to successfully implement.
>
> 2. Multiple implementations of what is essentially the same thing is somewhat
>    of a drain on resources.
>
> (These are not necessarily bad things -- it is useful and necessary to explore a
> problem space before settling on best practices)
>
> vmtest [0] aims to solve both problems.
>
> === Goals ===
>
> I'd like to do a short presentation on the design and ideas behind vmtest. I'd
> also like to show a quick demo. It shouldn't take very long. I'll probably
> also share what I'd like to implement next. I don't know what that's going
> to be at time of writing b/c I'm probably going to get to it before LSFMMBPF.
>
> For the rest of the time I'd like to discuss what the community would like to
> see in vmtest. And to hear what it'd take to see adoption from other projects.
> Obviously no one can be required to adopt vmtest but I think it'll save everyone
> a good deal of effort if done correctly.
>

FYI a lot of us have been working on kdevops
(https://github.com/linux-kdevops/kdevops), which has similar goals,
tho yours feels more in line with virt-me which I've used a bunch as
well.

I would very much love it if we could all get behind one project.  The
benefit of kdevops is it's very extensible, and being able to select
some config options and have an entire testing suite up and running is
very handy for new developers.

That being said I can definitely get behind have two sort of options,
the bigger swiss army knife that is kdevops, and something smaller
that's easier to do one-off runs.

kdevops using vagrant makes a lot of the pain of setting up the full
VM environments that existed before go away.  I can easily tear down
my 8 CI vm's and rebuild them all and having them testing again in 4
commands.  Nowadays I probably wouldn't use virt-me/anything lighter
because this is actually pretty easy to get up and running.

Again no trying to be discouraging, but you're absolutely right that
there's been a lot of fragmentation here, which is why I've spent
probably a lot more time than I should have making kdevops work for
me, as well have a lot of other kernel developers, and it's getting
pretty solid.  Thanks,

Josef

Re: [Lsf-pc] [LSF/MM/BPF TOPIC] vmtest: Reusable virtual machine testing infrastructure

[Daniel Xu]

Hi Josef,

Sorry about the late reply; I messed up my domain last week and emails
were getting black holed.

On Tue, Feb 28, 2023 at 10:38:18AM -0500, Josef Bacik wrote:
> On Mon, Feb 27, 2023 at 6:02 PM Daniel Xu <dxu@dxuuu.xyz> wrote:
> >
> > === Introduction ===
> >
> > Testing is paradoxically one of BPF's great strengths as well as one of it's
> > current weaknesses. Fortunately, this weakness is not too far from being
> > corrected.
> >
> > BPF_PROG_RUN is somewhat of a double edged sword. On the one hand, you can run
> > reproducibly run progs in near-production context. On the other hand, since BPF
> > is so deeply intertwined with the running kernel, you must make the kernel you
> > run tests on as close to your production kernel as possible to get full testing
> > benefits.
> >
> > This is going to be more of an issue going forward through the growth of kfuncs
> > because kfuncs do not possess a stable ABI [5]. Proper testing should be
> > encouraged at a community-wide level in order to avoid accidental surprises and
> > potential loss of faith in BPF "stability".
> >
> > Most successful kernel-dependent projects deploy some form of
> > virtual-machine-based testing [1][2][3][4] to solve the above issues. However,
> > there are two problems with this:
> >
> > 1. VM-based testing is not quite common knowledge yet and remains somewhat of
> >    a dark art to successfully implement.
> >
> > 2. Multiple implementations of what is essentially the same thing is somewhat
> >    of a drain on resources.
> >
> > (These are not necessarily bad things -- it is useful and necessary to explore a
> > problem space before settling on best practices)
> >
> > vmtest [0] aims to solve both problems.
> >
> > === Goals ===
> >
> > I'd like to do a short presentation on the design and ideas behind vmtest. I'd
> > also like to show a quick demo. It shouldn't take very long. I'll probably
> > also share what I'd like to implement next. I don't know what that's going
> > to be at time of writing b/c I'm probably going to get to it before LSFMMBPF.
> >
> > For the rest of the time I'd like to discuss what the community would like to
> > see in vmtest. And to hear what it'd take to see adoption from other projects.
> > Obviously no one can be required to adopt vmtest but I think it'll save everyone
> > a good deal of effort if done correctly.
> >
> 
> FYI a lot of us have been working on kdevops
> (https://github.com/linux-kdevops/kdevops), which has similar goals,
> tho yours feels more in line with virt-me which I've used a bunch as
> well.
> 
> I would very much love it if we could all get behind one project.  The
> benefit of kdevops is it's very extensible, and being able to select
> some config options and have an entire testing suite up and running is
> very handy for new developers.

Thanks for the tip; I had not heard of kdevops. I'll spend some time
this week playing with it.

> That being said I can definitely get behind have two sort of options,
> the bigger swiss army knife that is kdevops, and something smaller
> that's easier to do one-off runs.
> 
> kdevops using vagrant makes a lot of the pain of setting up the full
> VM environments that existed before go away.  I can easily tear down
> my 8 CI vm's and rebuild them all and having them testing again in 4
> commands.  Nowadays I probably wouldn't use virt-me/anything lighter
> because this is actually pretty easy to get up and running.
> 
> Again no trying to be discouraging, but you're absolutely right that
> there's been a lot of fragmentation here, which is why I've spent
> probably a lot more time than I should have making kdevops work for
> me, as well have a lot of other kernel developers, and it's getting
> pretty solid.  Thanks,

Yep, ideally I'd like to have pulled something off the shelf and moved
on. But nothing looked like it would solve my problem. So if kdevops is
close enough then I wouldn't mind contributing to that instead.


Thanks,
Daniel