* [LSF/MM/BPF TOPIC] vmtest: Reusable virtual machine testing infrastructure @ 2023-02-27 23:01 Daniel Xu 2023-02-28 15:38 ` [Lsf-pc] " Josef Bacik 0 siblings, 1 reply; 3+ messages in thread From: Daniel Xu @ 2023-02-27 23:01 UTC (permalink / raw) To: lsf-pc; +Cc: bpf === Introduction === Testing is paradoxically one of BPF's great strengths as well as one of it's current weaknesses. Fortunately, this weakness is not too far from being corrected. BPF_PROG_RUN is somewhat of a double edged sword. On the one hand, you can run reproducibly run progs in near-production context. On the other hand, since BPF is so deeply intertwined with the running kernel, you must make the kernel you run tests on as close to your production kernel as possible to get full testing benefits. This is going to be more of an issue going forward through the growth of kfuncs because kfuncs do not possess a stable ABI [5]. Proper testing should be encouraged at a community-wide level in order to avoid accidental surprises and potential loss of faith in BPF "stability". Most successful kernel-dependent projects deploy some form of virtual-machine-based testing [1][2][3][4] to solve the above issues. However, there are two problems with this: 1. VM-based testing is not quite common knowledge yet and remains somewhat of a dark art to successfully implement. 2. Multiple implementations of what is essentially the same thing is somewhat of a drain on resources. (These are not necessarily bad things -- it is useful and necessary to explore a problem space before settling on best practices) vmtest [0] aims to solve both problems. === Goals === I'd like to do a short presentation on the design and ideas behind vmtest. I'd also like to show a quick demo. It shouldn't take very long. I'll probably also share what I'd like to implement next. I don't know what that's going to be at time of writing b/c I'm probably going to get to it before LSFMMBPF. For the rest of the time I'd like to discuss what the community would like to see in vmtest. And to hear what it'd take to see adoption from other projects. Obviously no one can be required to adopt vmtest but I think it'll save everyone a good deal of effort if done correctly. Thanks, Daniel [0]: https://github.com/danobi/vmtest [1]: https://github.com/cilium/ebpf/blob/master/run-tests.sh [2]: https://github.com/osandov/drgn/tree/main/vmtest [3]: https://github.com/libbpf/ci/blob/master/run-qemu/run.sh [4]: https://github.com/torvalds/linux/blob/master/tools/testing/selftests/bpf/vmtest.sh [5]: https://github.com/torvalds/linux/commit/16c294a6aad86 ^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [Lsf-pc] [LSF/MM/BPF TOPIC] vmtest: Reusable virtual machine testing infrastructure 2023-02-27 23:01 [LSF/MM/BPF TOPIC] vmtest: Reusable virtual machine testing infrastructure Daniel Xu @ 2023-02-28 15:38 ` Josef Bacik 2023-03-06 19:07 ` Daniel Xu 0 siblings, 1 reply; 3+ messages in thread From: Josef Bacik @ 2023-02-28 15:38 UTC (permalink / raw) To: Daniel Xu; +Cc: lsf-pc, bpf On Mon, Feb 27, 2023 at 6:02 PM Daniel Xu <dxu@dxuuu.xyz> wrote: > > === Introduction === > > Testing is paradoxically one of BPF's great strengths as well as one of it's > current weaknesses. Fortunately, this weakness is not too far from being > corrected. > > BPF_PROG_RUN is somewhat of a double edged sword. On the one hand, you can run > reproducibly run progs in near-production context. On the other hand, since BPF > is so deeply intertwined with the running kernel, you must make the kernel you > run tests on as close to your production kernel as possible to get full testing > benefits. > > This is going to be more of an issue going forward through the growth of kfuncs > because kfuncs do not possess a stable ABI [5]. Proper testing should be > encouraged at a community-wide level in order to avoid accidental surprises and > potential loss of faith in BPF "stability". > > Most successful kernel-dependent projects deploy some form of > virtual-machine-based testing [1][2][3][4] to solve the above issues. However, > there are two problems with this: > > 1. VM-based testing is not quite common knowledge yet and remains somewhat of > a dark art to successfully implement. > > 2. Multiple implementations of what is essentially the same thing is somewhat > of a drain on resources. > > (These are not necessarily bad things -- it is useful and necessary to explore a > problem space before settling on best practices) > > vmtest [0] aims to solve both problems. > > === Goals === > > I'd like to do a short presentation on the design and ideas behind vmtest. I'd > also like to show a quick demo. It shouldn't take very long. I'll probably > also share what I'd like to implement next. I don't know what that's going > to be at time of writing b/c I'm probably going to get to it before LSFMMBPF. > > For the rest of the time I'd like to discuss what the community would like to > see in vmtest. And to hear what it'd take to see adoption from other projects. > Obviously no one can be required to adopt vmtest but I think it'll save everyone > a good deal of effort if done correctly. > FYI a lot of us have been working on kdevops (https://github.com/linux-kdevops/kdevops), which has similar goals, tho yours feels more in line with virt-me which I've used a bunch as well. I would very much love it if we could all get behind one project. The benefit of kdevops is it's very extensible, and being able to select some config options and have an entire testing suite up and running is very handy for new developers. That being said I can definitely get behind have two sort of options, the bigger swiss army knife that is kdevops, and something smaller that's easier to do one-off runs. kdevops using vagrant makes a lot of the pain of setting up the full VM environments that existed before go away. I can easily tear down my 8 CI vm's and rebuild them all and having them testing again in 4 commands. Nowadays I probably wouldn't use virt-me/anything lighter because this is actually pretty easy to get up and running. Again no trying to be discouraging, but you're absolutely right that there's been a lot of fragmentation here, which is why I've spent probably a lot more time than I should have making kdevops work for me, as well have a lot of other kernel developers, and it's getting pretty solid. Thanks, Josef ^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [Lsf-pc] [LSF/MM/BPF TOPIC] vmtest: Reusable virtual machine testing infrastructure 2023-02-28 15:38 ` [Lsf-pc] " Josef Bacik @ 2023-03-06 19:07 ` Daniel Xu 0 siblings, 0 replies; 3+ messages in thread From: Daniel Xu @ 2023-03-06 19:07 UTC (permalink / raw) To: Josef Bacik; +Cc: lsf-pc, bpf Hi Josef, Sorry about the late reply; I messed up my domain last week and emails were getting black holed. On Tue, Feb 28, 2023 at 10:38:18AM -0500, Josef Bacik wrote: > On Mon, Feb 27, 2023 at 6:02 PM Daniel Xu <dxu@dxuuu.xyz> wrote: > > > > === Introduction === > > > > Testing is paradoxically one of BPF's great strengths as well as one of it's > > current weaknesses. Fortunately, this weakness is not too far from being > > corrected. > > > > BPF_PROG_RUN is somewhat of a double edged sword. On the one hand, you can run > > reproducibly run progs in near-production context. On the other hand, since BPF > > is so deeply intertwined with the running kernel, you must make the kernel you > > run tests on as close to your production kernel as possible to get full testing > > benefits. > > > > This is going to be more of an issue going forward through the growth of kfuncs > > because kfuncs do not possess a stable ABI [5]. Proper testing should be > > encouraged at a community-wide level in order to avoid accidental surprises and > > potential loss of faith in BPF "stability". > > > > Most successful kernel-dependent projects deploy some form of > > virtual-machine-based testing [1][2][3][4] to solve the above issues. However, > > there are two problems with this: > > > > 1. VM-based testing is not quite common knowledge yet and remains somewhat of > > a dark art to successfully implement. > > > > 2. Multiple implementations of what is essentially the same thing is somewhat > > of a drain on resources. > > > > (These are not necessarily bad things -- it is useful and necessary to explore a > > problem space before settling on best practices) > > > > vmtest [0] aims to solve both problems. > > > > === Goals === > > > > I'd like to do a short presentation on the design and ideas behind vmtest. I'd > > also like to show a quick demo. It shouldn't take very long. I'll probably > > also share what I'd like to implement next. I don't know what that's going > > to be at time of writing b/c I'm probably going to get to it before LSFMMBPF. > > > > For the rest of the time I'd like to discuss what the community would like to > > see in vmtest. And to hear what it'd take to see adoption from other projects. > > Obviously no one can be required to adopt vmtest but I think it'll save everyone > > a good deal of effort if done correctly. > > > > FYI a lot of us have been working on kdevops > (https://github.com/linux-kdevops/kdevops), which has similar goals, > tho yours feels more in line with virt-me which I've used a bunch as > well. > > I would very much love it if we could all get behind one project. The > benefit of kdevops is it's very extensible, and being able to select > some config options and have an entire testing suite up and running is > very handy for new developers. Thanks for the tip; I had not heard of kdevops. I'll spend some time this week playing with it. > That being said I can definitely get behind have two sort of options, > the bigger swiss army knife that is kdevops, and something smaller > that's easier to do one-off runs. > > kdevops using vagrant makes a lot of the pain of setting up the full > VM environments that existed before go away. I can easily tear down > my 8 CI vm's and rebuild them all and having them testing again in 4 > commands. Nowadays I probably wouldn't use virt-me/anything lighter > because this is actually pretty easy to get up and running. > > Again no trying to be discouraging, but you're absolutely right that > there's been a lot of fragmentation here, which is why I've spent > probably a lot more time than I should have making kdevops work for > me, as well have a lot of other kernel developers, and it's getting > pretty solid. Thanks, Yep, ideally I'd like to have pulled something off the shelf and moved on. But nothing looked like it would solve my problem. So if kdevops is close enough then I wouldn't mind contributing to that instead. Thanks, Daniel ^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2023-03-06 19:08 UTC | newest] Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2023-02-27 23:01 [LSF/MM/BPF TOPIC] vmtest: Reusable virtual machine testing infrastructure Daniel Xu 2023-02-28 15:38 ` [Lsf-pc] " Josef Bacik 2023-03-06 19:07 ` Daniel Xu
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).