[RFC v3 0/3] Fix errors when try to build kvm selftests on specified output

[RFC v3 0/3] Fix errors when try to build kvm selftests on specified output

[Xiaoyao Li]

Hi Shuah,

Hope you're happy with this version that I only touch KVM's Makefile.

I attempted to build KVM selftests on a specified dir, unfortunately
neither	"make O=/path/to/mydir TARGETS=kvm" in tools/testing/selftests, nor
"make OUTPUT=/path/to/mydir" in tools/testing/selftests/kvm work.

This series aims to fix them.

Patch 1 and Patch 3 are the fix patch, please see the seperate for details.
While Patch 2 is more or less an fix for the future issue, since it works
without Patch 2 currently.

Changes in v3:
 - Drop all the patches that touch things out of KVM
 - Create OUTPUT dir in KVM's Makefile
 - Correct the INSTALL_HDR_PATH in KVM's Makefile

Changes in v2:
https://lore.kernel.org/kvm/20200325140133.103236-1-xiaoyao.li@intel.com/
 - fix the no directory issue in lib.mk
 - make kvm fixes seperate patch
 - Add the patch to fix linux src tree not clean issue

v1:
https://lore.kernel.org/kvm/20200315093425.33600-1-xiaoyao.li@intel.com/

Xiaoyao Li (3):
  kvm: selftests: Fix no directory error when OUTPUT specified
  kvm: selftests: Use the right INSTALL_HDR_PATH when OUTPUT specified
    and MAKELEVEL is 0
  kvm: selftests: Fix header path when built from parent level with O
    specified

 tools/testing/selftests/kvm/Makefile | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

-- 
2.20.1

[RFC v3 1/3] kvm: selftests: Fix no directory error when OUTPUT specified

[Xiaoyao Li]

When build kvm selftests to an specified directory with

	make OUTPUT=/home/lxy/kvm-selftests

it encouters following error：

  /usr/bin/ld: cannot open output file
  /home/lxy/kvm-selftests/x86_64/cr4_cpuid_sync_test: No such file or
  directory
  collect2: error: ld returned 1 exit status
  make: *** [../lib.mk:141:
  /home/lxy/kvm-selftests/x86_64/cr4_cpuid_sync_test] Error 1

Use "mkdir -p" to create the directory to fix this issue.

Signed-off-by: Xiaoyao Li <xiaoyao.li@intel.com>
---
 tools/testing/selftests/kvm/Makefile | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tools/testing/selftests/kvm/Makefile b/tools/testing/selftests/kvm/Makefile
index 67abc1dd50ee..91b41092def6 100644
--- a/tools/testing/selftests/kvm/Makefile
+++ b/tools/testing/selftests/kvm/Makefile
@@ -65,6 +65,7 @@ LDFLAGS += -pthread $(no-pie-option) $(pgste-option)
 # After inclusion, $(OUTPUT) is defined and
 # $(TEST_GEN_PROGS) starts with $(OUTPUT)/
 include ../lib.mk
+x := $(shell mkdir -p $(sort $(dir $(TEST_GEN_PROGS))))
 
 STATIC_LIBS := $(OUTPUT)/libkvm.a
 LIBKVM_OBJ := $(patsubst %.c, $(OUTPUT)/%.o, $(LIBKVM))
-- 
2.20.1

[RFC v3 2/3] kvm: selftests: Use the right INSTALL_HDR_PATH when OUTPUT specified and MAKELEVEL is 0

[Xiaoyao Li]

Assume linux headers exist in linux_src/usr works with OUTPUT specified
currently, even though headers are installed to $(OUTPUT)/usr based on
lib.mk. Because there are building output in linux_src/usr when
"make headers_install" to prepare headers and they are not cleaned.

Assign the correct path to INSTALL_HDR_PATH is always not bad.

Signed-off-by: Xiaoyao Li <xiaoyao.li@intel.com>
---
 tools/testing/selftests/kvm/Makefile | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/tools/testing/selftests/kvm/Makefile b/tools/testing/selftests/kvm/Makefile
index 91b41092def6..6a95878b2ab7 100644
--- a/tools/testing/selftests/kvm/Makefile
+++ b/tools/testing/selftests/kvm/Makefile
@@ -43,7 +43,15 @@ TEST_GEN_PROGS_s390x += kvm_create_max_vcpus
 TEST_GEN_PROGS += $(TEST_GEN_PROGS_$(UNAME_M))
 LIBKVM += $(LIBKVM_$(UNAME_M))
 
-INSTALL_HDR_PATH = $(top_srcdir)/usr
+ifeq (0,$(MAKELEVEL))
+    ifneq ($(OUTPUT),)
+	INSTALL_HDR_PATH := $(OUTPUT)/usr
+    else
+	INSTALL_HDR_PATH := $(top_srcdir)/usr
+    endif
+else
+	INSTALL_HDR_PATH := $(top_srcdir)/usr
+endif
 LINUX_HDR_PATH = $(INSTALL_HDR_PATH)/include/
 LINUX_TOOL_INCLUDE = $(top_srcdir)/tools/include
 LINUX_TOOL_ARCH_INCLUDE = $(top_srcdir)/tools/arch/x86/include
-- 
2.20.1

[RFC v3 3/3] kvm: selftests: Fix header path when built from parent level with O specified

[Xiaoyao Li]

When build kvm selftests in tools/testing/selftests directory with

	make O=/home/lxy/kselftests TARGETS=kvm

it fails building some kvm test binaries due to lack of header files,
e.g.,

  x86_64/vmx_set_nested_state_test.c: In function ‘set_default_vmx_state’:
  x86_64/vmx_set_nested_state_test.c:85:7: error: ‘struct
  kvm_nested_state’ has no member named ‘hdr’
    state->hdr.vmx.vmxon_pa = 0x1000;
  ...

With "O" is specified, it also takes effect on "make headers_install",
which causes no header files generated in "kernel-src/usr".

Fix INSTALL_HDR_PATH with correct path for this case.

Opportunistically fix the case when KBUILD_OUTPUT specified as well,
even though it can work currently.

Signed-off-by: Xiaoyao Li <xiaoyao.li@intel.com>
---
 tools/testing/selftests/kvm/Makefile | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/tools/testing/selftests/kvm/Makefile b/tools/testing/selftests/kvm/Makefile
index 6a95878b2ab7..bf8f56488914 100644
--- a/tools/testing/selftests/kvm/Makefile
+++ b/tools/testing/selftests/kvm/Makefile
@@ -50,7 +50,11 @@ ifeq (0,$(MAKELEVEL))
 	INSTALL_HDR_PATH := $(top_srcdir)/usr
     endif
 else
+    ifneq ($(O)$(KBUILD_OUTPUT),)
+	INSTALL_HDR_PATH := $(BUILD)/usr
+    else
 	INSTALL_HDR_PATH := $(top_srcdir)/usr
+    endif
 endif
 LINUX_HDR_PATH = $(INSTALL_HDR_PATH)/include/
 LINUX_TOOL_INCLUDE = $(top_srcdir)/tools/include
-- 
2.20.1

Re: [RFC v3 0/3] Fix errors when try to build kvm selftests on specified output

[shuah]

On 3/25/20 9:07 PM, Xiaoyao Li wrote:
> Hi Shuah,
> 
> Hope you're happy with this version that I only touch KVM's Makefile.
> 
> I attempted to build KVM selftests on a specified dir, unfortunately
> neither	"make O=/path/to/mydir TARGETS=kvm" in tools/testing/selftests, nor
> "make OUTPUT=/path/to/mydir" in tools/testing/selftests/kvm work.
> 

Why are you running "make OUTPUT=/path/to/mydir"

It isn't correct.

make O=/path/to/mydir is what you have to use. Please main Makefile
as well for O= and KBUILD_OUTPUT usages.

Please see Documentation/dev-tools/kselftest.rst for use-cases.

make O=/path/to/mydir TARGETS=kvm is a right use-case and I can see
it will fail to create x86_64 directory.

Let's start with the following two commands and try to fix the
problems you are seeing.

make O=/path/to/mydir in kvm directory (this is supported,
however, the following command from the main Makefile is
recommended use.)

 From main Makefile in kernel srcdir
make O=/path/to/mydir TARGETS=kvm

Also, just build isn't sufficient for you to be able to run the
tests.

make kselftest-install O=/path/to/mydir TARGETS=kvm will generate
run script.

thanks,
-- Shuah

Re: [RFC v3 0/3] Fix errors when try to build kvm selftests on specified output

[Xiaoyao Li]

On 3/27/2020 4:57 AM, shuah wrote:
> On 3/25/20 9:07 PM, Xiaoyao Li wrote:
>> Hi Shuah,
>>
>> Hope you're happy with this version that I only touch KVM's Makefile.
>>
>> I attempted to build KVM selftests on a specified dir, unfortunately
>> neither    "make O=/path/to/mydir TARGETS=kvm" in 
>> tools/testing/selftests, nor
>> "make OUTPUT=/path/to/mydir" in tools/testing/selftests/kvm work.
>>
> 
> Why are you running "make OUTPUT=/path/to/mydir"
> 
> It isn't correct.

So what's the meaning of

ifeq (0,$(MAKELEVEL))
     ifeq ($(OUTPUT),)
	OUTPUT := $(shell pwd)
	DEFAULT_INSTALL_HDR_PATH := 1
     endif
endif

in lib.mk?

> 
> make O=/path/to/mydir is what you have to use. Please main Makefile
> as well for O= and KBUILD_OUTPUT usages.
> 
> Please see Documentation/dev-tools/kselftest.rst for use-cases.
> 
> make O=/path/to/mydir TARGETS=kvm is a right use-case and I can see
> it will fail to create x86_64 directory.
> 
> Let's start with the following two commands and try to fix the
> problems you are seeing.
> 
> make O=/path/to/mydir in kvm directory (this is supported,
> however, the following command from the main Makefile is
> recommended use.)

Of course we can do this, but the "O=/path/to/mydir" only has effect on 
header install, the *.o files still generated in kvm/ directory.

And kvm's INSTALL_HDR_PATH cannot find the right headers.

That's why I choose to use "OUTPUT=/somewhere"

>  From main Makefile in kernel srcdir
> make O=/path/to/mydir TARGETS=kvm

I guess "kernel srcdir" means "kselftest srcdir", i.e., 
tools/testing/selftests/ ?

Well, as I said in the first place, I tried

	make O=/path/to/mydir TARGETS=kvm

but it doesn't work. So I did some fixup, and sent out the Patches.

If the patches are wrong, please point it out and give your comments how 
to make it right.

> Also, just build isn't sufficient for you to be able to run the
> tests.
> 
> make kselftest-install O=/path/to/mydir TARGETS=kvm will generate
> run script.

This command also has the x86_64 directory not created issue.
Since it generates header files in kernel_src/usr/include, it doesn't 
have headers path issue. But as result, the kernel_src directory is not 
clean, this requires me to run "make mrproper", I *really* don't like it.

Re: [RFC v3 0/3] Fix errors when try to build kvm selftests on specified output

[shuah]

On 3/27/20 7:31 AM, Xiaoyao Li wrote:
> On 3/27/2020 4:57 AM, shuah wrote:
>> On 3/25/20 9:07 PM, Xiaoyao Li wrote:
>>> Hi Shuah,
>>>
>>> Hope you're happy with this version that I only touch KVM's Makefile.
>>>
>>> I attempted to build KVM selftests on a specified dir, unfortunately
>>> neither    "make O=/path/to/mydir TARGETS=kvm" in 
>>> tools/testing/selftests, nor
>>> "make OUTPUT=/path/to/mydir" in tools/testing/selftests/kvm work.
>>>
>>
>> Why are you running "make OUTPUT=/path/to/mydir"
>>
>> It isn't correct.
> 
> So what's the meaning of
> 
> ifeq (0,$(MAKELEVEL))
>      ifeq ($(OUTPUT),)
>      OUTPUT := $(shell pwd)
>      DEFAULT_INSTALL_HDR_PATH := 1
>      endif
> endif
> 
> in lib.mk?
> 

O is the variable for selftests Makefile and which is handled in
lib.mk. OUTPUT is internal and shouldn't set when running make.

>>
>> make O=/path/to/mydir is what you have to use. Please main Makefile
>> as well for O= and KBUILD_OUTPUT usages.
>>
>> Please see Documentation/dev-tools/kselftest.rst for use-cases.
>>
>> make O=/path/to/mydir TARGETS=kvm is a right use-case and I can see
>> it will fail to create x86_64 directory.
>>
>> Let's start with the following two commands and try to fix the
>> problems you are seeing.
>>
>> make O=/path/to/mydir in kvm directory (this is supported,
>> however, the following command from the main Makefile is
>> recommended use.)
> 
> Of course we can do this, but the "O=/path/to/mydir" only has effect on 
> header install, the *.o files still generated in kvm/ directory.
> 

Right. That is what needs to be fixed. This is the reason for your
source directory getting dirty when kvm test is built.

> And kvm's INSTALL_HDR_PATH cannot find the right headers.
> 
> That's why I choose to use "OUTPUT=/somewhere"
> 
>>  From main Makefile in kernel srcdir
>> make O=/path/to/mydir TARGETS=kvm
> 
> I guess "kernel srcdir" means "kselftest srcdir", i.e., 
> tools/testing/selftests/ ?

This is kernel source root directory. The command you would
use is:

make kselftest-all O=/path/to/mydir TARGETS=kvm

> 
> Well, as I said in the first place, I tried
> 
>      make O=/path/to/mydir TARGETS=kvm
> 
> but it doesn't work. So I did some fixup, and sent out the Patches.
> 

Right. It doesn't work for a couple of reasons:

1. The Makefile doesn't create sub-dirs when build is relocatable.
2. Makes source directory dirty.

> If the patches are wrong, please point it out and give your comments how 
> to make it right.
> 

The patches you sent are based on running the command with OUTPUT
set. That is why I am asking you start with the right use-cases,
and gave you pointers on tests to refer to that have sub-dirs
and handle relocatable builds:

futex
arm64
android

>> Also, just build isn't sufficient for you to be able to run the
>> tests.
>>
>> make kselftest-install O=/path/to/mydir TARGETS=kvm will generate
>> run script.
> 
> This command also has the x86_64 directory not created issue.
> Since it generates header files in kernel_src/usr/include, it doesn't 
> have headers path issue. But as result, the kernel_src directory is not 
> clean, this requires me to run "make mrproper", I *really* don't like it.
> 
> 

If the test leverages lib.mk headers install logic correctly, you
shouldn't see this problem.

Yes. It does make the source directory dirty. That is the problem we
have to fix. I am seeing issues the issue of x86_64 not being created
in the case of relocatable builds.

Thanks for working on this by the way. It is one of the tests that
identified as the one doesn't support relocatable builds.

You will see fixes to others I already fixed in

https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git/log/?h=next

Start withe the following use0-cases to fix and then test your fixes
for these use-cases. The goal is to be able to run kvm from target
directory and source directory staying clean.

You will have to build the kernel first in all of these cases.
Let's use kselftest-install which is what you would need if you
want build and then run tests later. Also assuming you are doing
native build on x86_64.

 From main kernel Makefile: (from kernel source root dir)

Builds in the same source directory:
make kselftest-install TARGETS=kvm

Relocatable build: (from kernel source root dir)

make O=/path/objdir  - build kernel
make kselftest-install O=/path/objdir TARGETS=kvm

 From tools/testing/selftests/kvm directory:
make O=/path/objdir install

Install step is important especially for relocatable builds,
as it makes sure all run-time dependencies are copied to the
target directory.

thanks,
-- Shuah

Re: [RFC v3 0/3] Fix errors when try to build kvm selftests on specified output

[Xiaoyao Li]

On 3/28/2020 12:03 AM, shuah wrote:
> On 3/27/20 7:31 AM, Xiaoyao Li wrote:
>> On 3/27/2020 4:57 AM, shuah wrote:
>>> On 3/25/20 9:07 PM, Xiaoyao Li wrote:
> The patches you sent are based on running the command with OUTPUT
> set. 

The issues are also there when I use

   make O=/somewher TARGETS=kvm

from tools/testing/selftests/ directory,

Is it the right usecase?

> That is why I am asking you start with the right use-cases,
> and gave you pointers on tests to refer to that have sub-dirs
> and handle relocatable builds:
> 
> futex
> arm64
> android

I have read the Makefile in futex, arm64, android to learn how they
deal with subdir, they have Makefile in subdir but kvm doesn't.

I just want to create the subdir as easy as possible, so I follow the 
method how to create subdir for LIBKVM_OBJ in kvm.

If you dislike it way you can reply it Patch 1 to disclaim clear what 
you want to fix the subdir.

>>> Also, just build isn't sufficient for you to be able to run the
>>> tests.
>>>
>>> make kselftest-install O=/path/to/mydir TARGETS=kvm will generate
>>> run script.
>>
>> This command also has the x86_64 directory not created issue.
>> Since it generates header files in kernel_src/usr/include, it doesn't 
>> have headers path issue. But as result, the kernel_src directory is 
>> not clean, this requires me to run "make mrproper", I *really* don't 
>> like it.
>>
>>
> 
> If the test leverages lib.mk headers install logic correctly, you
> shouldn't see this problem.
> 
> Yes. It does make the source directory dirty. That is the problem we
> have to fix. I am seeing issues the issue of x86_64 not being created
> in the case of relocatable builds.
> 
> Thanks for working on this by the way. It is one of the tests that
> identified as the one doesn't support relocatable builds.
> 
> You will see fixes to others I already fixed in
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git/log/?h=next 
> 
> 
> Start withe the following use0-cases to fix and then test your fixes
> for these use-cases. The goal is to be able to run kvm from target
> directory and source directory staying clean.
> 
> You will have to build the kernel first in all of these cases.
> Let's use kselftest-install which is what you would need if you
> want build and then run tests later. Also assuming you are doing
> native build on x86_64.
> 
>  From main kernel Makefile: (from kernel source root dir)
> 
> Builds in the same source directory:
> make kselftest-install TARGETS=kvm
> 
> Relocatable build: (from kernel source root dir)
> 
> make O=/path/objdir  - build kernel
> make kselftest-install O=/path/objdir TARGETS=kvm

I don't want to build kernel at all.

>  From tools/testing/selftests/kvm directory:
> make O=/path/objdir install

Oh no. This needs to define INSTALL_PATH, and I don't want to install.
I just want to build the testcase of kvm to anywhere else to make srcdir 
clean.

Besides this command make both kernel src and selftest/kvm dirty.

> Install step is important especially for relocatable builds,
> as it makes sure all run-time dependencies are copied to the
> target directory.
> 

OK.

So, again.

is

   make O=/tmp/kvm-selftest -C tools/testing/selftests \
        TARGETS=kvm install INSTALL_PATH=/home/lxy/kvm-selftest

the right command for me to start on?

Re: [RFC v3 0/3] Fix errors when try to build kvm selftests on specified output

[shuah]

On 3/28/20 2:51 AM, Xiaoyao Li wrote:
> On 3/28/2020 12:03 AM, shuah wrote:
>> On 3/27/20 7:31 AM, Xiaoyao Li wrote:
>>> On 3/27/2020 4:57 AM, shuah wrote:
>>>> On 3/25/20 9:07 PM, Xiaoyao Li wrote:
>> The patches you sent are based on running the command with OUTPUT
>> set. 
> 
> The issues are also there when I use
> 
>    make O=/somewher TARGETS=kvm
> 
> from tools/testing/selftests/ directory,
> 
> Is it the right usecase?
> 
>> That is why I am asking you start with the right use-cases,
>> and gave you pointers on tests to refer to that have sub-dirs
>> and handle relocatable builds:
>>
>> futex
>> arm64
>> android
> 
> I have read the Makefile in futex, arm64, android to learn how they
> deal with subdir, they have Makefile in subdir but kvm doesn't.
> 
> I just want to create the subdir as easy as possible, so I follow the 
> method how to create subdir for LIBKVM_OBJ in kvm.
> 
> If you dislike it way you can reply it Patch 1 to disclaim clear what 
> you want to fix the subdir.
> 
>>>> Also, just build isn't sufficient for you to be able to run the
>>>> tests.
>>>>
>>>> make kselftest-install O=/path/to/mydir TARGETS=kvm will generate
>>>> run script.
>>>
>>> This command also has the x86_64 directory not created issue.
>>> Since it generates header files in kernel_src/usr/include, it doesn't 
>>> have headers path issue. But as result, the kernel_src directory is 
>>> not clean, this requires me to run "make mrproper", I *really* don't 
>>> like it.
>>>
>>>
>>
>> If the test leverages lib.mk headers install logic correctly, you
>> shouldn't see this problem.
>>
>> Yes. It does make the source directory dirty. That is the problem we
>> have to fix. I am seeing issues the issue of x86_64 not being created
>> in the case of relocatable builds.
>>
>> Thanks for working on this by the way. It is one of the tests that
>> identified as the one doesn't support relocatable builds.
>>
>> You will see fixes to others I already fixed in
>>
>> https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git/log/?h=next 
>>
>>
>> Start withe the following use0-cases to fix and then test your fixes
>> for these use-cases. The goal is to be able to run kvm from target
>> directory and source directory staying clean.
>>
>> You will have to build the kernel first in all of these cases.
>> Let's use kselftest-install which is what you would need if you
>> want build and then run tests later. Also assuming you are doing
>> native build on x86_64.
>>
>>  From main kernel Makefile: (from kernel source root dir)
>>
>> Builds in the same source directory:
>> make kselftest-install TARGETS=kvm
>>
>> Relocatable build: (from kernel source root dir)
>>
>> make O=/path/objdir  - build kernel
>> make kselftest-install O=/path/objdir TARGETS=kvm
> 
> I don't want to build kernel at all.
> 
>>  From tools/testing/selftests/kvm directory:
>> make O=/path/objdir install
> 
> Oh no. This needs to define INSTALL_PATH, and I don't want to install.
> I just want to build the testcase of kvm to anywhere else to make srcdir 
> clean.
> 
> Besides this command make both kernel src and selftest/kvm dirty.
> 
>> Install step is important especially for relocatable builds,
>> as it makes sure all run-time dependencies are copied to the
>> target directory.
>>
> 
> OK.
> 
> So, again.
> 
> is
> 
>    make O=/tmp/kvm-selftest -C tools/testing/selftests \
>         TARGETS=kvm install INSTALL_PATH=/home/lxy/kvm-selftest
> 
> the right command for me to start on?
> 

Sorry for the delay. Yes the above command will work. Please check

https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git 
next branch

I fixed kvm build/cross-build problems and applied it to next for
testing.

https://patchwork.kernel.org/patch/11513471/

thanks,
-- Shuah