Arm userland not building with gcc-9 and later

Arm userland not building with gcc-9 and later

[Jan Kiszka]

Hi all,

just wanted to debug the RTnet issues we now see in CI on arm and arm64.
I picked arm as first target, but that apparently starts to break with
gcc-9 or newer (tried 9.2 and 10.2):

make[5]: Entering directory '/xenomai/lib/cobalt/arch/arm'
  CC       libarch_la-features.lo
features.c: In function 'cobalt_arch_check_features':
features.c:82:1: error: r7 cannot be used in 'asm' here
   82 | }
      | ^

That seems to be related to passing the syscall number via r7 on ARM. Is
our ABI soon no longer compilable, or can we fix this?

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: Arm userland not building with gcc-9 and later

[Philippe Gerum]

Jan Kiszka <jan.kiszka@siemens.com> writes:

> Hi all,
>
> just wanted to debug the RTnet issues we now see in CI on arm and arm64.
> I picked arm as first target, but that apparently starts to break with
> gcc-9 or newer (tried 9.2 and 10.2):
>
> make[5]: Entering directory '/xenomai/lib/cobalt/arch/arm'
>   CC       libarch_la-features.lo
> features.c: In function 'cobalt_arch_check_features':
> features.c:82:1: error: r7 cannot be used in 'asm' here
>    82 | }
>       | ^
>
> That seems to be related to passing the syscall number via r7 on ARM. Is
> our ABI soon no longer compilable, or can we fix this?
>
> Jan

r7 may be used as a scratch register by gcc in some
cases. -fomit-frame-pointer for debug builds may help (i.e. when the
optimizer is switched off).

-- 
Philippe.

Re: Arm userland not building with gcc-9 and later

[Jan Kiszka]

On 06.08.21 15:09, Philippe Gerum wrote:
> 
> Jan Kiszka <jan.kiszka@siemens.com> writes:
> 
>> Hi all,
>>
>> just wanted to debug the RTnet issues we now see in CI on arm and arm64.
>> I picked arm as first target, but that apparently starts to break with
>> gcc-9 or newer (tried 9.2 and 10.2):
>>
>> make[5]: Entering directory '/xenomai/lib/cobalt/arch/arm'
>>   CC       libarch_la-features.lo
>> features.c: In function 'cobalt_arch_check_features':
>> features.c:82:1: error: r7 cannot be used in 'asm' here
>>    82 | }
>>       | ^
>>
>> That seems to be related to passing the syscall number via r7 on ARM. Is
>> our ABI soon no longer compilable, or can we fix this?
>>
>> Jan
> 
> r7 may be used as a scratch register by gcc in some
> cases. -fomit-frame-pointer for debug builds may help (i.e. when the
> optimizer is switched off).
> 

Good hint. I had --enable-debug=full set, and it builds without it (and
now reports other issues that Debian's gcc-10 sees).

But how to resolve this properly?

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: Arm userland not building with gcc-9 and later

[Philippe Gerum]

Jan Kiszka <jan.kiszka@siemens.com> writes:

> On 06.08.21 15:09, Philippe Gerum wrote:
>> 
>> Jan Kiszka <jan.kiszka@siemens.com> writes:
>> 
>>> Hi all,
>>>
>>> just wanted to debug the RTnet issues we now see in CI on arm and arm64.
>>> I picked arm as first target, but that apparently starts to break with
>>> gcc-9 or newer (tried 9.2 and 10.2):
>>>
>>> make[5]: Entering directory '/xenomai/lib/cobalt/arch/arm'
>>>   CC       libarch_la-features.lo
>>> features.c: In function 'cobalt_arch_check_features':
>>> features.c:82:1: error: r7 cannot be used in 'asm' here
>>>    82 | }
>>>       | ^
>>>
>>> That seems to be related to passing the syscall number via r7 on ARM. Is
>>> our ABI soon no longer compilable, or can we fix this?
>>>
>>> Jan
>> 
>> r7 may be used as a scratch register by gcc in some
>> cases. -fomit-frame-pointer for debug builds may help (i.e. when the
>> optimizer is switched off).
>> 
>
> Good hint. I had --enable-debug=full set, and it builds without it (and
> now reports other issues that Debian's gcc-10 sees).
>
> But how to resolve this properly?
>
> Jan

XENOMAI_SYSCALL2(sc_cobalt_archcall) in features.c is likely the issue.

Either move those bits into some new syscall wrapper which would live in
lib/cobalt/internal.c, where there is no register allocation conflict so
far. Or discourage gcc from using r7 as a scratch register in
arm/features.c entirely? e.g.:

diff --git a/lib/cobalt/arch/arm/Makefile.am b/lib/cobalt/arch/arm/Makefile.am
index a5095be3d..d5e542ebe 100644
--- a/lib/cobalt/arch/arm/Makefile.am
+++ b/lib/cobalt/arch/arm/Makefile.am
@@ -6,6 +6,7 @@ libarch_la_SOURCES = features.c
 
 libarch_la_CPPFLAGS =			\
 	@XENO_COBALT_CFLAGS@ 		\
+	-fomit-frame-pointer		\
 	-I$(srcdir)/../..		\
 	-I$(top_srcdir)/include/cobalt	\
 	-I$(top_srcdir)/include

-- 
Philippe.

Re: Arm userland not building with gcc-9 and later

[Jan Kiszka]

On 06.08.21 15:34, Philippe Gerum wrote:
> 
> Jan Kiszka <jan.kiszka@siemens.com> writes:
> 
>> On 06.08.21 15:09, Philippe Gerum wrote:
>>>
>>> Jan Kiszka <jan.kiszka@siemens.com> writes:
>>>
>>>> Hi all,
>>>>
>>>> just wanted to debug the RTnet issues we now see in CI on arm and arm64.
>>>> I picked arm as first target, but that apparently starts to break with
>>>> gcc-9 or newer (tried 9.2 and 10.2):
>>>>
>>>> make[5]: Entering directory '/xenomai/lib/cobalt/arch/arm'
>>>>   CC       libarch_la-features.lo
>>>> features.c: In function 'cobalt_arch_check_features':
>>>> features.c:82:1: error: r7 cannot be used in 'asm' here
>>>>    82 | }
>>>>       | ^
>>>>
>>>> That seems to be related to passing the syscall number via r7 on ARM. Is
>>>> our ABI soon no longer compilable, or can we fix this?
>>>>
>>>> Jan
>>>
>>> r7 may be used as a scratch register by gcc in some
>>> cases. -fomit-frame-pointer for debug builds may help (i.e. when the
>>> optimizer is switched off).
>>>
>>
>> Good hint. I had --enable-debug=full set, and it builds without it (and
>> now reports other issues that Debian's gcc-10 sees).
>>
>> But how to resolve this properly?
>>
>> Jan
> 
> XENOMAI_SYSCALL2(sc_cobalt_archcall) in features.c is likely the issue.
> 
> Either move those bits into some new syscall wrapper which would live in
> lib/cobalt/internal.c, where there is no register allocation conflict so
> far. Or discourage gcc from using r7 as a scratch register in
> arm/features.c entirely? e.g.:
> 
> diff --git a/lib/cobalt/arch/arm/Makefile.am b/lib/cobalt/arch/arm/Makefile.am
> index a5095be3d..d5e542ebe 100644
> --- a/lib/cobalt/arch/arm/Makefile.am
> +++ b/lib/cobalt/arch/arm/Makefile.am
> @@ -6,6 +6,7 @@ libarch_la_SOURCES = features.c
>  
>  libarch_la_CPPFLAGS =			\
>  	@XENO_COBALT_CFLAGS@ 		\
> +	-fomit-frame-pointer		\
>  	-I$(srcdir)/../..		\
>  	-I$(top_srcdir)/include/cobalt	\
>  	-I$(top_srcdir)/include
> 

-fomit-frame-pointer does not help with -O0. It even keeps the problem
alive with -O2.

Looks like arm needs a non-unlined helper function for syscalls, to be safe.

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: Arm userland not building with gcc-9 and later

[Jan Kiszka]

On 06.08.21 18:16, Jan Kiszka wrote:
> On 06.08.21 15:34, Philippe Gerum wrote:
>>
>> Jan Kiszka <jan.kiszka@siemens.com> writes:
>>
>>> On 06.08.21 15:09, Philippe Gerum wrote:
>>>>
>>>> Jan Kiszka <jan.kiszka@siemens.com> writes:
>>>>
>>>>> Hi all,
>>>>>
>>>>> just wanted to debug the RTnet issues we now see in CI on arm and arm64.
>>>>> I picked arm as first target, but that apparently starts to break with
>>>>> gcc-9 or newer (tried 9.2 and 10.2):
>>>>>
>>>>> make[5]: Entering directory '/xenomai/lib/cobalt/arch/arm'
>>>>>   CC       libarch_la-features.lo
>>>>> features.c: In function 'cobalt_arch_check_features':
>>>>> features.c:82:1: error: r7 cannot be used in 'asm' here
>>>>>    82 | }
>>>>>       | ^
>>>>>
>>>>> That seems to be related to passing the syscall number via r7 on ARM. Is
>>>>> our ABI soon no longer compilable, or can we fix this?
>>>>>
>>>>> Jan
>>>>
>>>> r7 may be used as a scratch register by gcc in some
>>>> cases. -fomit-frame-pointer for debug builds may help (i.e. when the
>>>> optimizer is switched off).
>>>>
>>>
>>> Good hint. I had --enable-debug=full set, and it builds without it (and
>>> now reports other issues that Debian's gcc-10 sees).
>>>
>>> But how to resolve this properly?
>>>
>>> Jan
>>
>> XENOMAI_SYSCALL2(sc_cobalt_archcall) in features.c is likely the issue.
>>
>> Either move those bits into some new syscall wrapper which would live in
>> lib/cobalt/internal.c, where there is no register allocation conflict so
>> far. Or discourage gcc from using r7 as a scratch register in
>> arm/features.c entirely? e.g.:
>>
>> diff --git a/lib/cobalt/arch/arm/Makefile.am b/lib/cobalt/arch/arm/Makefile.am
>> index a5095be3d..d5e542ebe 100644
>> --- a/lib/cobalt/arch/arm/Makefile.am
>> +++ b/lib/cobalt/arch/arm/Makefile.am
>> @@ -6,6 +6,7 @@ libarch_la_SOURCES = features.c
>>  
>>  libarch_la_CPPFLAGS =			\
>>  	@XENO_COBALT_CFLAGS@ 		\
>> +	-fomit-frame-pointer		\
>>  	-I$(srcdir)/../..		\
>>  	-I$(top_srcdir)/include/cobalt	\
>>  	-I$(top_srcdir)/include
>>
> 
> -fomit-frame-pointer does not help with -O0. It even keeps the problem
> alive with -O2.
> 
> Looks like arm needs a non-unlined helper function for syscalls, to be safe.
> 

What might be cheaper, a call-out to a helper or some push/pop around
the syscall like below? But I suspect the helper will do push/pop as well...

#define __SYS_CALLOP "push {r7}; mov %%r7,%[__r7]; swi\t0; pop {r7}"

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: Arm userland not building with gcc-9 and later

[Philippe Gerum]

Jan Kiszka <jan.kiszka@siemens.com> writes:

> Hi all,
>
> just wanted to debug the RTnet issues we now see in CI on arm and arm64.
> I picked arm as first target, but that apparently starts to break with
> gcc-9 or newer (tried 9.2 and 10.2):
>
> make[5]: Entering directory '/xenomai/lib/cobalt/arch/arm'
>   CC       libarch_la-features.lo
> features.c: In function 'cobalt_arch_check_features':
> features.c:82:1: error: r7 cannot be used in 'asm' here
>    82 | }
>       | ^
>
> That seems to be related to passing the syscall number via r7 on ARM. Is
> our ABI soon no longer compilable, or can we fix this?
>

Just tried to reproduce this issue with the latest stable (10.3-2021.07)
from ARM [1], and a fairly recent linaro snapshot (11.0.1-2021.03) [2],
to no avail. I tried with --enable-debug=symbols, --disable-debug,
--enable-debug=full, no issue so far.

Can you please share you configure settings?

[1]
https://developer.arm.com/-/media/Files/downloads/gnu-a/10.3-2021.07/binrel/gcc-arm-10.3-2021.07-x86_64-arm-none-linux-gnueabihf.tar.xz

[2]
https://snapshots.linaro.org/gnu-toolchain/11.0-2021.03-1/arm-linux-gnueabihf/gcc-linaro-11.0.1-2021.03-aarch64_arm-linux-gnueabihf.tar.xz

-- 
Philippe.

Re: Arm userland not building with gcc-9 and later

[Jan Kiszka]

On 07.08.21 19:00, Philippe Gerum wrote:
> 
> Jan Kiszka <jan.kiszka@siemens.com> writes:
> 
>> Hi all,
>>
>> just wanted to debug the RTnet issues we now see in CI on arm and arm64.
>> I picked arm as first target, but that apparently starts to break with
>> gcc-9 or newer (tried 9.2 and 10.2):
>>
>> make[5]: Entering directory '/xenomai/lib/cobalt/arch/arm'
>>   CC       libarch_la-features.lo
>> features.c: In function 'cobalt_arch_check_features':
>> features.c:82:1: error: r7 cannot be used in 'asm' here
>>    82 | }
>>       | ^
>>
>> That seems to be related to passing the syscall number via r7 on ARM. Is
>> our ABI soon no longer compilable, or can we fix this?
>>
> 
> Just tried to reproduce this issue with the latest stable (10.3-2021.07)
> from ARM [1], and a fairly recent linaro snapshot (11.0.1-2021.03) [2],
> to no avail. I tried with --enable-debug=symbols, --disable-debug,
> --enable-debug=full, no issue so far.
> 
> Can you please share you configure settings?

I'm on ARM gcc 10.3-2021.07 as well. Debian's gcc from bullseye show 
this, too.

./configure --host=arm-none-linux-gnueabihf --enable-debug=full

on master. And that results in

  CC       libarch_la-features.lo
../../../../../lib/cobalt/arch/arm/features.c: In function ‘cobalt_arch_check_features’:
../../../../../lib/cobalt/arch/arm/features.c:82:1: error: r7 cannot be used in ‘asm’ here
   82 | }
      | ^

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: Arm userland not building with gcc-9 and later

[Philippe Gerum]

Jan Kiszka <jan.kiszka@siemens.com> writes:

> On 07.08.21 19:00, Philippe Gerum wrote:
>> 
>> Jan Kiszka <jan.kiszka@siemens.com> writes:
>> 
>>> Hi all,
>>>
>>> just wanted to debug the RTnet issues we now see in CI on arm and arm64.
>>> I picked arm as first target, but that apparently starts to break with
>>> gcc-9 or newer (tried 9.2 and 10.2):
>>>
>>> make[5]: Entering directory '/xenomai/lib/cobalt/arch/arm'
>>>   CC       libarch_la-features.lo
>>> features.c: In function 'cobalt_arch_check_features':
>>> features.c:82:1: error: r7 cannot be used in 'asm' here
>>>    82 | }
>>>       | ^
>>>
>>> That seems to be related to passing the syscall number via r7 on ARM. Is
>>> our ABI soon no longer compilable, or can we fix this?
>>>
>> 
>> Just tried to reproduce this issue with the latest stable (10.3-2021.07)
>> from ARM [1], and a fairly recent linaro snapshot (11.0.1-2021.03) [2],
>> to no avail. I tried with --enable-debug=symbols, --disable-debug,
>> --enable-debug=full, no issue so far.
>> 
>> Can you please share you configure settings?
>
> I'm on ARM gcc 10.3-2021.07 as well. Debian's gcc from bullseye show 
> this, too.
>
> ./configure --host=arm-none-linux-gnueabihf --enable-debug=full
>
> on master. And that results in
>
>   CC       libarch_la-features.lo
> ../../../../../lib/cobalt/arch/arm/features.c: In function ‘cobalt_arch_check_features’:
> ../../../../../lib/cobalt/arch/arm/features.c:82:1: error: r7 cannot be used in ‘asm’ here
>    82 | }
>       | ^
>

Confirmed, found it. next is fine with this compiler, which is the
version I tested. However, building stable and master does break.

-- 
Philippe.

Re: Arm userland not building with gcc-9 and later

[Jan Kiszka]

On 08.08.21 19:21, Philippe Gerum wrote:
> 
> Jan Kiszka <jan.kiszka@siemens.com> writes:
> 
>> On 07.08.21 19:00, Philippe Gerum wrote:
>>>
>>> Jan Kiszka <jan.kiszka@siemens.com> writes:
>>>
>>>> Hi all,
>>>>
>>>> just wanted to debug the RTnet issues we now see in CI on arm and arm64.
>>>> I picked arm as first target, but that apparently starts to break with
>>>> gcc-9 or newer (tried 9.2 and 10.2):
>>>>
>>>> make[5]: Entering directory '/xenomai/lib/cobalt/arch/arm'
>>>>   CC       libarch_la-features.lo
>>>> features.c: In function 'cobalt_arch_check_features':
>>>> features.c:82:1: error: r7 cannot be used in 'asm' here
>>>>    82 | }
>>>>       | ^
>>>>
>>>> That seems to be related to passing the syscall number via r7 on ARM. Is
>>>> our ABI soon no longer compilable, or can we fix this?
>>>>
>>>
>>> Just tried to reproduce this issue with the latest stable (10.3-2021.07)
>>> from ARM [1], and a fairly recent linaro snapshot (11.0.1-2021.03) [2],
>>> to no avail. I tried with --enable-debug=symbols, --disable-debug,
>>> --enable-debug=full, no issue so far.
>>>
>>> Can you please share you configure settings?
>>
>> I'm on ARM gcc 10.3-2021.07 as well. Debian's gcc from bullseye show 
>> this, too.
>>
>> ./configure --host=arm-none-linux-gnueabihf --enable-debug=full
>>
>> on master. And that results in
>>
>>   CC       libarch_la-features.lo
>> ../../../../../lib/cobalt/arch/arm/features.c: In function ‘cobalt_arch_check_features’:
>> ../../../../../lib/cobalt/arch/arm/features.c:82:1: error: r7 cannot be used in ‘asm’ here
>>    82 | }
>>       | ^
>>
> 
> Confirmed, found it. next is fine with this compiler, which is the
> version I tested. However, building stable and master does break.
> 

Yeah, I've already pushed the two related fixes into next for testing.
Can still be changed if we find a better solution.

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: Arm userland not building with gcc-9 and later

[Philippe Gerum]

Jan Kiszka <jan.kiszka@siemens.com> writes:

> On 08.08.21 19:21, Philippe Gerum wrote:
>> 
>> Jan Kiszka <jan.kiszka@siemens.com> writes:
>> 
>>> On 07.08.21 19:00, Philippe Gerum wrote:
>>>>
>>>> Jan Kiszka <jan.kiszka@siemens.com> writes:
>>>>
>>>>> Hi all,
>>>>>
>>>>> just wanted to debug the RTnet issues we now see in CI on arm and arm64.
>>>>> I picked arm as first target, but that apparently starts to break with
>>>>> gcc-9 or newer (tried 9.2 and 10.2):
>>>>>
>>>>> make[5]: Entering directory '/xenomai/lib/cobalt/arch/arm'
>>>>>   CC       libarch_la-features.lo
>>>>> features.c: In function 'cobalt_arch_check_features':
>>>>> features.c:82:1: error: r7 cannot be used in 'asm' here
>>>>>    82 | }
>>>>>       | ^
>>>>>
>>>>> That seems to be related to passing the syscall number via r7 on ARM. Is
>>>>> our ABI soon no longer compilable, or can we fix this?
>>>>>
>>>>
>>>> Just tried to reproduce this issue with the latest stable (10.3-2021.07)
>>>> from ARM [1], and a fairly recent linaro snapshot (11.0.1-2021.03) [2],
>>>> to no avail. I tried with --enable-debug=symbols, --disable-debug,
>>>> --enable-debug=full, no issue so far.
>>>>
>>>> Can you please share you configure settings?
>>>
>>> I'm on ARM gcc 10.3-2021.07 as well. Debian's gcc from bullseye show 
>>> this, too.
>>>
>>> ./configure --host=arm-none-linux-gnueabihf --enable-debug=full
>>>
>>> on master. And that results in
>>>
>>>   CC       libarch_la-features.lo
>>> ../../../../../lib/cobalt/arch/arm/features.c: In function ‘cobalt_arch_check_features’:
>>> ../../../../../lib/cobalt/arch/arm/features.c:82:1: error: r7 cannot be used in ‘asm’ here
>>>    82 | }
>>>       | ^
>>>
>> 
>> Confirmed, found it. next is fine with this compiler, which is the
>> version I tested. However, building stable and master does break.
>> 
>
> Yeah, I've already pushed the two related fixes into next for testing.
> Can still be changed if we find a better solution.
>

Ack.

-- 
Philippe.

Re: Arm userland not building with gcc-9 and later

[Jan Kiszka]

On 06.08.21 15:00, Jan Kiszka via Xenomai wrote:
> Hi all,
> 
> just wanted to debug the RTnet issues we now see in CI on arm and arm64.
> I picked arm as first target, but that apparently starts to break with
> gcc-9 or newer (tried 9.2 and 10.2):
> 
> make[5]: Entering directory '/xenomai/lib/cobalt/arch/arm'
>   CC       libarch_la-features.lo
> features.c: In function 'cobalt_arch_check_features':
> features.c:82:1: error: r7 cannot be used in 'asm' here
>    82 | }
>       | ^
> 
> That seems to be related to passing the syscall number via r7 on ARM. Is
> our ABI soon no longer compilable, or can we fix this?
> 
> Jan
> 

Correction: The issue is likely older. I've just checked buster's gcc-8,
and it shows the very same behavior. Will update the commit messages if
we stick with my solutions.

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux