fortify_source and syscall wrapping

fortify_source and syscall wrapping

[Jan Leupold]

Hi all,

I just had a long debugging session and wanted to comment the result here
on the list.

When using -D_FORTIFY_SOURCE=2 the first ioctl() after open() suddenly
failed. This is due to an inline function "open()" (in bits/fcntl2.h),
that will analyse the call arguments and forward the function call
accordingly. When the second argument is not a compile-time constant,
then the call is forwarded to "__open_2()" instead of "__open_alias()".

So when you write:

open("/dev/rtdm/rtser0", O_RDWR);

then __cobalt_open() will be called. But when you use a function like
this:

int my_function(int mode) {
    return open("/dev/rtdm/rtser0", mode);
}

then __open_2() will be called. The consequence is that the following
ioctl(RTSER_RTIOC_SET_CONFIG) fails, as the file descriptor was not
opened by __cobalt_open().

This is a nasty little difference. My solution was to write:

int my_function(int mode) {
    return __WRAP(open("/dev/rtdm/rtser0", mode));
}

Maybe this is just ancient history for you, but I had some "fun" to find
this.

Regards,
Jan


Mit freundlichen Grüßen

*Jan Leupold*
Entwicklung
Development
-- 
---------------------------------------------------------------------------
R-S-I Logo <https://www.rsi-elektrotechnik.de/>

R-S-I Elektrotechnik GmbH & Co. KG
Woelkestraße 11
D-85301 Schweitenkirchen
www.rsi-elektrotechnik.de <https://www.rsi-elektrotechnik.de/>

Tel: +49 8444 9204-32
Fax: +49 8444 9204-50
leupold@rsi-elektrotechnik.de <mailto:leupold@rsi-elektrotechnik.de>
---------------------------------------------------------------------------

Amtsgericht Ingolstadt - GmbH: HRB 191328 - KG: HRA 170363
Geschäftsführer: Dr.-Ing. Michael Sorg, Dipl.-Ing. Franz Sorg
USt-IdNr.: DE 128592548

Re: fortify_source and syscall wrapping

[Jan Kiszka]

On 10.09.20 12:37, Jan Leupold via Xenomai wrote:
> Hi all,
> 
> I just had a long debugging session and wanted to comment the result here
> on the list.
> 
> When using -D_FORTIFY_SOURCE=2 the first ioctl() after open() suddenly
> failed. This is due to an inline function "open()" (in bits/fcntl2.h),
> that will analyse the call arguments and forward the function call
> accordingly. When the second argument is not a compile-time constant,
> then the call is forwarded to "__open_2()" instead of "__open_alias()".
> 
> So when you write:
> 
> open("/dev/rtdm/rtser0", O_RDWR);
> 
> then __cobalt_open() will be called. But when you use a function like
> this:
> 
> int my_function(int mode) {
>     return open("/dev/rtdm/rtser0", mode);
> }
> 
> then __open_2() will be called. The consequence is that the following
> ioctl(RTSER_RTIOC_SET_CONFIG) fails, as the file descriptor was not
> opened by __cobalt_open().
> 
> This is a nasty little difference. My solution was to write:
> 
> int my_function(int mode) {
>     return __WRAP(open("/dev/rtdm/rtser0", mode));
> }
> 
> Maybe this is just ancient history for you, but I had some "fun" to find
> this.
> 

Not at all - thanks for reporting! We already handle the open vs. open64
case. I do not see a reason that speaks against including this case as
well, i.e. adding more wrappers. Patches welcome.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux