Signedness of char in BTF

Signedness of char in BTF

[Lorenz Bauer]

Hi Yonghong and Andrii,

I have some questions re: signedness of chars in BTF. According to [1] BTF_INT_ENCODING() may be one of SIGNED, CHAR or BOOL. If I read [2] correctly the signedness of char is implementation defined. Does this mean that I need to know which implementation generated the BTF to interpret CHAR correctly?

Somewhat related, how to I make clang emit BTF_INT_CHAR in the first place? I've tried with clang-14, but only ever get

    [6] INT 'unsigned char' size=1 bits_offset=0 nr_bits=8 encoding=(none)
    [6] INT 'char' size=1 bits_offset=0 nr_bits=8 encoding=SIGNED

The kernel seems to agree that CHAR isn't a thing [3].

Thanks!
Lorenz

1: https://www.kernel.org/doc/html/latest/bpf/btf.html#btf-kind-int
2: https://stackoverflow.com/a/2054941/19544965
3: https://sourcegraph.com/github.com/torvalds/linux@353f7988dd8413c47718f7ca79c030b6fb62cfe5/-/blob/kernel/bpf/btf.c?L2928-2934

Re: Signedness of char in BTF

[Jose E. Marchesi]

> Hi Yonghong and Andrii,
>
> I have some questions re: signedness of chars in BTF. According to [1]
> BTF_INT_ENCODING() may be one of SIGNED, CHAR or BOOL.

I have always assumed that the bits in `encoding' are non-exclusive
i.e. it is a bitmap, not an enumerated.

> If I read [2] correctly the signedness of char is implementation
> defined. Does this mean that I need to know which implementation
> generated the BTF to interpret CHAR correctly?
>
> Somewhat related, how to I make clang emit BTF_INT_CHAR in the first
> place? I've tried with clang-14, but only ever get
>
>     [6] INT 'unsigned char' size=1 bits_offset=0 nr_bits=8 encoding=(none)
>     [6] INT 'char' size=1 bits_offset=0 nr_bits=8 encoding=SIGNED

Hm, in GCC we currently generate:

[1] int 'unsigned char'(0x00000001U#B) size=0x00000001U#B offset=0x00UB#b bits=0x08UB#b CHAR
[2] int 'char'(0x00000001U#B) size=0x00000001U#B offset=0x00UB#b bits=0x08UB#b SIGNED CHAR

Which turns out is not correct?

We used a signed type for `char' because that was what the LLVM BPF
toolchain uses, but then we assumed we had to emit the CHAR bit as
well... wrong assumption apparently (I just tried with clang 15 and it
doesn't set the CHAR bits for neither `char' nor `unsigned char').

But then what is the CHAR bit for?

> The kernel seems to agree that CHAR isn't a thing [3].
>
> Thanks!
> Lorenz
>
> 1: https://www.kernel.org/doc/html/latest/bpf/btf.html#btf-kind-int
> 2: https://stackoverflow.com/a/2054941/19544965
> 3:
> https://sourcegraph.com/github.com/torvalds/linux@353f7988dd8413c47718f7ca79c030b6fb62cfe5/-/blob/kernel/bpf/btf.c?L2928-2934

Re: Signedness of char in BTF

[Yonghong Song]

On 7/21/22 7:31 AM, Lorenz Bauer wrote:
> Hi Yonghong and Andrii,
> 
> I have some questions re: signedness of chars in BTF. According to [1] BTF_INT_ENCODING() may be one of SIGNED, CHAR or BOOL. If I read [2] correctly the signedness of char is implementation defined. Does this mean that I need to know which implementation generated the BTF to interpret CHAR correctly?
> 
> Somewhat related, how to I make clang emit BTF_INT_CHAR in the first place? I've tried with clang-14, but only ever get
> 
>      [6] INT 'unsigned char' size=1 bits_offset=0 nr_bits=8 encoding=(none)
>      [6] INT 'char' size=1 bits_offset=0 nr_bits=8 encoding=SIGNED
> 
> The kernel seems to agree that CHAR isn't a thing [3].

clang does not generate BTF_INT_CHAR.

BTFTypeInt::BTFTypeInt(uint32_t Encoding, uint32_t SizeInBits,
                        uint32_t OffsetInBits, StringRef TypeName)
     : Name(TypeName) {
   // Translate IR int encoding to BTF int encoding.
   uint8_t BTFEncoding;
   switch (Encoding) {
   case dwarf::DW_ATE_boolean:
     BTFEncoding = BTF::INT_BOOL;
     break;
   case dwarf::DW_ATE_signed:
   case dwarf::DW_ATE_signed_char:
     BTFEncoding = BTF::INT_SIGNED;
     break;
   case dwarf::DW_ATE_unsigned:
   case dwarf::DW_ATE_unsigned_char:
     BTFEncoding = 0;  /* INT_UNSIGNED */
     break;
   default:
     llvm_unreachable("Unknown BTFTypeInt Encoding");
   }

pahole does not generate INT_CHAR type either.
in pahole:

static int32_t btf_encoder__add_base_type(struct btf_encoder *encoder, 
const struct base_type *bt, co
nst char *name)
{
         const struct btf_type *t;
         uint8_t encoding = 0;  /* unsigned */
         uint16_t byte_sz;
         int32_t id;

         if (bt->is_signed) {
                 encoding = BTF_INT_SIGNED;
         } else if (bt->is_bool) {
                 encoding = BTF_INT_BOOL;
         } else if (bt->float_type && encoder->gen_floats) {
                 /* for floats */
         }
         ...
}

So for both clang and pahole, CHAR goes to INT_SIGNED or INT_UNSIGNED.

The reason is originally BTF tries to mimic CTF but a
simplified version, and CTF has CTF_TYPE_INT_CHAR, but later on
found BTF_INT_CHAR is not that useful so llvm and pahole
doesn't generate it any more.

The libbpf and kernel still supports BTF_INT_CHAR and when it is used
to print out values it is interpreted as type 'char'.

> 
> Thanks!
> Lorenz
> 
> 1: https://www.kernel.org/doc/html/latest/bpf/btf.html#btf-kind-int
> 2: https://stackoverflow.com/a/2054941/19544965
> 3: https://sourcegraph.com/github.com/torvalds/linux@353f7988dd8413c47718f7ca79c030b6fb62cfe5/-/blob/kernel/bpf/btf.c?L2928-2934

Re: Signedness of char in BTF

[Yonghong Song]

On 7/21/22 7:54 AM, Jose E. Marchesi wrote:
> 
>> Hi Yonghong and Andrii,
>>
>> I have some questions re: signedness of chars in BTF. According to [1]
>> BTF_INT_ENCODING() may be one of SIGNED, CHAR or BOOL.
> 
> I have always assumed that the bits in `encoding' are non-exclusive
> i.e. it is a bitmap, not an enumerated.

Based on current BTF design, it is enumerated. So signed char
is 'signed 1-byte int', unsigned char is 'unsigned 1-byte int'
and 'char' could be BTF_INT_CHAR but since in debuginfo
any 'char' has a signedness bit, so it is folded into
'signed 1-byte int' or 'unsigned 1-byte int'.

> 
>> If I read [2] correctly the signedness of char is implementation
>> defined. Does this mean that I need to know which implementation
>> generated the BTF to interpret CHAR correctly?
>>
>> Somewhat related, how to I make clang emit BTF_INT_CHAR in the first
>> place? I've tried with clang-14, but only ever get
>>
>>      [6] INT 'unsigned char' size=1 bits_offset=0 nr_bits=8 encoding=(none)
>>      [6] INT 'char' size=1 bits_offset=0 nr_bits=8 encoding=SIGNED
> 
> Hm, in GCC we currently generate:
> 
> [1] int 'unsigned char'(0x00000001U#B) size=0x00000001U#B offset=0x00UB#b bits=0x08UB#b CHAR
> [2] int 'char'(0x00000001U#B) size=0x00000001U#B offset=0x00UB#b bits=0x08UB#b SIGNED CHAR
> 
> Which turns out is not correct?
> 
> We used a signed type for `char' because that was what the LLVM BPF
> toolchain uses, but then we assumed we had to emit the CHAR bit as
> well... wrong assumption apparently (I just tried with clang 15 and it
> doesn't set the CHAR bits for neither `char' nor `unsigned char').
> 
> But then what is the CHAR bit for?

This is not generated by llvm or pahole but apparently it may still
have some meaning when printing the value, a 'char c' may have
a dump like 'c' instead of '0x63'. In kernel/bpf/btf.c, we have

                 /*
                  * BTF_INT_CHAR encoding never seems to be set for
                  * char arrays, so if size is 1 and element is
                  * printable as a char, we'll do that.
                  */
                 if (elem_size == 1)
                         encoding = BTF_INT_CHAR;

> 
>> The kernel seems to agree that CHAR isn't a thing [3].
>>
>> Thanks!
>> Lorenz
>>
>> 1: https://www.kernel.org/doc/html/latest/bpf/btf.html#btf-kind-int
>> 2: https://stackoverflow.com/a/2054941/19544965
>> 3:
>> https://sourcegraph.com/github.com/torvalds/linux@353f7988dd8413c47718f7ca79c030b6fb62cfe5/-/blob/kernel/bpf/btf.c?L2928-2934

Re: Signedness of char in BTF

[Jose E. Marchesi]

Hi Yonghong.

> On 7/21/22 7:54 AM, Jose E. Marchesi wrote:
>> 
>>> Hi Yonghong and Andrii,
>>>
>>> I have some questions re: signedness of chars in BTF. According to [1]
>>> BTF_INT_ENCODING() may be one of SIGNED, CHAR or BOOL.
>> I have always assumed that the bits in `encoding' are non-exclusive
>> i.e. it is a bitmap, not an enumerated.
>
> Based on current BTF design, it is enumerated. So signed char
> is 'signed 1-byte int', unsigned char is 'unsigned 1-byte int'
> and 'char' could be BTF_INT_CHAR but since in debuginfo
> any 'char' has a signedness bit, so it is folded into
> 'signed 1-byte int' or 'unsigned 1-byte int'.

Ok, we will change GCC so it does the same thing.

What about BOOL?  I don't think we ever use that bit.  Does LLVM
generate it for any case?

>>> If I read [2] correctly the signedness of char is implementation
>>> defined. Does this mean that I need to know which implementation
>>> generated the BTF to interpret CHAR correctly?
>>>
>>> Somewhat related, how to I make clang emit BTF_INT_CHAR in the first
>>> place? I've tried with clang-14, but only ever get
>>>
>>>      [6] INT 'unsigned char' size=1 bits_offset=0 nr_bits=8 encoding=(none)
>>>      [6] INT 'char' size=1 bits_offset=0 nr_bits=8 encoding=SIGNED
>> Hm, in GCC we currently generate:
>> [1] int 'unsigned char'(0x00000001U#B) size=0x00000001U#B
>> offset=0x00UB#b bits=0x08UB#b CHAR
>> [2] int 'char'(0x00000001U#B) size=0x00000001U#B offset=0x00UB#b bits=0x08UB#b SIGNED CHAR
>> Which turns out is not correct?
>> We used a signed type for `char' because that was what the LLVM BPF
>> toolchain uses, but then we assumed we had to emit the CHAR bit as
>> well... wrong assumption apparently (I just tried with clang 15 and it
>> doesn't set the CHAR bits for neither `char' nor `unsigned char').
>> But then what is the CHAR bit for?
>
> This is not generated by llvm or pahole but apparently it may still
> have some meaning when printing the value, a 'char c' may have
> a dump like 'c' instead of '0x63'. In kernel/bpf/btf.c, we have
>
>                 /*
>                  * BTF_INT_CHAR encoding never seems to be set for
>                  * char arrays, so if size is 1 and element is
>                  * printable as a char, we'll do that.
>                  */
>                 if (elem_size == 1)
>                         encoding = BTF_INT_CHAR;
>
>> 
>>> The kernel seems to agree that CHAR isn't a thing [3].
>>>
>>> Thanks!
>>> Lorenz
>>>
>>> 1: https://www.kernel.org/doc/html/latest/bpf/btf.html#btf-kind-int
>>> 2: https://stackoverflow.com/a/2054941/19544965
>>> 3:
>>> https://sourcegraph.com/github.com/torvalds/linux@353f7988dd8413c47718f7ca79c030b6fb62cfe5/-/blob/kernel/bpf/btf.c?L2928-2934

Re: Signedness of char in BTF

[Yonghong Song]

On 7/21/22 3:21 PM, Jose E. Marchesi wrote:
> 
> Hi Yonghong.
> 
>> On 7/21/22 7:54 AM, Jose E. Marchesi wrote:
>>>
>>>> Hi Yonghong and Andrii,
>>>>
>>>> I have some questions re: signedness of chars in BTF. According to [1]
>>>> BTF_INT_ENCODING() may be one of SIGNED, CHAR or BOOL.
>>> I have always assumed that the bits in `encoding' are non-exclusive
>>> i.e. it is a bitmap, not an enumerated.
>>
>> Based on current BTF design, it is enumerated. So signed char
>> is 'signed 1-byte int', unsigned char is 'unsigned 1-byte int'
>> and 'char' could be BTF_INT_CHAR but since in debuginfo
>> any 'char' has a signedness bit, so it is folded into
>> 'signed 1-byte int' or 'unsigned 1-byte int'.
> 
> Ok, we will change GCC so it does the same thing.
> 
> What about BOOL?  I don't think we ever use that bit.  Does LLVM
> generate it for any case?

The llvm and pahole generate BTF_INT_BOOL when the dwarf type has
attribute DW_ATE_boolean.
But BTF_INT_BOOL is actually used in libbpf to differentiate
configuration values (CONFIG_* = 'y' vs. CONFIG_* = <value>)

In llvm,
   uint8_t BTFEncoding;
   switch (Encoding) {
   case dwarf::DW_ATE_boolean:
     BTFEncoding = BTF::INT_BOOL;
     break;
   case dwarf::DW_ATE_signed:
   case dwarf::DW_ATE_signed_char:
     BTFEncoding = BTF::INT_SIGNED;
     break;
   case dwarf::DW_ATE_unsigned:
   case dwarf::DW_ATE_unsigned_char:
     BTFEncoding = 0;
     break;
   default:
     llvm_unreachable("Unknown BTFTypeInt Encoding");
   }

For a concrete example,

[$ ~/tmp1] cat t.c
int test(_Bool g) {
    return g;
}
[$ ~/tmp1] clang -target bpf -O2 -g -c t.c
[$ ~/tmp1] bpftool btf dump file t.o
[1] INT '_Bool' size=1 bits_offset=0 nr_bits=8 encoding=BOOL
[2] FUNC_PROTO '(anon)' ret_type_id=3 vlen=1
         'g' type_id=1
[3] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED
[4] FUNC 'test' type_id=2 linkage=global
[$ ~/tmp1]

> 
>>>> If I read [2] correctly the signedness of char is implementation
>>>> defined. Does this mean that I need to know which implementation
>>>> generated the BTF to interpret CHAR correctly?
>>>>
>>>> Somewhat related, how to I make clang emit BTF_INT_CHAR in the first
>>>> place? I've tried with clang-14, but only ever get
>>>>
>>>>       [6] INT 'unsigned char' size=1 bits_offset=0 nr_bits=8 encoding=(none)
>>>>       [6] INT 'char' size=1 bits_offset=0 nr_bits=8 encoding=SIGNED
>>> Hm, in GCC we currently generate:
>>> [1] int 'unsigned char'(0x00000001U#B) size=0x00000001U#B
>>> offset=0x00UB#b bits=0x08UB#b CHAR
>>> [2] int 'char'(0x00000001U#B) size=0x00000001U#B offset=0x00UB#b bits=0x08UB#b SIGNED CHAR
>>> Which turns out is not correct?
>>> We used a signed type for `char' because that was what the LLVM BPF
>>> toolchain uses, but then we assumed we had to emit the CHAR bit as
>>> well... wrong assumption apparently (I just tried with clang 15 and it
>>> doesn't set the CHAR bits for neither `char' nor `unsigned char').
>>> But then what is the CHAR bit for?
>>
>> This is not generated by llvm or pahole but apparently it may still
>> have some meaning when printing the value, a 'char c' may have
>> a dump like 'c' instead of '0x63'. In kernel/bpf/btf.c, we have
>>
>>                  /*
>>                   * BTF_INT_CHAR encoding never seems to be set for
>>                   * char arrays, so if size is 1 and element is
>>                   * printable as a char, we'll do that.
>>                   */
>>                  if (elem_size == 1)
>>                          encoding = BTF_INT_CHAR;
>>
>>>
>>>> The kernel seems to agree that CHAR isn't a thing [3].
>>>>
>>>> Thanks!
>>>> Lorenz
>>>>
>>>> 1: https://www.kernel.org/doc/html/latest/bpf/btf.html#btf-kind-int
>>>> 2: https://stackoverflow.com/a/2054941/19544965
>>>> 3:
>>>> https://sourcegraph.com/github.com/torvalds/linux@353f7988dd8413c47718f7ca79c030b6fb62cfe5/-/blob/kernel/bpf/btf.c?L2928-2934

Re: Signedness of char in BTF

[Jose E. Marchesi]

> The llvm and pahole generate BTF_INT_BOOL when the dwarf type has
> attribute DW_ATE_boolean.
> But BTF_INT_BOOL is actually used in libbpf to differentiate
> configuration values (CONFIG_* = 'y' vs. CONFIG_* = <value>)
>
> In llvm,
>   uint8_t BTFEncoding;
>   switch (Encoding) {
>   case dwarf::DW_ATE_boolean:
>     BTFEncoding = BTF::INT_BOOL;
>     break;
>   case dwarf::DW_ATE_signed:
>   case dwarf::DW_ATE_signed_char:
>     BTFEncoding = BTF::INT_SIGNED;
>     break;
>   case dwarf::DW_ATE_unsigned:
>   case dwarf::DW_ATE_unsigned_char:
>     BTFEncoding = 0;
>     break;
>   default:
>     llvm_unreachable("Unknown BTFTypeInt Encoding");
>   }

I just sent a patch to make GCC behave the same way:
https://gcc.gnu.org/pipermail/gcc-patches/2022-July/598702.html

Re: Signedness of char in BTF

[Yonghong Song]

On 7/22/22 4:25 AM, Jose E. Marchesi wrote:
> 
>> The llvm and pahole generate BTF_INT_BOOL when the dwarf type has
>> attribute DW_ATE_boolean.
>> But BTF_INT_BOOL is actually used in libbpf to differentiate
>> configuration values (CONFIG_* = 'y' vs. CONFIG_* = <value>)
>>
>> In llvm,
>>    uint8_t BTFEncoding;
>>    switch (Encoding) {
>>    case dwarf::DW_ATE_boolean:
>>      BTFEncoding = BTF::INT_BOOL;
>>      break;
>>    case dwarf::DW_ATE_signed:
>>    case dwarf::DW_ATE_signed_char:
>>      BTFEncoding = BTF::INT_SIGNED;
>>      break;
>>    case dwarf::DW_ATE_unsigned:
>>    case dwarf::DW_ATE_unsigned_char:
>>      BTFEncoding = 0;
>>      break;
>>    default:
>>      llvm_unreachable("Unknown BTFTypeInt Encoding");
>>    }
> 
> I just sent a patch to make GCC behave the same way:
> https://gcc.gnu.org/pipermail/gcc-patches/2022-July/598702.html

Sounds good. Thanks!

Re: Signedness of char in BTF

[Jose E. Marchesi]

>> The llvm and pahole generate BTF_INT_BOOL when the dwarf type has
>> attribute DW_ATE_boolean.
>> But BTF_INT_BOOL is actually used in libbpf to differentiate
>> configuration values (CONFIG_* = 'y' vs. CONFIG_* = <value>)
>>
>> In llvm,
>>   uint8_t BTFEncoding;
>>   switch (Encoding) {
>>   case dwarf::DW_ATE_boolean:
>>     BTFEncoding = BTF::INT_BOOL;
>>     break;
>>   case dwarf::DW_ATE_signed:
>>   case dwarf::DW_ATE_signed_char:
>>     BTFEncoding = BTF::INT_SIGNED;
>>     break;
>>   case dwarf::DW_ATE_unsigned:
>>   case dwarf::DW_ATE_unsigned_char:
>>     BTFEncoding = 0;
>>     break;
>>   default:
>>     llvm_unreachable("Unknown BTFTypeInt Encoding");
>>   }
>
> I just sent a patch to make GCC behave the same way:
> https://gcc.gnu.org/pipermail/gcc-patches/2022-July/598702.html

This patch is now applied in GCC master.