* [PATCH] fix libbpf hashmap with size_t shorter than long long @ 2020-06-21 14:25 Jakub Bogusz 2020-06-23 5:44 ` Andrii Nakryiko 0 siblings, 1 reply; 6+ messages in thread From: Jakub Bogusz @ 2020-06-21 14:25 UTC (permalink / raw) To: bpf; +Cc: Alexei Starovoitov, Daniel Borkmann [-- Attachment #1: Type: text/plain, Size: 296 bytes --] Hello, I noticed that _bpftool crashes when building kernel tools (5.7.x) for 32-bit targets because in libbpf hashmap implementation hash_bits() function returning numbers exceeding hashmap buckets capacity. Attached patch fixes this problem. Regards, -- Jakub Bogusz http://qboosh.pl/ [-- Attachment #2: kernel-tools-bpf-hashmap.patch --] [-- Type: text/plain, Size: 914 bytes --] Ensure that hash_bits returns value fits in given bits (for bits > 0): multiplier is long long (which is the same or wider than size_t), so shift bits must be based on long long size, not __WORDSIZE. Signed-off-by: Jakub Bogusz <qboosh@pld-linux.org> --- linux-5.7/tools/lib/bpf/hashmap.h.orig 2020-06-01 01:49:15.000000000 +0200 +++ linux-5.7/tools/lib/bpf/hashmap.h 2020-06-21 15:22:07.298466419 +0200 @@ -10,17 +10,12 @@ #include <stdbool.h> #include <stddef.h> -#ifdef __GLIBC__ -#include <bits/wordsize.h> -#else -#include <bits/reg.h> -#endif #include "libbpf_internal.h" static inline size_t hash_bits(size_t h, int bits) { /* shuffle bits and return requested number of upper bits */ - return (h * 11400714819323198485llu) >> (__WORDSIZE - bits); + return (h * 11400714819323198485llu) >> (__SIZEOF_LONG_LONG__ * 8 - bits); } typedef size_t (*hashmap_hash_fn)(const void *key, void *ctx); ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] fix libbpf hashmap with size_t shorter than long long 2020-06-21 14:25 [PATCH] fix libbpf hashmap with size_t shorter than long long Jakub Bogusz @ 2020-06-23 5:44 ` Andrii Nakryiko 2020-06-23 19:29 ` Jakub Bogusz 0 siblings, 1 reply; 6+ messages in thread From: Andrii Nakryiko @ 2020-06-23 5:44 UTC (permalink / raw) To: Jakub Bogusz; +Cc: bpf, Alexei Starovoitov, Daniel Borkmann On Sun, Jun 21, 2020 at 7:34 AM Jakub Bogusz <qboosh@pld-linux.org> wrote: > > Hello, > > I noticed that _bpftool crashes when building kernel tools (5.7.x) for > 32-bit targets because in libbpf hashmap implementation hash_bits() > function returning numbers exceeding hashmap buckets capacity. > > Attached patch fixes this problem. > Thanks! But this was already fixed by Arnaldo Carvalho de Melo <acme@kernel.org> in 8ca8d4a84173 ("libbpf: Define __WORDSIZE if not available"). > > Regards, > > -- > Jakub Bogusz http://qboosh.pl/ ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] fix libbpf hashmap with size_t shorter than long long 2020-06-23 5:44 ` Andrii Nakryiko @ 2020-06-23 19:29 ` Jakub Bogusz 2020-06-23 19:40 ` Andrii Nakryiko 0 siblings, 1 reply; 6+ messages in thread From: Jakub Bogusz @ 2020-06-23 19:29 UTC (permalink / raw) To: Andrii Nakryiko; +Cc: bpf, Alexei Starovoitov, Daniel Borkmann On Mon, Jun 22, 2020 at 10:44:56PM -0700, Andrii Nakryiko wrote: > On Sun, Jun 21, 2020 at 7:34 AM Jakub Bogusz <qboosh@pld-linux.org> wrote: > > > > Hello, > > > > I noticed that _bpftool crashes when building kernel tools (5.7.x) for > > 32-bit targets because in libbpf hashmap implementation hash_bits() > > function returning numbers exceeding hashmap buckets capacity. > > > > Attached patch fixes this problem. > > > > Thanks! But this was already fixed by Arnaldo Carvalho de Melo <acme@kernel.org> > in 8ca8d4a84173 ("libbpf: Define __WORDSIZE if not available"). No, it's not: This change worked around __WORDSIZE not always being available. But the issue on (I)LP32 platforms is that 64-bit value is shifted by (32-bits) instead of (64-bits). (__SIZEOF_LONG__ * 8) is 32 on such architectures (i686, arm). I used __SIZEOF_LONG_LONG__ to get proper bit shift both on (I)LP32 and LP64 architectures. Should I provide an updated patch to apply on top of acme change? Regards, -- Jakub Bogusz http://qboosh.pl/ ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] fix libbpf hashmap with size_t shorter than long long 2020-06-23 19:29 ` Jakub Bogusz @ 2020-06-23 19:40 ` Andrii Nakryiko 2020-06-27 9:07 ` Jakub Bogusz 0 siblings, 1 reply; 6+ messages in thread From: Andrii Nakryiko @ 2020-06-23 19:40 UTC (permalink / raw) To: Jakub Bogusz; +Cc: bpf, Alexei Starovoitov, Daniel Borkmann On Tue, Jun 23, 2020 at 12:29 PM Jakub Bogusz <qboosh@pld-linux.org> wrote: > > On Mon, Jun 22, 2020 at 10:44:56PM -0700, Andrii Nakryiko wrote: > > On Sun, Jun 21, 2020 at 7:34 AM Jakub Bogusz <qboosh@pld-linux.org> wrote: > > > > > > Hello, > > > > > > I noticed that _bpftool crashes when building kernel tools (5.7.x) for > > > 32-bit targets because in libbpf hashmap implementation hash_bits() > > > function returning numbers exceeding hashmap buckets capacity. > > > > > > Attached patch fixes this problem. > > > > > > > Thanks! But this was already fixed by Arnaldo Carvalho de Melo <acme@kernel.org> > > in 8ca8d4a84173 ("libbpf: Define __WORDSIZE if not available"). > > No, it's not: > This change worked around __WORDSIZE not always being available. > > But the issue on (I)LP32 platforms is that 64-bit value is shifted by > (32-bits) instead of (64-bits). > > (__SIZEOF_LONG__ * 8) is 32 on such architectures (i686, arm). > I used __SIZEOF_LONG_LONG__ to get proper bit shift both on (I)LP32 and > LP64 architectures. > Ah, I see. I actually mentioned __SIZEOF_ constants on the original fix patch. But I think in this case it has to use __SIZEOF_SIZE_T, which on 32-bit should be 4, right? > Should I provide an updated patch to apply on top of acme change? Yes, that would be good. But I think there is no need to penalize 32-bit arches with use of 64-bit long longs, and instead it's better to use #ifdef for 32-bit case vs 64-bit case. The multiplication constant will change, of course, should be 2654435769. I'd appreciate it if you can do the patch, thanks! > > > Regards, > > -- > Jakub Bogusz http://qboosh.pl/ ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] fix libbpf hashmap with size_t shorter than long long 2020-06-23 19:40 ` Andrii Nakryiko @ 2020-06-27 9:07 ` Jakub Bogusz 2020-06-27 20:25 ` Andrii Nakryiko 0 siblings, 1 reply; 6+ messages in thread From: Jakub Bogusz @ 2020-06-27 9:07 UTC (permalink / raw) To: Andrii Nakryiko; +Cc: bpf, Alexei Starovoitov, Daniel Borkmann [-- Attachment #1: Type: text/plain, Size: 2163 bytes --] On Tue, Jun 23, 2020 at 12:40:02PM -0700, Andrii Nakryiko wrote: > On Tue, Jun 23, 2020 at 12:29 PM Jakub Bogusz <qboosh@pld-linux.org> wrote: > > > > On Mon, Jun 22, 2020 at 10:44:56PM -0700, Andrii Nakryiko wrote: > > > On Sun, Jun 21, 2020 at 7:34 AM Jakub Bogusz <qboosh@pld-linux.org> wrote: > > > > > > > > Hello, > > > > > > > > I noticed that _bpftool crashes when building kernel tools (5.7.x) for > > > > 32-bit targets because in libbpf hashmap implementation hash_bits() > > > > function returning numbers exceeding hashmap buckets capacity. > > > > > > > > Attached patch fixes this problem. > > > > > > > > > > Thanks! But this was already fixed by Arnaldo Carvalho de Melo <acme@kernel.org> > > > in 8ca8d4a84173 ("libbpf: Define __WORDSIZE if not available"). > > > > No, it's not: > > This change worked around __WORDSIZE not always being available. > > > > But the issue on (I)LP32 platforms is that 64-bit value is shifted by > > (32-bits) instead of (64-bits). > > > > (__SIZEOF_LONG__ * 8) is 32 on such architectures (i686, arm). > > I used __SIZEOF_LONG_LONG__ to get proper bit shift both on (I)LP32 and > > LP64 architectures. > > > > Ah, I see. I actually mentioned __SIZEOF_ constants on the original > fix patch. But I think in this case it has to use __SIZEOF_SIZE_T, > which on 32-bit should be 4, right? After changing constant to 32-bit, yes (to be precise, it should use maximum of __SIZEOF_SIZE_T__ and __SIZEOF_LONG__ if constant is specified with UL suffix; there is no constant suffix available for size_t). > > Should I provide an updated patch to apply on top of acme change? > > Yes, that would be good. But I think there is no need to penalize > 32-bit arches with use of 64-bit long longs, and instead it's better > to use #ifdef for 32-bit case vs 64-bit case. The multiplication > constant will change, of course, should be 2654435769. I'd appreciate > it if you can do the patch, thanks! OK, so now the patch provides two variants: - "long long" case for LP64 architectures - "long" case for (I)LP32 architectures (selected basing of __SIZEOF_ constants) matter) Regards, -- Jakub Bogusz http://qboosh.pl/ [-- Attachment #2: kernel-tools-bpf-hashmap.patch --] [-- Type: text/plain, Size: 1208 bytes --] Fix libbpf hashmap on (I)LP32 architectures On ILP32, 64-bit result was shifted by value calculated for 32-bit long type and returned value was much outside hashmap capacity. As advised by Andrii Nakryiko, this patch uses different hashing variant for architectures with size_t shorter than long long. Signed-off-by: Jakub Bogusz <qboosh@pld-linux.org> --- linux/tools/lib/bpf/hashmap.h.orig 2020-06-01 01:49:15.000000000 +0200 +++ linux/tools/lib/bpf/hashmap.h 2020-06-21 15:22:07.298466419 +0200 @@ -11,14 +11,18 @@ #include <stdbool.h> #include <stddef.h> #include <limits.h> -#ifndef __WORDSIZE -#define __WORDSIZE (__SIZEOF_LONG__ * 8) -#endif static inline size_t hash_bits(size_t h, int bits) { /* shuffle bits and return requested number of upper bits */ - return (h * 11400714819323198485llu) >> (__WORDSIZE - bits); +#if (__SIZEOF_SIZE_T__ == __SIZEOF_LONG_LONG__) + /* LP64 case */ + return (h * 11400714819323198485llu) >> (__SIZEOF_LONG_LONG__ * 8 - bits); +#elif (__SIZEOF_SIZE_T__ <= __SIZEOF_LONG__) + return (h * 2654435769lu) >> (__SIZEOF_LONG__ * 8 - bits); +#else +# error "Unsupported size_t size" +#endif } typedef size_t (*hashmap_hash_fn)(const void *key, void *ctx); ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] fix libbpf hashmap with size_t shorter than long long 2020-06-27 9:07 ` Jakub Bogusz @ 2020-06-27 20:25 ` Andrii Nakryiko 0 siblings, 0 replies; 6+ messages in thread From: Andrii Nakryiko @ 2020-06-27 20:25 UTC (permalink / raw) To: Jakub Bogusz; +Cc: bpf, Alexei Starovoitov, Daniel Borkmann On Sat, Jun 27, 2020 at 2:07 AM Jakub Bogusz <qboosh@pld-linux.org> wrote: > > On Tue, Jun 23, 2020 at 12:40:02PM -0700, Andrii Nakryiko wrote: > > On Tue, Jun 23, 2020 at 12:29 PM Jakub Bogusz <qboosh@pld-linux.org> wrote: > > > > > > On Mon, Jun 22, 2020 at 10:44:56PM -0700, Andrii Nakryiko wrote: > > > > On Sun, Jun 21, 2020 at 7:34 AM Jakub Bogusz <qboosh@pld-linux.org> wrote: > > > > > > > > > > Hello, > > > > > > > > > > I noticed that _bpftool crashes when building kernel tools (5.7.x) for > > > > > 32-bit targets because in libbpf hashmap implementation hash_bits() > > > > > function returning numbers exceeding hashmap buckets capacity. > > > > > > > > > > Attached patch fixes this problem. > > > > > > > > > > > > > Thanks! But this was already fixed by Arnaldo Carvalho de Melo <acme@kernel.org> > > > > in 8ca8d4a84173 ("libbpf: Define __WORDSIZE if not available"). > > > > > > No, it's not: > > > This change worked around __WORDSIZE not always being available. > > > > > > But the issue on (I)LP32 platforms is that 64-bit value is shifted by > > > (32-bits) instead of (64-bits). > > > > > > (__SIZEOF_LONG__ * 8) is 32 on such architectures (i686, arm). > > > I used __SIZEOF_LONG_LONG__ to get proper bit shift both on (I)LP32 and > > > LP64 architectures. > > > > > > > Ah, I see. I actually mentioned __SIZEOF_ constants on the original > > fix patch. But I think in this case it has to use __SIZEOF_SIZE_T, > > which on 32-bit should be 4, right? > > After changing constant to 32-bit, yes (to be precise, it should use maximum > of __SIZEOF_SIZE_T__ and __SIZEOF_LONG__ if constant is specified with > UL suffix; there is no constant suffix available for size_t). > > > > Should I provide an updated patch to apply on top of acme change? > > > > Yes, that would be good. But I think there is no need to penalize > > 32-bit arches with use of 64-bit long longs, and instead it's better > > to use #ifdef for 32-bit case vs 64-bit case. The multiplication > > constant will change, of course, should be 2654435769. I'd appreciate > > it if you can do the patch, thanks! > > OK, so now the patch provides two variants: > - "long long" case for LP64 architectures > - "long" case for (I)LP32 architectures > (selected basing of __SIZEOF_ constants) > matter) > > Change looks good, thanks! But it would be more convenient for everyone if you submitted it not as an attachment, but as a proper patch email message. Acked-by: Andrii Nakryiko <andriin@fb.com> > Regards, > > -- > Jakub Bogusz http://qboosh.pl/ ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2020-06-27 20:25 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-06-21 14:25 [PATCH] fix libbpf hashmap with size_t shorter than long long Jakub Bogusz 2020-06-23 5:44 ` Andrii Nakryiko 2020-06-23 19:29 ` Jakub Bogusz 2020-06-23 19:40 ` Andrii Nakryiko 2020-06-27 9:07 ` Jakub Bogusz 2020-06-27 20:25 ` Andrii Nakryiko
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.