* [PATCH] fix libbpf hashmap with size_t shorter than long long
@ 2020-06-21 14:25 Jakub Bogusz
2020-06-23 5:44 ` Andrii Nakryiko
0 siblings, 1 reply; 6+ messages in thread
From: Jakub Bogusz @ 2020-06-21 14:25 UTC (permalink / raw)
To: bpf; +Cc: Alexei Starovoitov, Daniel Borkmann
[-- Attachment #1: Type: text/plain, Size: 296 bytes --]
Hello,
I noticed that _bpftool crashes when building kernel tools (5.7.x) for
32-bit targets because in libbpf hashmap implementation hash_bits()
function returning numbers exceeding hashmap buckets capacity.
Attached patch fixes this problem.
Regards,
--
Jakub Bogusz http://qboosh.pl/
[-- Attachment #2: kernel-tools-bpf-hashmap.patch --]
[-- Type: text/plain, Size: 914 bytes --]
Ensure that hash_bits returns value fits in given bits (for bits > 0):
multiplier is long long (which is the same or wider than size_t), so shift bits
must be based on long long size, not __WORDSIZE.
Signed-off-by: Jakub Bogusz <qboosh@pld-linux.org>
--- linux-5.7/tools/lib/bpf/hashmap.h.orig 2020-06-01 01:49:15.000000000 +0200
+++ linux-5.7/tools/lib/bpf/hashmap.h 2020-06-21 15:22:07.298466419 +0200
@@ -10,17 +10,12 @@
#include <stdbool.h>
#include <stddef.h>
-#ifdef __GLIBC__
-#include <bits/wordsize.h>
-#else
-#include <bits/reg.h>
-#endif
#include "libbpf_internal.h"
static inline size_t hash_bits(size_t h, int bits)
{
/* shuffle bits and return requested number of upper bits */
- return (h * 11400714819323198485llu) >> (__WORDSIZE - bits);
+ return (h * 11400714819323198485llu) >> (__SIZEOF_LONG_LONG__ * 8 - bits);
}
typedef size_t (*hashmap_hash_fn)(const void *key, void *ctx);
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] fix libbpf hashmap with size_t shorter than long long
2020-06-21 14:25 [PATCH] fix libbpf hashmap with size_t shorter than long long Jakub Bogusz
@ 2020-06-23 5:44 ` Andrii Nakryiko
2020-06-23 19:29 ` Jakub Bogusz
0 siblings, 1 reply; 6+ messages in thread
From: Andrii Nakryiko @ 2020-06-23 5:44 UTC (permalink / raw)
To: Jakub Bogusz; +Cc: bpf, Alexei Starovoitov, Daniel Borkmann
On Sun, Jun 21, 2020 at 7:34 AM Jakub Bogusz <qboosh@pld-linux.org> wrote:
>
> Hello,
>
> I noticed that _bpftool crashes when building kernel tools (5.7.x) for
> 32-bit targets because in libbpf hashmap implementation hash_bits()
> function returning numbers exceeding hashmap buckets capacity.
>
> Attached patch fixes this problem.
>
Thanks! But this was already fixed by Arnaldo Carvalho de Melo <acme@kernel.org>
in 8ca8d4a84173 ("libbpf: Define __WORDSIZE if not available").
>
> Regards,
>
> --
> Jakub Bogusz http://qboosh.pl/
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] fix libbpf hashmap with size_t shorter than long long
2020-06-23 5:44 ` Andrii Nakryiko
@ 2020-06-23 19:29 ` Jakub Bogusz
2020-06-23 19:40 ` Andrii Nakryiko
0 siblings, 1 reply; 6+ messages in thread
From: Jakub Bogusz @ 2020-06-23 19:29 UTC (permalink / raw)
To: Andrii Nakryiko; +Cc: bpf, Alexei Starovoitov, Daniel Borkmann
On Mon, Jun 22, 2020 at 10:44:56PM -0700, Andrii Nakryiko wrote:
> On Sun, Jun 21, 2020 at 7:34 AM Jakub Bogusz <qboosh@pld-linux.org> wrote:
> >
> > Hello,
> >
> > I noticed that _bpftool crashes when building kernel tools (5.7.x) for
> > 32-bit targets because in libbpf hashmap implementation hash_bits()
> > function returning numbers exceeding hashmap buckets capacity.
> >
> > Attached patch fixes this problem.
> >
>
> Thanks! But this was already fixed by Arnaldo Carvalho de Melo <acme@kernel.org>
> in 8ca8d4a84173 ("libbpf: Define __WORDSIZE if not available").
No, it's not:
This change worked around __WORDSIZE not always being available.
But the issue on (I)LP32 platforms is that 64-bit value is shifted by
(32-bits) instead of (64-bits).
(__SIZEOF_LONG__ * 8) is 32 on such architectures (i686, arm).
I used __SIZEOF_LONG_LONG__ to get proper bit shift both on (I)LP32 and
LP64 architectures.
Should I provide an updated patch to apply on top of acme change?
Regards,
--
Jakub Bogusz http://qboosh.pl/
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] fix libbpf hashmap with size_t shorter than long long
2020-06-23 19:29 ` Jakub Bogusz
@ 2020-06-23 19:40 ` Andrii Nakryiko
2020-06-27 9:07 ` Jakub Bogusz
0 siblings, 1 reply; 6+ messages in thread
From: Andrii Nakryiko @ 2020-06-23 19:40 UTC (permalink / raw)
To: Jakub Bogusz; +Cc: bpf, Alexei Starovoitov, Daniel Borkmann
On Tue, Jun 23, 2020 at 12:29 PM Jakub Bogusz <qboosh@pld-linux.org> wrote:
>
> On Mon, Jun 22, 2020 at 10:44:56PM -0700, Andrii Nakryiko wrote:
> > On Sun, Jun 21, 2020 at 7:34 AM Jakub Bogusz <qboosh@pld-linux.org> wrote:
> > >
> > > Hello,
> > >
> > > I noticed that _bpftool crashes when building kernel tools (5.7.x) for
> > > 32-bit targets because in libbpf hashmap implementation hash_bits()
> > > function returning numbers exceeding hashmap buckets capacity.
> > >
> > > Attached patch fixes this problem.
> > >
> >
> > Thanks! But this was already fixed by Arnaldo Carvalho de Melo <acme@kernel.org>
> > in 8ca8d4a84173 ("libbpf: Define __WORDSIZE if not available").
>
> No, it's not:
> This change worked around __WORDSIZE not always being available.
>
> But the issue on (I)LP32 platforms is that 64-bit value is shifted by
> (32-bits) instead of (64-bits).
>
> (__SIZEOF_LONG__ * 8) is 32 on such architectures (i686, arm).
> I used __SIZEOF_LONG_LONG__ to get proper bit shift both on (I)LP32 and
> LP64 architectures.
>
Ah, I see. I actually mentioned __SIZEOF_ constants on the original
fix patch. But I think in this case it has to use __SIZEOF_SIZE_T,
which on 32-bit should be 4, right?
> Should I provide an updated patch to apply on top of acme change?
Yes, that would be good. But I think there is no need to penalize
32-bit arches with use of 64-bit long longs, and instead it's better
to use #ifdef for 32-bit case vs 64-bit case. The multiplication
constant will change, of course, should be 2654435769. I'd appreciate
it if you can do the patch, thanks!
>
>
> Regards,
>
> --
> Jakub Bogusz http://qboosh.pl/
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] fix libbpf hashmap with size_t shorter than long long
2020-06-23 19:40 ` Andrii Nakryiko
@ 2020-06-27 9:07 ` Jakub Bogusz
2020-06-27 20:25 ` Andrii Nakryiko
0 siblings, 1 reply; 6+ messages in thread
From: Jakub Bogusz @ 2020-06-27 9:07 UTC (permalink / raw)
To: Andrii Nakryiko; +Cc: bpf, Alexei Starovoitov, Daniel Borkmann
[-- Attachment #1: Type: text/plain, Size: 2163 bytes --]
On Tue, Jun 23, 2020 at 12:40:02PM -0700, Andrii Nakryiko wrote:
> On Tue, Jun 23, 2020 at 12:29 PM Jakub Bogusz <qboosh@pld-linux.org> wrote:
> >
> > On Mon, Jun 22, 2020 at 10:44:56PM -0700, Andrii Nakryiko wrote:
> > > On Sun, Jun 21, 2020 at 7:34 AM Jakub Bogusz <qboosh@pld-linux.org> wrote:
> > > >
> > > > Hello,
> > > >
> > > > I noticed that _bpftool crashes when building kernel tools (5.7.x) for
> > > > 32-bit targets because in libbpf hashmap implementation hash_bits()
> > > > function returning numbers exceeding hashmap buckets capacity.
> > > >
> > > > Attached patch fixes this problem.
> > > >
> > >
> > > Thanks! But this was already fixed by Arnaldo Carvalho de Melo <acme@kernel.org>
> > > in 8ca8d4a84173 ("libbpf: Define __WORDSIZE if not available").
> >
> > No, it's not:
> > This change worked around __WORDSIZE not always being available.
> >
> > But the issue on (I)LP32 platforms is that 64-bit value is shifted by
> > (32-bits) instead of (64-bits).
> >
> > (__SIZEOF_LONG__ * 8) is 32 on such architectures (i686, arm).
> > I used __SIZEOF_LONG_LONG__ to get proper bit shift both on (I)LP32 and
> > LP64 architectures.
> >
>
> Ah, I see. I actually mentioned __SIZEOF_ constants on the original
> fix patch. But I think in this case it has to use __SIZEOF_SIZE_T,
> which on 32-bit should be 4, right?
After changing constant to 32-bit, yes (to be precise, it should use maximum
of __SIZEOF_SIZE_T__ and __SIZEOF_LONG__ if constant is specified with
UL suffix; there is no constant suffix available for size_t).
> > Should I provide an updated patch to apply on top of acme change?
>
> Yes, that would be good. But I think there is no need to penalize
> 32-bit arches with use of 64-bit long longs, and instead it's better
> to use #ifdef for 32-bit case vs 64-bit case. The multiplication
> constant will change, of course, should be 2654435769. I'd appreciate
> it if you can do the patch, thanks!
OK, so now the patch provides two variants:
- "long long" case for LP64 architectures
- "long" case for (I)LP32 architectures
(selected basing of __SIZEOF_ constants)
matter)
Regards,
--
Jakub Bogusz http://qboosh.pl/
[-- Attachment #2: kernel-tools-bpf-hashmap.patch --]
[-- Type: text/plain, Size: 1208 bytes --]
Fix libbpf hashmap on (I)LP32 architectures
On ILP32, 64-bit result was shifted by value calculated for 32-bit long type
and returned value was much outside hashmap capacity.
As advised by Andrii Nakryiko, this patch uses different hashing variant for
architectures with size_t shorter than long long.
Signed-off-by: Jakub Bogusz <qboosh@pld-linux.org>
--- linux/tools/lib/bpf/hashmap.h.orig 2020-06-01 01:49:15.000000000 +0200
+++ linux/tools/lib/bpf/hashmap.h 2020-06-21 15:22:07.298466419 +0200
@@ -11,14 +11,18 @@
#include <stdbool.h>
#include <stddef.h>
#include <limits.h>
-#ifndef __WORDSIZE
-#define __WORDSIZE (__SIZEOF_LONG__ * 8)
-#endif
static inline size_t hash_bits(size_t h, int bits)
{
/* shuffle bits and return requested number of upper bits */
- return (h * 11400714819323198485llu) >> (__WORDSIZE - bits);
+#if (__SIZEOF_SIZE_T__ == __SIZEOF_LONG_LONG__)
+ /* LP64 case */
+ return (h * 11400714819323198485llu) >> (__SIZEOF_LONG_LONG__ * 8 - bits);
+#elif (__SIZEOF_SIZE_T__ <= __SIZEOF_LONG__)
+ return (h * 2654435769lu) >> (__SIZEOF_LONG__ * 8 - bits);
+#else
+# error "Unsupported size_t size"
+#endif
}
typedef size_t (*hashmap_hash_fn)(const void *key, void *ctx);
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] fix libbpf hashmap with size_t shorter than long long
2020-06-27 9:07 ` Jakub Bogusz
@ 2020-06-27 20:25 ` Andrii Nakryiko
0 siblings, 0 replies; 6+ messages in thread
From: Andrii Nakryiko @ 2020-06-27 20:25 UTC (permalink / raw)
To: Jakub Bogusz; +Cc: bpf, Alexei Starovoitov, Daniel Borkmann
On Sat, Jun 27, 2020 at 2:07 AM Jakub Bogusz <qboosh@pld-linux.org> wrote:
>
> On Tue, Jun 23, 2020 at 12:40:02PM -0700, Andrii Nakryiko wrote:
> > On Tue, Jun 23, 2020 at 12:29 PM Jakub Bogusz <qboosh@pld-linux.org> wrote:
> > >
> > > On Mon, Jun 22, 2020 at 10:44:56PM -0700, Andrii Nakryiko wrote:
> > > > On Sun, Jun 21, 2020 at 7:34 AM Jakub Bogusz <qboosh@pld-linux.org> wrote:
> > > > >
> > > > > Hello,
> > > > >
> > > > > I noticed that _bpftool crashes when building kernel tools (5.7.x) for
> > > > > 32-bit targets because in libbpf hashmap implementation hash_bits()
> > > > > function returning numbers exceeding hashmap buckets capacity.
> > > > >
> > > > > Attached patch fixes this problem.
> > > > >
> > > >
> > > > Thanks! But this was already fixed by Arnaldo Carvalho de Melo <acme@kernel.org>
> > > > in 8ca8d4a84173 ("libbpf: Define __WORDSIZE if not available").
> > >
> > > No, it's not:
> > > This change worked around __WORDSIZE not always being available.
> > >
> > > But the issue on (I)LP32 platforms is that 64-bit value is shifted by
> > > (32-bits) instead of (64-bits).
> > >
> > > (__SIZEOF_LONG__ * 8) is 32 on such architectures (i686, arm).
> > > I used __SIZEOF_LONG_LONG__ to get proper bit shift both on (I)LP32 and
> > > LP64 architectures.
> > >
> >
> > Ah, I see. I actually mentioned __SIZEOF_ constants on the original
> > fix patch. But I think in this case it has to use __SIZEOF_SIZE_T,
> > which on 32-bit should be 4, right?
>
> After changing constant to 32-bit, yes (to be precise, it should use maximum
> of __SIZEOF_SIZE_T__ and __SIZEOF_LONG__ if constant is specified with
> UL suffix; there is no constant suffix available for size_t).
>
> > > Should I provide an updated patch to apply on top of acme change?
> >
> > Yes, that would be good. But I think there is no need to penalize
> > 32-bit arches with use of 64-bit long longs, and instead it's better
> > to use #ifdef for 32-bit case vs 64-bit case. The multiplication
> > constant will change, of course, should be 2654435769. I'd appreciate
> > it if you can do the patch, thanks!
>
> OK, so now the patch provides two variants:
> - "long long" case for LP64 architectures
> - "long" case for (I)LP32 architectures
> (selected basing of __SIZEOF_ constants)
> matter)
>
>
Change looks good, thanks! But it would be more convenient for
everyone if you submitted it not as an attachment, but as a proper
patch email message.
Acked-by: Andrii Nakryiko <andriin@fb.com>
> Regards,
>
> --
> Jakub Bogusz http://qboosh.pl/
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2020-06-27 20:25 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-06-21 14:25 [PATCH] fix libbpf hashmap with size_t shorter than long long Jakub Bogusz
2020-06-23 5:44 ` Andrii Nakryiko
2020-06-23 19:29 ` Jakub Bogusz
2020-06-23 19:40 ` Andrii Nakryiko
2020-06-27 9:07 ` Jakub Bogusz
2020-06-27 20:25 ` Andrii Nakryiko
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.