Re: [PATCH 1/2] lib: Add lz4 compressor module

Re: [PATCH 1/2] lib: Add lz4 compressor module

[Geert Uytterhoeven]

On Thu, Mar 14, 2013 at 10:48 AM, Chanho Min <chanho.min@lge.com> wrote:
> +#ifdef __BIG_ENDIAN
> +#define LZ4_NBCOMMONBYTES(val) (__builtin_clzll(val) >> 3)
> +#else
> +#define LZ4_NBCOMMONBYTES(val) (__builtin_ctzll(val) >> 3)
> +#endif
>
>  #else  /* 32-bit */
>  #define STEPSIZE 4
> @@ -83,6 +130,14 @@ typedef struct _U64_S { u64 v; } U64_S;
>         } while (0)
>
>  #define LZ4_SECURECOPY LZ4_WILDCOPY
> +#define HTYPE const u8*
> +
> +#ifdef __BIG_ENDIAN
> +#define LZ4_NBCOMMONBYTES(val) (__builtin_clz(val) >> 3)
> +#else
> +#define LZ4_NBCOMMONBYTES(val) (__builtin_ctz(val) >> 3)
> +#endif

It seems at least m68k and sparc don't have the __builtin_clz() functions:

m68k-allmodconfig (http://kisskb.ellerman.id.au/kisskb/buildresult/8572593/):

ERROR: "__clzsi2" [lib/lz4/lz4hc_compress.ko] undefined!
ERROR: "__clzsi2" [lib/lz4/lz4_compress.ko] undefined!

sparc64-allmodconfig (http://kisskb.ellerman.id.au/kisskb/buildresult/8572790/):

ERROR: "__clzdi2" [lib/lz4/lz4hc_compress.ko] undefined!
ERROR: "__clzdi2" [lib/lz4/lz4_compress.ko] undefined!

sparc-allmodconfig (http://kisskb.ellerman.id.au/kisskb/buildresult/8572795/):

ERROR: "__clzdi2" [lib/lz4/lz4hc_compress.ko] undefined!
ERROR: "__clzdi2" [lib/lz4/lz4_compress.ko] undefined!

There may be more, hidden by build errors before reaching the link or modpost
phases.

The gmail-whitespace-damaged patch below fixes the m68k and sparc builds
for me, does this look acceptable?

diff --git a/lib/lz4/lz4defs.h b/lib/lz4/lz4defs.h
index abcecdc..81d3251 100644
--- a/lib/lz4/lz4defs.h
+++ b/lib/lz4/lz4defs.h
@@ -107,12 +107,6 @@ typedef struct _U64_S { u64 v; } U64_S;
  } while (0)
 #define HTYPE u32

-#ifdef __BIG_ENDIAN
-#define LZ4_NBCOMMONBYTES(val) (__builtin_clzll(val) >> 3)
-#else
-#define LZ4_NBCOMMONBYTES(val) (__builtin_ctzll(val) >> 3)
-#endif
-
 #else /* 32-bit */
 #define STEPSIZE 4

@@ -132,12 +126,12 @@ typedef struct _U64_S { u64 v; } U64_S;
 #define LZ4_SECURECOPY LZ4_WILDCOPY
 #define HTYPE const u8*

-#ifdef __BIG_ENDIAN
-#define LZ4_NBCOMMONBYTES(val) (__builtin_clz(val) >> 3)
-#else
-#define LZ4_NBCOMMONBYTES(val) (__builtin_ctz(val) >> 3)
 #endif

+#ifdef __BIG_ENDIAN
+#define LZ4_NBCOMMONBYTES(val) (((STEPSIZE * 8) - 1 - __fls(val)) >> 3)
+#else
+#define LZ4_NBCOMMONBYTES(val) (__ffs(val) >> 3)
 #endif

 #define LZ4_READ_LITTLEENDIAN_16(d, s, p) \

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [PATCH 1/2] lib: Add lz4 compressor module

[Geert Uytterhoeven]

On Mon, Apr 22, 2013 at 11:22 AM, Chanho Min <chanho.min@lge.com> wrote:
>>> +#define HTYPE const u8*
>>> +
>>> +#ifdef __BIG_ENDIAN
>>> +#define LZ4_NBCOMMONBYTES(val) (__builtin_clz(val) >> 3)
>>> +#else
>>> +#define LZ4_NBCOMMONBYTES(val) (__builtin_ctz(val) >> 3)
>>> +#endif
>>
>>It seems at least m68k and sparc don't have the __builtin_clz() functions:
>>
>>m68k-allmodconfig (http://kisskb.ellerman.id.au/kisskb/buildresult/8572593/):
>>
>>ERROR: "__clzsi2" [lib/lz4/lz4hc_compress.ko] undefined!
>>ERROR: "__clzsi2" [lib/lz4/lz4_compress.ko] undefined!
>
> gcc seems to define __builtin_clz as __clzsi2 in some architecture.
> But, kernel doesn't link libgcc.a.
> If kernel should use gcc's built-in function without libgcc.a,
> do we need to port __clzsi2 to 'arch/*/lib/*'?

That's another option.

So far no one used __builtin_c[lt]z in generic code. We always used
__fls and __ffs.

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [PATCH 1/2] lib: Add lz4 compressor module

[Andrew Morton]

On Mon, 22 Apr 2013 11:24:21 +0200 Geert Uytterhoeven <geert@linux-m68k.org> wrote:

> On Mon, Apr 22, 2013 at 11:22 AM, Chanho Min <chanho.min@lge.com> wrote:
> >>> +#define HTYPE const u8*
> >>> +
> >>> +#ifdef __BIG_ENDIAN
> >>> +#define LZ4_NBCOMMONBYTES(val) (__builtin_clz(val) >> 3)
> >>> +#else
> >>> +#define LZ4_NBCOMMONBYTES(val) (__builtin_ctz(val) >> 3)
> >>> +#endif
> >>
> >>It seems at least m68k and sparc don't have the __builtin_clz() functions:
> >>
> >>m68k-allmodconfig (http://kisskb.ellerman.id.au/kisskb/buildresult/8572593/):
> >>
> >>ERROR: "__clzsi2" [lib/lz4/lz4hc_compress.ko] undefined!
> >>ERROR: "__clzsi2" [lib/lz4/lz4_compress.ko] undefined!
> >
> > gcc seems to define __builtin_clz as __clzsi2 in some architecture.
> > But, kernel doesn't link libgcc.a.
> > If kernel should use gcc's built-in function without libgcc.a,
> > do we need to port __clzsi2 to 'arch/*/lib/*'?
> 
> That's another option.

Without having seen the patch ....  Yes, if we fix it this way then we
also fix it for future callers of __builtin_clz().

Re: [PATCH 1/2] lib: Add lz4 compressor module

[Andrew Morton]

On Mon, 22 Apr 2013 18:22:18 +0900 "Chanho Min" <chanho.min@lge.com> wrote:

> >> +#define HTYPE const u8*
> >> +
> >> +#ifdef __BIG_ENDIAN
> >> +#define LZ4_NBCOMMONBYTES(val) (__builtin_clz(val) >> 3)
> >> +#else
> >> +#define LZ4_NBCOMMONBYTES(val) (__builtin_ctz(val) >> 3)
> >> +#endif
> >
> >It seems at least m68k and sparc don't have the __builtin_clz() functions:
> >
> >m68k-allmodconfig (http://kisskb.ellerman.id.au/kisskb/buildresult/8572593/):
> >
> >ERROR: "__clzsi2" [lib/lz4/lz4hc_compress.ko] undefined!
> >ERROR: "__clzsi2" [lib/lz4/lz4_compress.ko] undefined!
> 
> gcc seems to define __builtin_clz as __clzsi2 in some architecture. 
> But, kernel doesn't link libgcc.a.
> If kernel should use gcc's built-in function without libgcc.a,
> do we need to port __clzsi2 to 'arch/*/lib/*'?

This breaks alpha (gcc-4.4.4) as well.  Can we please get this fixed
promptly?

Re: [PATCH 1/2] lib: Add lz4 compressor module

[Chanho Min]

>> gcc seems to define __builtin_clz as __clzsi2 in some architecture.
>> But, kernel doesn't link libgcc.a.
>> If kernel should use gcc's built-in function without libgcc.a,
>> do we need to port __clzsi2 to 'arch/*/lib/*'?
>
>This breaks alpha (gcc-4.4.4) as well.  Can we please get this fixed
>promptly?

__clzsi2 can be implemented by using generic functions.
It can be overridden by linking arch-specific versions may not be implemented.
does this way look acceptable?

diff --git a/lib/Makefile b/lib/Makefile
index af79e8c..e17b3ee 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -23,7 +23,7 @@ lib-y += kobject.o klist.o

 obj-y += bcd.o div64.o sort.o parser.o halfmd4.o debug_locks.o random32.o \
       bust_spinlocks.o hexdump.o kasprintf.o bitmap.o scatterlist.o \
-      gcd.o lcm.o list_sort.o uuid.o flex_array.o \
+      gcd.o lcm.o list_sort.o uuid.o flex_array.o clz.o\
       bsearch.o find_last_bit.o find_next_bit.o llist.o memweight.o kfifo.o
 obj-y += string_helpers.o
 obj-$(CONFIG_TEST_STRING_HELPERS) += test-string_helpers.o
diff --git a/lib/clz.c b/lib/clz.c
index e69de29..6794b83 100644
--- a/lib/clz.c
+++ b/lib/clz.c
@@ -0,0 +1,23 @@
+#include <linux/kernel.h>
+
+int __attribute__((weak)) __clzsi2(int val)
+{
+     return BITS_PER_LONG - fls(val);
+}
+EXPORT_SYMBOL(__clzsi2);
+
+#if BITS_PER_LONG == 32
+int __attribute__((weak)) __clzdi2(long val)
+{
+     return BITS_PER_LONG - fls((int)val);
+}
+EXPORT_SYMBOL(__clzdi2);
+#elif BITS_PER_LONG == 64
+int __attribute__((weak)) __clzdi2(long val)
+{
+     return BITS_PER_LONG - fls64((u64)val);
+}
+EXPORT_SYMBOL(__clzdi2);
+#else
+#error BITS_PER_LONG not 32 or 64
+#endif

Re: [PATCH 1/2] lib: Add lz4 compressor module

[Stephen Rothwell]

[-- Attachment #1: Type: text/plain, Size: 308 bytes --]

Hi,

On Fri, 26 Apr 2013 14:02:01 +0900 "Chanho Min" <chanho.min@lge.com> wrote:
>
> 
> @@ -0,0 +1,23 @@
> +#include <linux/kernel.h>
> +
> +int __attribute__((weak)) __clzsi2(int val)

We have __weak in <linux/compiler.h>

-- 
Cheers,
Stephen Rothwell                    sfr@canb.auug.org.au

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 1/2] lib: Add lz4 compressor module

[Andrew Morton]

On Fri, 26 Apr 2013 15:51:05 +1000 Stephen Rothwell <sfr@canb.auug.org.au> wrote:

> Hi,
> 
> On Fri, 26 Apr 2013 14:02:01 +0900 "Chanho Min" <chanho.min@lge.com> wrote:
> >
> > 
> > @@ -0,0 +1,23 @@
> > +#include <linux/kernel.h>
> > +
> > +int __attribute__((weak)) __clzsi2(int val)
> 
> We have __weak in <linux/compiler.h>

And lib/clz.c needs a few more includes, for EXPORT_SYMBOL,
BITS_PER_LONG maybe.

Plus a changelog and a signed-off-by, please.