[Buildroot] Crosstool/Buildroot integration issue

[Buildroot] Crosstool/Buildroot integration issue

[Andy Gibbs]

Hi Yann, Thomas

I know you've been working on improving the integration of crosstool and buildroot recently.  Thank you - I really appreciate it, but I'd like to point out an extra detail that may have been missed so far, and looking at toolchain/external-toolchain/ext-tool.mk in the git trunk, I think this still is missing.

Crosstool generates character set libraries in the 'sys-root/usr/lib/gconv' folder, but these are not being copied across into the target folder when all the other libraries are copied.  These means that certain applications that rely on these libraries don't function correctly.  It's a simple two line patch to the bottom of ext-tool.mk:
 
  mkdir -p $(TARGET_DIR)/lib
+ mkdir -p $(TARGET_DIR)/usr/lib
  @echo "Copy external toolchain libraries to target..."
  $(Q)for libs in $(LIB_EXTERNAL_LIBS); do \
  $(call copy_toolchain_lib_root,$(ARCH_SYSROOT_DIR),$$libs,/lib,$(BR2_TOOLCHAIN_EXTERNAL_STRIP)); \
  done
  $(Q)for libs in $(USR_LIB_EXTERNAL_LIBS); do \
  $(call copy_toolchain_lib_root,$(ARCH_SYSROOT_DIR),$$libs,/usr/lib,$(BR2_TOOLCHAIN_EXTERNAL_STRIP)); \
+ $(Q)cp -dr $(ARCH_SYSROOT_DIR)/usr/lib/gconv $(TARGET_DIR)/usr/lib
  done
  @echo "Copy external toolchain sysroot to staging..."
  $(Q)$(call copy_toolchain_sysroot,$(SYSROOT_DIR),$(ARCH_SYSROOT_DIR),$(ARCH_SUBDIR))

I'm afraid I've not attached it as a proper patch since I'm not at my desk currently and can't form an "official" git patch.  However, if you wish me to, then let me know and I will be happy to do so.

Thanks
Andy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20100707/2ae46a5f/attachment.html>

[Buildroot] Crosstool/Buildroot integration issue

[Thomas Petazzoni]

Hello Andy,

On Wed, 7 Jul 2010 16:19:44 +0200
"Andy Gibbs" <andyg1001@hotmail.co.uk> wrote:

> Hi Yann, Thomas
> 
> I know you've been working on improving the integration of crosstool
> and buildroot recently.  Thank you - I really appreciate it,

Thanks, glad to know this feature is useful.

> Crosstool generates character set libraries in the
> 'sys-root/usr/lib/gconv' folder, but these are not being copied
> across into the target folder when all the other libraries are
> copied.  These means that certain applications that rely on these
> libraries don't function correctly.

I definitely haven't looked at this. Can you provide a simple example
of an application where this is causing an issue, that I could use as a
test case ?

> I'm afraid I've not attached it as a proper patch since I'm not at my
> desk currently and can't form an "official" git patch.  However, if
> you wish me to, then let me know and I will be happy to do so.

Sure. However, I first would like to fully understand what is needed
exactly, for which applications, in which cases, etc. As requested
above, a simple test case would be very perfect.

Thank you very much for your feedback on this!

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] Crosstool/Buildroot integration issue

[Yann E. MORIN]

Andy, All,

On Wednesday 07 July 2010 16:19:44 Andy Gibbs wrote:
> I know you've been working on improving the integration of crosstool
> and buildroot recently.  Thank you - I really appreciate it,

Hey! Nice to see someone is interested in that! :-)

> but I'd like to point out an extra detail that may have been missed so
> far, and looking at toolchain/external-toolchain/ext-tool.mk in the git
> trunk, I think this still is missing.
> 
> Crosstool generates character set libraries in the 'sys-root/usr/lib/gconv'

There's no sysrot/usr/lib/gconv in my uClibc-based toolchain! :-P

> folder, but these are not being copied across into the target folder when
> all the other libraries are copied.  These means that certain applications
> that rely on these libraries don't function correctly.  It's a simple two
> line patch to the bottom of ext-tool.mk:
[--SNIp--]
> + $(Q)cp -dr $(ARCH_SYSROOT_DIR)/usr/lib/gconv $(TARGET_DIR)/usr/lib

Well, I'd rather we had something like:
- in menuconfig:
    () Additional dirs to copy from sysroot
      Help:
        This is a space-separated list of paths that you want to copy
        from the sysroot to the target. Make the paths relative to the
        sysroot. Example:
          /usr/lib/gconv /usr/include

- and in ext-tool.mk (to replace your changes, above):
  $(Q)for d in $(BR_EXT_TOOL_DIRS_EXTRA); do \
        if [ -d "$(ARCH_SYSROOT_DIR)$${d}" ]; then
          mkdir -p "$(TARGET_DIR)$${d}"
          cp -a "$${d}/." "$(TARGET_DIR)${d}/."
        else
          echo "No such directory '$${d}' whiel trying to copy from sysroot"
        fi
      done

But that's just my 2-cent advice...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] Crosstool/Buildroot integration issue

[Andy Gibbs]

On Wednesday, July 07, 2010 7:19 PM, Yann E. MORIN wrote:

>> Crosstool generates character set libraries in the 
>> 'sys-root/usr/lib/gconv'
>
> There's no sysrot/usr/lib/gconv in my uClibc-based toolchain! :-P
>

Yes, it may just be part of a glibc/eglibc toolchain (gcc 4.4.4 / eglibc 
2.10 in my case).

>> folder, but these are not being copied across into the target folder when
>> all the other libraries are copied.  These means that certain 
>> applications
>> that rely on these libraries don't function correctly.  It's a simple two
>> line patch to the bottom of ext-tool.mk:
> [--SNIp--]
>> + $(Q)cp -dr $(ARCH_SYSROOT_DIR)/usr/lib/gconv $(TARGET_DIR)/usr/lib
>
> Well, I'd rather we had something like:
> - in menuconfig:
>    () Additional dirs to copy from sysroot
>      Help:
>        This is a space-separated list of paths that you want to copy
>        from the sysroot to the target. Make the paths relative to the
>        sysroot. Example:
>          /usr/lib/gconv /usr/include

I think this would be a perfect way of providing the solution.  Especially 
since, as you noted, it may be that different toolchains have different 
library dependencies.

> - and in ext-tool.mk (to replace your changes, above):
>  $(Q)for d in $(BR_EXT_TOOL_DIRS_EXTRA); do \
>        if [ -d "$(ARCH_SYSROOT_DIR)$${d}" ]; then
>          mkdir -p "$(TARGET_DIR)$${d}"
>          cp -a "$${d}/." "$(TARGET_DIR)${d}/."
>        else
>          echo "No such directory '$${d}' whiel trying to copy from 
> sysroot"
>        fi
>      done
>

Can I suggest that it exits the make process on failure to find the 
directory, not just print the warning?  The reason is that there are a lot 
of messages, and it is difficult often to spot a problem if the make process 
doesn't end where an error occurs.

Its the same with the kernel's make modules step in buildroot -- this can 
fail and the buildroot process continues, leaving the target folder without 
kernel modules!  (In this case it was failing because prior to the 'make 
modules', buildroot calls 'make prepare' rather than 'make modules_prepare', 
plus for some reason the linux make was not building the dependencies for 
the modules, requiring an extra 'make arch/powerpc/lib/' in my case). 
Again, I only mention this in case someone trawling this list with a similar 
problem can find a solution here!

Cheers, and thanks again for all the good work!!

Andy

[Buildroot] Crosstool/Buildroot integration issue

[Thomas Petazzoni]

On Wed, 7 Jul 2010 19:19:28 +0200
"Yann E. MORIN" <yann.morin.1998@anciens.enib.fr> wrote:

> Well, I'd rather we had something like:
> - in menuconfig:
>     () Additional dirs to copy from sysroot
>       Help:
>         This is a space-separated list of paths that you want to copy
>         from the sysroot to the target. Make the paths relative to the
>         sysroot. Example:
>           /usr/lib/gconv /usr/include
> 
> - and in ext-tool.mk (to replace your changes, above):
>   $(Q)for d in $(BR_EXT_TOOL_DIRS_EXTRA); do \
>         if [ -d "$(ARCH_SYSROOT_DIR)$${d}" ]; then
>           mkdir -p "$(TARGET_DIR)$${d}"
>           cp -a "$${d}/." "$(TARGET_DIR)${d}/."
>         else
>           echo "No such directory '$${d}' whiel trying to copy from sysroot"
>         fi
>       done
> 
> But that's just my 2-cent advice...

At first look, I'm not a big fan of this solution, as it requires a lot
of knowledge from the Buildroot user about dependency details that
precisely Buildroot is supposed to handle.

I'd prefer to understand why and when these gconv files are needed, and
depending on that, see how it should fit in our configuration system.

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] Crosstool/Buildroot integration issue

[Andy Gibbs]

On Thursday, July 08, 2010 9:34 AM, Thomas Petazzoni wrote:

> At first look, I'm not a big fan of this solution, as it requires a lot
> of knowledge from the Buildroot user about dependency details that
> precisely Buildroot is supposed to handle.
>
> I'd prefer to understand why and when these gconv files are needed, and
> depending on that, see how it should fit in our configuration system.
>

I've had a google around for some more information, and it comes down to it 
being part of glibc (and by extension eglibc).  There is a whole page of 
information at 
http://www.gnu.org/s/libc/manual/html_node/glibc-iconv-Implementation.html. 
It is part of the character set conversion functions that are part of glibc. 
I understand from Yann's comments that uclibc doesn't include this folder 
(therefore either handles this another way, or doesn't at all!).

Further information is at 
http://www.gnu.org/s/libc/manual/html_node/Generic-Charset-Conversion.html.

Maybe if buildroot can determine that the external toolchain is glibc/eglibc 
based, it can simply then copy this folder over.  Below I've included a 
listing of the files contained in the folder (at least in my case). 
Basically they're all .so files, but there's also a 'gconv-modules' text 
file that holds the configuration data.

Maybe, for those who won't want / don't need it there could still be a 
buildroot configuration option controlling the copy, but the default 
position to do the copy?  Or maybe this should be something at the crosstool 
end?

Cheers
Andy


ANSI_X3.110.so       IBM037.so       IBM423.so           ISO_2033.so
ARMSCII-8.so         IBM038.so       IBM424.so           ISO_5427-EXT.so
ASMO_449.so          IBM1004.so      IBM437.so           ISO_5427.so
BIG5HKSCS.so         IBM1008_420.so  IBM4517.so          ISO_5428.so
BIG5.so              IBM1008.so      IBM4899.so          ISO646.so
BRF.so               IBM1025.so      IBM4909.so          ISO_6937-2.so
CP10007.so           IBM1026.so      IBM4971.so          ISO_6937.so
CP1125.so            IBM1046.so      IBM500.so           ISO8859-10.so
CP1250.so            IBM1047.so      IBM5347.so          ISO8859-11.so
CP1251.so            IBM1097.so      IBM803.so           ISO8859-13.so
CP1252.so            IBM1112.so      IBM850.so           ISO8859-14.so
CP1253.so            IBM1122.so      IBM851.so           ISO8859-15.so
CP1254.so            IBM1123.so      IBM852.so           ISO8859-16.so
CP1255.so            IBM1124.so      IBM855.so           ISO8859-1.so
CP1256.so            IBM1129.so      IBM856.so           ISO8859-2.so
CP1257.so            IBM1130.so      IBM857.so           ISO8859-3.so
CP1258.so            IBM1132.so      IBM860.so           ISO8859-4.so
CP737.so             IBM1133.so      IBM861.so           ISO8859-5.so
CP775.so             IBM1137.so      IBM862.so           ISO8859-6.so
CP932.so             IBM1140.so      IBM863.so           ISO8859-7.so
CSN_369103.so        IBM1141.so      IBM864.so           ISO8859-8.so
CWI.so               IBM1142.so      IBM865.so           ISO8859-9E.so
DEC-MCS.so           IBM1143.so      IBM866NAV.so        ISO8859-9.so
EBCDIC-AT-DE-A.so    IBM1144.so      IBM866.so           ISO-IR-197.so
EBCDIC-AT-DE.so      IBM1145.so      IBM868.so           ISO-IR-209.so
EBCDIC-CA-FR.so      IBM1146.so      IBM869.so           JOHAB.so
EBCDIC-DK-NO-A.so    IBM1147.so      IBM870.so           KOI8-R.so
EBCDIC-DK-NO.so      IBM1148.so      IBM871.so           KOI8-RU.so
EBCDIC-ES-A.so       IBM1149.so      IBM874.so           KOI-8.so
EBCDIC-ES.so         IBM1153.so      IBM875.so           KOI8-T.so
EBCDIC-ES-S.so       IBM1154.so      IBM880.so           KOI8-U.so
EBCDIC-FI-SE-A.so    IBM1155.so      IBM891.so           LATIN-GREEK-1.so
EBCDIC-FI-SE.so      IBM1156.so      IBM901.so           LATIN-GREEK.so
EBCDIC-FR.so         IBM1157.so      IBM902.so           libCNS.so
EBCDIC-IS-FRISS.so   IBM1158.so      IBM9030.so          libGB.so
EBCDIC-IT.so         IBM1160.so      IBM903.so           libISOIR165.so
EBCDIC-PT.so         IBM1161.so      IBM904.so           libJIS.so
EBCDIC-UK.so         IBM1162.so      IBM905.so           libJISX0213.so
EBCDIC-US.so         IBM1163.so      IBM9066.so          libKSC.so
ECMA-CYRILLIC.so     IBM1164.so      IBM918.so 
MAC-CENTRALEUROPE.so
EUC-CN.so            IBM1166.so      IBM921.so           MACINTOSH.so
EUC-JISX0213.so      IBM1167.so      IBM922.so           MAC-IS.so
EUC-JP-MS.so         IBM12712.so     IBM930.so           MAC-SAMI.so
EUC-JP.so            IBM1364.so      IBM932.so           MAC-UK.so
EUC-KR.so            IBM1371.so      IBM933.so           MIK.so
EUC-TW.so            IBM1388.so      IBM935.so           NATS-DANO.so
GB18030.so           IBM1390.so      IBM937.so           NATS-SEFI.so
GBBIG5.so            IBM1399.so      IBM939.so           PT154.so
GBGBK.so             IBM16804.so     IBM943.so           RK1048.so
GBK.so               IBM256.so       IBM9448.so          SAMI-WS2.so
gconv-modules        IBM273.so       IEC_P27-1.so        SHIFT_JISX0213.so
GEORGIAN-ACADEMY.so  IBM274.so       INIS-8.so           SJIS.so
GEORGIAN-PS.so       IBM275.so       INIS-CYRILLIC.so    T.61.so
GOST_19768-74.so     IBM277.so       INIS.so             TCVN5712-1.so
GREEK7-OLD.so        IBM278.so       ISIRI-3342.so       TIS-620.so
GREEK7.so            IBM280.so       ISO_10367-BOX.so    TSCII.so
GREEK-CCITT.so       IBM281.so       ISO_11548-1.so      UHC.so
HP-GREEK8.so         IBM284.so       ISO-2022-CN-EXT.so  UNICODE.so
HP-ROMAN8.so         IBM285.so       ISO-2022-CN.so      UTF-16.so
HP-ROMAN9.so         IBM290.so       ISO-2022-JP-3.so    UTF-32.so
HP-THAI8.so          IBM297.so       ISO-2022-JP.so      UTF-7.so
HP-TURKISH8.so       IBM420.so       ISO-2022-KR.so      VISCII.so

[Buildroot] Crosstool/Buildroot integration issue

[Andy Gibbs]

On Wednesday, July 07, 2010 7:19 PM, Yann E. MORIN wrote:

> Well, I'd rather we had something like:
> - in menuconfig:
>    () Additional dirs to copy from sysroot
>      Help:
>        This is a space-separated list of paths that you want to copy
>        from the sysroot to the target. Make the paths relative to the
>        sysroot. Example:
>          /usr/lib/gconv /usr/include
>
> - and in ext-tool.mk (to replace your changes, above):
>  $(Q)for d in $(BR_EXT_TOOL_DIRS_EXTRA); do \
>        if [ -d "$(ARCH_SYSROOT_DIR)$${d}" ]; then
>          mkdir -p "$(TARGET_DIR)$${d}"
>          cp -a "$${d}/." "$(TARGET_DIR)${d}/."
>        else
>          echo "No such directory '$${d}' whiel trying to copy from 
> sysroot"
>        fi
>      done
>
> But that's just my 2-cent advice...
>

You know the more I think about it, the more I like this sort of approach. 
I understand Thomas' view that dependencies should be automatic as much as 
possible, but the more I look into it, the more I think that buildroot could 
also benefit from some "advanced" setting to allow additional files and 
folders to be copied across.

Beyond the 'gconv' folder mentioned in this thread, there are other 
libraries, binaries and miscellaneous text files that could (should in my 
case) also be copied across from the sys-root folder depending on the target 
system's requirements.

I've attached to this post a snippet of bash script that I now run prior to 
the buildroot process to copy across the missing files and folders that 
buildroot doesn't itself copy.  There is some duplication - for example, the 
script copies /lib/libnss_*.so* since buildroot only copies two of these 
files, and not the libnss_compat.so that my system needs.

Maybe this can form the basis of an "advanced" copy function inside 
buildroot?  The files list would then obviously be changed to a 
configuration variable for the user, rather than a static list.  To my mind, 
it makes sense to do this since buildroot cannot necessarily know or detect 
all the necessary dependencies, and what may be dependencies in one case may 
not be in the general case, and it certainly doesn't make sense to do a 
blanket copy of the whole sys-root directory.

If there is interest in this, and it seems an appropriate route to take, 
I'll happily put together a proposal patch...

Cheers
Andy

-------------- next part --------------
A non-text attachment was scrubbed...
Name: copy
Type: application/octet-stream
Size: 1317 bytes
Desc: not available
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20100709/07e8e775/attachment.obj>