[Buildroot] [Bug 7016] New: Git issues in resulting buildroot tar

[Buildroot] [Bug 7016] New: Git issues in resulting buildroot tar

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=7016

           Summary: Git issues in resulting buildroot tar
           Product: buildroot
           Version: 2014.02
          Platform: PC
        OS/Version: Linux
            Status: NEW
          Severity: normal
          Priority: P5
         Component: Other
        AssignedTo: unassigned at buildroot.uclibc.org
        ReportedBy: brian at brianclements.net
                CC: buildroot at uclibc.org
   Estimated Hours: 0.0


Hi There,
Sorry if this isn't the correct place for this bug, please forward me if so.

I've built a pretty standard busybox chain using many different options and
versions of git by this point, but the end result has been the same: using git
in this freshly built image fails on anything involved decompressing objects
(git clone among other things).

The context here has been to make a Docker image of busybox with a complete git
install contained therein. All my reproducible steps are here:

https://github.com/brianclements/dockerfile-busyboxplus/blob/gitbug/tarmaker/Dockerfile

I've even tried substituting different versions of buildroot (2013.08.1,
2014.02) and different versions of git (1.8.3, 1.8.5.3, 1.9.0) all with the
same result. The build succeeds, and the resulting image works as far as I can
tell, but running "git clone" on a larger repository that involves compression
of objects (smaller repos of text only work fine) will fail with: 

fatal: serious inflate inconsistency   
fatal: index-pack failed

Any thoughts or further testing I can try to tweak the git build?

Thanks.

-- 
Configure bugmail: https://bugs.busybox.net/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Buildroot] [Bug 7016] Git issues in resulting buildroot tar

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=7016

--- Comment #1 from Thomas De Schampheleire <patrickdepinguin+buildroot@gmail.com> 2014-04-02 07:35:03 UTC ---
Can you provide more details on how you actually run this image? I am not
familiar with Docker, so more details about how you run this is very welcome.

Have you tried using a similar configuration on a non-docker environment, like
qemu or a real target?

Also, did you try to run git through 'strace' to see what is actually going on?
Have you tried making a new local git repo using the git inside this image, and
then clone that with the same git? Or have you only tried cloning an external
git repo? If there is a problem with importing git data, the inflate can
obviously fail. Generating a local git repo would help excluding that.

For reference to others, here seems to be the basic flow to generate docker
image using buildroot: (Jerome Petazzoni)
http://blog.docker.io/2013/06/create-light-weight-docker-containers-buildroot/

-- 
Configure bugmail: https://bugs.busybox.net/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Buildroot] [Bug 7016] Git issues in resulting buildroot tar

[brianclements]

https://bugs.busybox.net/show_bug.cgi?id=7016

>Can you provide more details on how you actually run this image? I am not
>familiar with Docker, so more details about how you run this is very
welcome.

Sure. You can go to https://www.docker.io/gettingstarted/ to install docker
on a 64 bit linux system. It uses linux containers to borrow the host's
kernel and run isolated processes with very little overhead. 

Then to build, clone the "gitbug" branch of this github repo:
`git clone https://github.com/brianclements/dockerfile-busyboxplus.git
--branch gitbug`

Then as sudo -i (so you don't have to babysit your sudo timeout) run the
script:
`./run.sh git`

It will then start one container to build the rootfs.tar, then the script
copies it out of the build container and into the host filesystem (in the
/git folder in the repo) and import that resulting image back into docker
under the name "brianclements/busyboxplus:git"

To see the details of how I build it, you can look at this Dockerfile (which
works much like a bash script)
https://github.com/brianclements/dockerfile-busyboxplus/blob/gitbug/git/tarmaker/Dockerfile

You can then run commands in this new imported image by prefixing your
command with:
`sudo docker run brianclements/busyboxplus:git <COMMAND GOES HERE>`

or, more intuitively, you can "enter" the container and be interactive by
running:
`sudo docker run -it brianclements/busyboxplus:git /bin/sh`
which should give you a root prompt inside the busybox container. There you
can `git clone` and test away. Numerous tests have been cataloged in the
github issue tracker in this repo for your perusal (the only 2 issues in
that repo relate to this git issue).
https://github.com/brianclements/dockerfile-busyboxplus/tree/gitbug

>Have you tried using a similar configuration on a non-docker environment,
like
>qemu or a real target?

Yes, a build on my host, Ubuntu 13.10, gave the same exact behavior with the
resulting rootfs.tar. I don't have any experience with qemo, but perhaps you
could reproduce it on those systems quicker then I? Even when using the
`make menuconfig` all I selected was x86_64 architecture, wide character,
largefile, inet_ipv6, bridge_utils, iproute2, iptables, then git. In fact
all you need to do is the bare minimum and it still gives the error (x86_64,
largefile, git).

>Also, did you try to run git through 'strace' to see what is actually going
on?

I admit I have basic knowledge of strace, let me know if you need it run
with different options. I compiled a container identical in all ways with
the exception of the strace package to get the attached file.

strace-git.strace-git
<http://buildroot-busybox.2317881.n4.nabble.com/file/n66989/strace-git.strace-git>  

>Have you tried making a new local git repo using the git inside this image,
and
>then clone that with the same git? Or have you only tried cloning an
external
>git repo? If there is a problem with importing git data, the inflate can
>obviously fail. Generating a local git repo would help excluding that.

The issues on github give some of the details of what I've tried, but in
short, it seems to be the "complexity" of the repo being cloned (number of
objects I'm thinking, not the size) and NOT it's source that causes the
issue (local or internet). Docker has the ability to insert files into a
container and I've added the repository that the original bug reporter
listed (which was large with over 56,000 objects) and had the same issue
cloning internally as I did over the Internet. This to me ruled out any data
corruption over the network bridge as it passed into the container and
refocuses squarely on how git decompresses these objects in my image.




--
View this message in context: http://buildroot-busybox.2317881.n4.nabble.com/Bug-7016-New-Git-issues-in-resulting-buildroot-tar-tp66457p66989.html
Sent from the Buildroot (busybox) mailing list archive at Nabble.com.

[Buildroot] [Bug 7016] Git issues in resulting buildroot tar

[Thomas De Schampheleire]

Hi Brian,

On Mon, Apr 7, 2014 at 9:26 AM, brianclements <brian@brianclements.net> wrote:
> https://bugs.busybox.net/show_bug.cgi?id=7016
>
>>Can you provide more details on how you actually run this image? I am not
>>familiar with Docker, so more details about how you run this is very
> welcome.
>
> Sure. You can go to https://www.docker.io/gettingstarted/ to install docker
> on a 64 bit linux system. It uses linux containers to borrow the host's
> kernel and run isolated processes with very little overhead.
>
> Then to build, clone the "gitbug" branch of this github repo:
> `git clone https://github.com/brianclements/dockerfile-busyboxplus.git
> --branch gitbug`
>
> Then as sudo -i (so you don't have to babysit your sudo timeout) run the
> script:
> `./run.sh git`
>
> It will then start one container to build the rootfs.tar, then the script
> copies it out of the build container and into the host filesystem (in the
> /git folder in the repo) and import that resulting image back into docker
> under the name "brianclements/busyboxplus:git"
>
> To see the details of how I build it, you can look at this Dockerfile (which
> works much like a bash script)
> https://github.com/brianclements/dockerfile-busyboxplus/blob/gitbug/git/tarmaker/Dockerfile
>
> You can then run commands in this new imported image by prefixing your
> command with:
> `sudo docker run brianclements/busyboxplus:git <COMMAND GOES HERE>`
>
> or, more intuitively, you can "enter" the container and be interactive by
> running:
> `sudo docker run -it brianclements/busyboxplus:git /bin/sh`
> which should give you a root prompt inside the busybox container. There you
> can `git clone` and test away. Numerous tests have been cataloged in the
> github issue tracker in this repo for your perusal (the only 2 issues in
> that repo relate to this git issue).
> https://github.com/brianclements/dockerfile-busyboxplus/tree/gitbug
>
>>Have you tried using a similar configuration on a non-docker environment,
> like
>>qemu or a real target?
>
> Yes, a build on my host, Ubuntu 13.10, gave the same exact behavior with the
> resulting rootfs.tar. I don't have any experience with qemo, but perhaps you
> could reproduce it on those systems quicker then I? Even when using the
> `make menuconfig` all I selected was x86_64 architecture, wide character,
> largefile, inet_ipv6, bridge_utils, iproute2, iptables, then git. In fact
> all you need to do is the bare minimum and it still gives the error (x86_64,
> largefile, git).

It is interesting that you can reproduce without docker.
Can you clarify how you started the system in this case? How do you
deploy the rootfs.tar file in this case?

One of the reasons that I asked about this, is that it is not correct
to simply start an application built with buildroot on an existing
system with other libraries, because they may not be compatible.
Therefore, you need to run as a 'full' system, like in qemu, or
possibly with chroot if the target and guest systems are of the same
architecture. See also
http://buildroot.uclibc.org/downloads/manual/manual.html#_chroot

Depending on how you are running in the non-docker case, you may be
experiencing precisely this problem in both cases.

>
>>Also, did you try to run git through 'strace' to see what is actually going
> on?
>
> I admit I have basic knowledge of strace, let me know if you need it run
> with different options. I compiled a container identical in all ways with
> the exception of the strace package to get the attached file.
>
> strace-git.strace-git
> <http://buildroot-busybox.2317881.n4.nabble.com/file/n66989/strace-git.strace-git>

I have performed the same git clone on my laptop to compare the output.
In my case, I get the following part corresponding to some place near
the end of your log:

3071  clone(child_stack=0,
flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
child_tidptr=0x7f3b51c0da10) = 3080
3080  set_robust_list(0x7f3b51c0da20, 0x18 <unfinished ...>
3071  close(10 <unfinished ...>
3080  <... set_robust_list resumed> )   = 0
3071  <... close resumed> )             = 0
3080  close(9 <unfinished ...>
3071  read(9,  <unfinished ...>
3080  <... close resumed> )             = 0
3080  fcntl(10, F_GETFD)                = 0
3080  fcntl(10, F_SETFD, FD_CLOEXEC)    = 0
3080  dup2(3, 0)                        = 0
3080  close(3)                          = 0
3080  dup2(8, 1)                        = 1
3080  close(7)                          = 0
3080  close(8)                          = 0
3080  execve("/usr/lib/git-core/git", ["git", "index-pack", "--stdin",
"-v", "--fix-thin", "--keep=fetch-pack 3071 on argent"...], [/* 63
vars */]) = 0

Here, thread 3071 forks and creates a child with pid 3080, which
launches 'git index-pack --stdin -v ...'.
However, in your case, there is:

clone(child_stack=0,
flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
child_tidptr=0x7fde4b077970) = 13
close(10)                               = 0
read(9, "", 1)                          = 0
close(9)                                = 0
close(5)                                = 0
close(8)                                = 0
read(7, "", 46)                         = 0
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=13,
si_status=128, si_utime=436, si_stime=171} ---
close(7)                                = 0
wait4(13, [{WIFEXITED(s) && WEXITSTATUS(s) == 128}], 0, NULL) = 13
write(2, "fatal: ", 7)                  = 7
write(2, "index-pack failed", 17)       = 17
write(2, "\n", 1)                       = 1

Here the child tid is 13, and shortly after its creation it already
stops, generating a SIGCHLD signal.

This even seems to be before the 'execve' call, but maybe this is just
a side-effect of the tracing display rather than the actual problem.
I'm not entirely sure if this helps you in analyzing the problem, though.
Could you check if '/usr/lib/git-core/git' is available and if it can
be executed?

Best regards,
Thomas

[Buildroot] [Bug 7016] Git issues in resulting buildroot tar

[brianclements]

Hi Thomas,

>It is interesting that you can reproduce without docker.
>Can you clarify how you started the system in this case? How do you
>deploy the rootfs.tar file in this case?

>One of the reasons that I asked about this, is that it is not correct
>to simply start an application built with buildroot on an existing
>system with other libraries, because they may not be compatible.
>Therefore, you need to run as a 'full' system, like in qemu, or
>possibly with chroot if the target and guest systems are of the same
>architecture. See also
>http://buildroot.uclibc.org/downloads/manual/manual.html#_chroot

>Depending on how you are running in the non-docker case, you may be
>experiencing precisely this problem in both cases.

I'm sorry I misunderstood your question before. What I replicated was not
the running of the git binary with the errors, but the building of the
rootfs.tar from my host and not from within a docker container. In this case
the two rootfs.tar images resulted in the same error with git once imported
into Docker. I will have to look into your suggestions later for actually
running this image using a different virtualization method then Docker and
update you on it.

>I have performed the same git clone on my laptop to compare the output.
>In my case, I get the following part corresponding to some place near
>the end of your log:

>3071  clone(child_stack=0,
>flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
>child_tidptr=0x7f3b51c0da10) = 3080
>3080  set_robust_list(0x7f3b51c0da20, 0x18 <unfinished ...>
>3071  close(10 <unfinished ...>
>3080  <... set_robust_list resumed> )   = 0
>3071  <... close resumed> )             = 0
>3080  close(9 <unfinished ...>
>3071  read(9,  <unfinished ...>
>3080  <... close resumed> )             = 0
>3080  fcntl(10, F_GETFD)                = 0
>3080  fcntl(10, F_SETFD, FD_CLOEXEC)    = 0
>3080  dup2(3, 0)                        = 0
>3080  close(3)                          = 0
>3080  dup2(8, 1)                        = 1
>3080  close(7)                          = 0
>3080  close(8)                          = 0
>3080  execve("/usr/lib/git-core/git", ["git", "index-pack", "--stdin",
>"-v", "--fix-thin", "--keep=fetch-pack 3071 on argent"...], [/* 63
>vars */]) = 0

>Here, thread 3071 forks and creates a child with pid 3080, which
>launches 'git index-pack --stdin -v ...'.
>However, in your case, there is:

>clone(child_stack=0,
>flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
>child_tidptr=0x7fde4b077970) = 13
>close(10)                               = 0
>read(9, "", 1)                          = 0
>close(9)                                = 0
>close(5)                                = 0
>close(8)                                = 0
>read(7, "", 46)                         = 0
>--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=13,
>si_status=128, si_utime=436, si_stime=171} ---
>close(7)                                = 0
>wait4(13, [{WIFEXITED(s) && WEXITSTATUS(s) == 128}], 0, NULL) = 13
>write(2, "fatal: ", 7)                  = 7
>write(2, "index-pack failed", 17)       = 17
>write(2, "\n", 1)                       = 1

>Here the child tid is 13, and shortly after its creation it already
>stops, generating a SIGCHLD signal.

>This even seems to be before the 'execve' call, but maybe this is just
>a side-effect of the tracing display rather than the actual problem.
>I'm not entirely sure if this helps you in analyzing the problem, though.
>Could you check if '/usr/lib/git-core/git' is available and if it can
>be executed?

I have to admit the strace information and system calls are completely
foreign to me at the moment. I'll have to dig into them a little bit too to
see if they can shed some light on the situation.

As for the git binary, it's actually in a different location
"/usr/libexec/git-core", does that matter? Here is the status of it as it
resides in the image:

/usr/libexec/git-core # ls -hal git
-rwxr-xr-x    1 root     root        1.2M Apr  7 06:49 git
/usr/libexec/git-core # ./git --version
git version 1.8.5.3

it seems to run OK from that path. Did I do that right? Side note: git-clone
most everything in that directory have the same permissions and seem
accessible.

Let me ask you this, can you possibly just compile a version of busybox with
x86_64, largefile, and git and link me up with the resulting tar? (only
probably around 10mb). That might be a quicker way to duplicate this error
then me trying to learn my way through qemu and introduce another set of
variables for me to mess up. That way I can see if a working version is even
possible when implemented with Docker and we can possibly get rid of that
variable as well, especially if your image works for you.



--
View this message in context: http://buildroot-busybox.2317881.n4.nabble.com/Bug-7016-New-Git-issues-in-resulting-buildroot-tar-tp66457p67022.html
Sent from the Buildroot (busybox) mailing list archive at Nabble.com.

[Buildroot] [Bug 7016] Git issues in resulting buildroot tar

[Thomas De Schampheleire]

Hi Brian,

On Mon, Apr 7, 2014 at 8:59 PM, brianclements <brian@brianclements.net> wrote:
> Hi Thomas,
>
>>It is interesting that you can reproduce without docker.
>>Can you clarify how you started the system in this case? How do you
>>deploy the rootfs.tar file in this case?
>
>>One of the reasons that I asked about this, is that it is not correct
>>to simply start an application built with buildroot on an existing
>>system with other libraries, because they may not be compatible.
>>Therefore, you need to run as a 'full' system, like in qemu, or
>>possibly with chroot if the target and guest systems are of the same
>>architecture. See also
>>http://buildroot.uclibc.org/downloads/manual/manual.html#_chroot
>
>>Depending on how you are running in the non-docker case, you may be
>>experiencing precisely this problem in both cases.
>
> I'm sorry I misunderstood your question before. What I replicated was not
> the running of the git binary with the errors, but the building of the
> rootfs.tar from my host and not from within a docker container. In this case
> the two rootfs.tar images resulted in the same error with git once imported
> into Docker. I will have to look into your suggestions later for actually
> running this image using a different virtualization method then Docker and
> update you on it.
>
[..]
>
>>This even seems to be before the 'execve' call, but maybe this is just
>>a side-effect of the tracing display rather than the actual problem.
>>I'm not entirely sure if this helps you in analyzing the problem, though.
>>Could you check if '/usr/lib/git-core/git' is available and if it can
>>be executed?
>
> I have to admit the strace information and system calls are completely
> foreign to me at the moment. I'll have to dig into them a little bit too to
> see if they can shed some light on the situation.
>
> As for the git binary, it's actually in a different location
> "/usr/libexec/git-core", does that matter? Here is the status of it as it
> resides in the image:
>
> /usr/libexec/git-core # ls -hal git
> -rwxr-xr-x    1 root     root        1.2M Apr  7 06:49 git
> /usr/libexec/git-core # ./git --version
> git version 1.8.5.3
>
> it seems to run OK from that path. Did I do that right? Side note: git-clone
> most everything in that directory have the same permissions and seem
> accessible.
>
> Let me ask you this, can you possibly just compile a version of busybox with
> x86_64, largefile, and git and link me up with the resulting tar? (only
> probably around 10mb). That might be a quicker way to duplicate this error
> then me trying to learn my way through qemu and introduce another set of
> variables for me to mess up. That way I can see if a working version is even
> possible when implemented with Docker and we can possibly get rid of that
> variable as well, especially if your image works for you.

Just to let you know I haven't forgot about this, but I'm a little
short on time at the moment.
I will try to reproduce the issue in qemu as you suggested.
When I have more news I'll let you know.

In the mean time, if there is any progress at your side, let us know.

Best regards,
Thomas

[Buildroot] [Bug 7016] Git issues in resulting buildroot tar

[Thomas De Schampheleire]

Hi Brian,

On Mon, Apr 7, 2014 at 8:59 PM, brianclements <brian@brianclements.net> wrote:
> Hi Thomas,
>
[..]

> Let me ask you this, can you possibly just compile a version of busybox with
> x86_64, largefile, and git and link me up with the resulting tar? (only
> probably around 10mb). That might be a quicker way to duplicate this error
> then me trying to learn my way through qemu and introduce another set of
> variables for me to mess up. That way I can see if a working version is even
> possible when implemented with Docker and we can possibly get rid of that
> variable as well, especially if your image works for you.
>

I now took the time to generate such an image.
I started from the qemu_x86_64 defconfig, and changed:
- use of external toolchain rather than buildroot-built toolchain (to
save built time). This external toolchain includes largefile support.
- add git, dhcpcd (I needed this to get proper networking from qemu)

When running the image, I did:
- login (root)
- dhcpcd (get ip address)
- echo 'nameserver 8.8.8.8' > /etc/resolv.conf  (set up DNS with
Google's public DNS)
- git clone git://github.com/EllisLab/CodeIgniter

This works fine. I had to increase the size of the ext2 rootfs (used
in qemu instead of rootfs.tar) to get enough space, but other than
that no problems.

The rootfs.tar is 15M, the bzImage (kernel) is 2.2M. If you have a
place where I can upload it to, let me know.

The defconfig is as below:

BR2_x86_64=y
BR2_DL_DIR="/home/tdescham/repo/contrib/dl"
BR2_TOOLCHAIN_EXTERNAL=y
BR2_TARGET_GENERIC_GETTY_PORT="tty1"
BR2_LINUX_KERNEL=y
BR2_LINUX_KERNEL_CUSTOM_VERSION=y
BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE="3.13.5"
BR2_LINUX_KERNEL_USE_CUSTOM_CONFIG=y
BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE="board/qemu/x86_64/linux-3.13.config"
BR2_PACKAGE_BUSYBOX_SHOW_OTHERS=y
BR2_PACKAGE_GIT=y
BR2_PACKAGE_DHCPCD=y
BR2_TARGET_ROOTFS_EXT2=y
BR2_TARGET_ROOTFS_EXT2_BLOCKS=200000

Best regards,
Thomas

[Buildroot] [Bug 7016] Git issues in resulting buildroot tar

[brianclements]

Hi Thomas,
I finally have an update for you. I was able to import your rootfs.tar into
docker and use it to successfully pull that git repo without any issue. This
is great! I've tested numerous times and it seems to be sticking that way.
Now, to figure out what's different between your image and mine. I suspect
it's one of two things you did that made the difference:

1) using ext2 in the image and specifying that disksize (less likely) or
2) using an external toolchain (most likely)

the ext2 option is no problem, I'll just include that in the config, but how
do you specify and use an external toolchain? Where do I obtain the
toolchain you used and compile using it?


Thomas De Schampheleire wrote
> Hi Brian,
> 
> On Mon, Apr 7, 2014 at 8:59 PM, brianclements <

> brian@

> > wrote:
>> Hi Thomas,
>>
> [..]
> 
>> Let me ask you this, can you possibly just compile a version of busybox
>> with
>> x86_64, largefile, and git and link me up with the resulting tar? (only
>> probably around 10mb). That might be a quicker way to duplicate this
>> error
>> then me trying to learn my way through qemu and introduce another set of
>> variables for me to mess up. That way I can see if a working version is
>> even
>> possible when implemented with Docker and we can possibly get rid of that
>> variable as well, especially if your image works for you.
>>
> 
> I now took the time to generate such an image.
> I started from the qemu_x86_64 defconfig, and changed:
> - use of external toolchain rather than buildroot-built toolchain (to
> save built time). This external toolchain includes largefile support.
> - add git, dhcpcd (I needed this to get proper networking from qemu)
> 
> When running the image, I did:
> - login (root)
> - dhcpcd (get ip address)
> - echo 'nameserver 8.8.8.8' > /etc/resolv.conf  (set up DNS with
> Google's public DNS)
> - git clone git://github.com/EllisLab/CodeIgniter
> 
> This works fine. I had to increase the size of the ext2 rootfs (used
> in qemu instead of rootfs.tar) to get enough space, but other than
> that no problems.
> 
> The rootfs.tar is 15M, the bzImage (kernel) is 2.2M. If you have a
> place where I can upload it to, let me know.
> 
> The defconfig is as below:
> 
> BR2_x86_64=y
> BR2_DL_DIR="/home/tdescham/repo/contrib/dl"
> BR2_TOOLCHAIN_EXTERNAL=y
> BR2_TARGET_GENERIC_GETTY_PORT="tty1"
> BR2_LINUX_KERNEL=y
> BR2_LINUX_KERNEL_CUSTOM_VERSION=y
> BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE="3.13.5"
> BR2_LINUX_KERNEL_USE_CUSTOM_CONFIG=y
> BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE="board/qemu/x86_64/linux-3.13.config"
> BR2_PACKAGE_BUSYBOX_SHOW_OTHERS=y
> BR2_PACKAGE_GIT=y
> BR2_PACKAGE_DHCPCD=y
> BR2_TARGET_ROOTFS_EXT2=y
> BR2_TARGET_ROOTFS_EXT2_BLOCKS=200000
> 
> Best regards,
> Thomas
> _______________________________________________
> buildroot mailing list

> buildroot@

> http://lists.busybox.net/mailman/listinfo/buildroot





--
View this message in context: http://buildroot-busybox.2317881.n4.nabble.com/Bug-7016-New-Git-issues-in-resulting-buildroot-tar-tp66457p68658.html
Sent from the Buildroot (busybox) mailing list archive at Nabble.com.

[Buildroot] [Bug 7016] Git issues in resulting buildroot tar

[Thomas De Schampheleire]

hi Brian,

(Please don't top post)

brianclements <brian@brianclements.net> schreef:
>Hi Thomas,
>I finally have an update for you. I was able to import your rootfs.tar into
>docker and use it to successfully pull that git repo without any issue. This
>is great! I've tested numerous times and it seems to be sticking that way.
>Now, to figure out what's different between your image and mine.

Good news!

> I suspect
>it's one of two things you did that made the difference:
>
>1) using ext2 in the image and specifying that disksize (less likely) or
>2) using an external toolchain (most likely)
>
>the ext2 option is no problem, I'll just include that in the config, but how
>do you specify and use an external toolchain? Where do I obtain the
>toolchain you used and compile using it?

In the menuconfig, in the toolchain menu, you can specify the toolchain type. It is set to 'Buildroot' by default, but you can specify 'external' here. I then used the default proposed toolchain (Sourcery)

Best regards,
Thomas

[Buildroot] [Bug 7016] Git issues in resulting buildroot tar

[brianclements]

>hi Brian,
>(Please don't top post)

Apologies, I'm new to mailing list etiquette. 

>In the menuconfig, in the toolchain menu, you can specify the toolchain
type. It is set to 'Buildroot' by >default, but you can specify 'external'
here. I then used the default proposed toolchain (Sourcery)

So I've found the fix. Selecting the external toolchain "Sourcery" (and WOW
is it much faster) with my other selections same as before of x86_64 and
git, was all I needed to get git to work. So I know what I need to do now to
get it to work for my uses. But I'm sure that still means there is a bug
somewhere in the buildroot toolchain. I'm not sure how to go about finding
it for you, but I'd be more then happy to test and reproduce things along
the way for you if you desire. Let me know,

Much Thanks!
Brian
_______________________________________________
buildroot mailing list
buildroot@
http://lists.busybox.net/mailman/listinfo/buildroot




--
View this message in context: http://buildroot-busybox.2317881.n4.nabble.com/Bug-7016-New-Git-issues-in-resulting-buildroot-tar-tp66457p68677.html
Sent from the Buildroot (busybox) mailing list archive at Nabble.com.

[Buildroot] [Bug 7016] Git issues in resulting buildroot tar

[Thomas De Schampheleire]

Hi Brian,

On Thu, May 1, 2014 at 10:26 PM, brianclements <brian@brianclements.net> wrote:
[..]
>
>>In the menuconfig, in the toolchain menu, you can specify the toolchain
> type. It is set to 'Buildroot' by >default, but you can specify 'external'
> here. I then used the default proposed toolchain (Sourcery)
>
> So I've found the fix. Selecting the external toolchain "Sourcery" (and WOW
> is it much faster) with my other selections same as before of x86_64 and
> git, was all I needed to get git to work. So I know what I need to do now to
> get it to work for my uses. But I'm sure that still means there is a bug
> somewhere in the buildroot toolchain. I'm not sure how to go about finding
> it for you, but I'd be more then happy to test and reproduce things along
> the way for you if you desire. Let me know,

Could you send me the buildroot config you were using before?
I will build the exact same config this time and see if I can reproduce on qemu.

Best regards,
Thomas

[Buildroot] [Bug 7016] Git issues in resulting buildroot tar

[brianclements]

On Thu, May 1, 2014 at 10:26 PM, brianclements <brian@> wrote:
[..]
>
>>In the menuconfig, in the toolchain menu, you can specify the toolchain
>> type. It is set to 'Buildroot' by >default, but you can specify
>> 'external'
>> here. I then used the default proposed toolchain (Sourcery)
>>
>> So I've found the fix. Selecting the external toolchain "Sourcery" (and
>> WOW
>> is it much faster) with my other selections same as before of x86_64 and
>> git, was all I needed to get git to work. So I know what I need to do now
>> to
>> get it to work for my uses. But I'm sure that still means there is a bug
>> somewhere in the buildroot toolchain. I'm not sure how to go about
>> finding
>> it for you, but I'd be more then happy to test and reproduce things along
>> the way for you if you desire. Let me know,

>Could you send me the buildroot config you were using before?
>I will build the exact same config this time and see if I can reproduce on
qemu.

I wasn't sure which you wanted, so I included both. Both were created with
the command `make menuconfig` and this is with comments and whitespace
removed.

First the old config that broke git which I think is what you meant:

BR2_HAVE_DOT_CONFIG=y
BR2_ARCH_IS_64=y
BR2_x86_64=y
BR2_ARCH="x86_64"
BR2_ENDIAN="LITTLE"
BR2_GCC_TARGET_TUNE="generic"
BR2_x86_generic=y
BR2_WGET="wget --passive-ftp -nd -t 3"
BR2_SVN="svn"
BR2_BZR="bzr"
BR2_GIT="git"
BR2_LOCALFILES="cp"
BR2_SCP="scp"
BR2_SSH="ssh"
BR2_HG="hg"
BR2_ZCAT="gzip -d -c"
BR2_BZCAT="bzcat"
BR2_XZCAT="xzcat"
BR2_TAR_OPTIONS=""
BR2_DEFCONFIG="$(CONFIG_DIR)/defconfig"
BR2_DL_DIR="$(TOPDIR)/dl"
BR2_HOST_DIR="$(BASE_DIR)/host"
BR2_PRIMARY_SITE=""
BR2_BACKUP_SITE="http://sources.buildroot.net/"
BR2_KERNEL_MIRROR="http://www.kernel.org/pub/"
BR2_GNU_MIRROR="http://ftp.gnu.org/pub/gnu"
BR2_DEBIAN_MIRROR="http://ftp.debian.org"
BR2_JLEVEL=0
BR2_STRIP_strip=y
BR2_STRIP_EXCLUDE_FILES=""
BR2_STRIP_EXCLUDE_DIRS=""
BR2_OPTIMIZE_S=y
BR2_PACKAGE_OVERRIDE_FILE="$(TOPDIR)/local.mk"
BR2_GLOBAL_PATCH_DIR=""
BR2_TOOLCHAIN_USES_UCLIBC=y
BR2_TOOLCHAIN_BUILDROOT=y
BR2_KERNEL_HEADERS_3_10=y
BR2_DEFAULT_KERNEL_HEADERS="3.10.10"
BR2_TOOLCHAIN_BUILDROOT_UCLIBC=y
BR2_TOOLCHAIN_BUILDROOT_LIBC="uclibc"
BR2_UCLIBC_VERSION_0_9_33=y
BR2_UCLIBC_VERSION_STRING="0.9.33.2"
BR2_UCLIBC_CONFIG="package/uclibc/uClibc-0.9.33.config"
BR2_TOOLCHAIN_BUILDROOT_LARGEFILE=y
BR2_TOOLCHAIN_BUILDROOT_INET_IPV6=y
BR2_TOOLCHAIN_BUILDROOT_WCHAR=y
BR2_PTHREADS_NATIVE=y
BR2_UCLIBC_INSTALL_UTILS=y
BR2_UCLIBC_TARGET_ARCH="x86_64"
BR2_BINUTILS_VERSION_2_21_1=y
BR2_BINUTILS_VERSION="2.21.1"
BR2_BINUTILS_EXTRA_CONFIG_OPTIONS=""
BR2_GCC_NEEDS_MPC=y
BR2_GCC_VERSION_4_7_X=y
BR2_GCC_SUPPORTS_FINEGRAINEDMTUNE=y
BR2_GCC_VERSION="4.7.3"
BR2_EXTRA_GCC_CONFIG_OPTIONS=""
BR2_GCC_ENABLE_TLS=y
BR2_LARGEFILE=y
BR2_INET_IPV6=y
BR2_USE_WCHAR=y
BR2_TOOLCHAIN_HAS_THREADS=y
BR2_TOOLCHAIN_HAS_SHADOW_PASSWORDS=y
BR2_GENERATE_LOCALE=""
BR2_NEEDS_GETTEXT=y
BR2_USE_MMU=y
BR2_TARGET_OPTIMIZATION="-pipe"
BR2_TARGET_LDFLAGS=""
BR2_TARGET_GENERIC_HOSTNAME="buildroot"
BR2_TARGET_GENERIC_ISSUE="Welcome to Buildroot"
BR2_TARGET_GENERIC_PASSWD_MD5=y
BR2_TARGET_GENERIC_PASSWD_METHOD="md5"
BR2_ROOTFS_DEVICE_CREATION_STATIC=y
BR2_INIT_BUSYBOX=y
BR2_ROOTFS_DEVICE_TABLE="system/device_table.txt"
BR2_ROOTFS_STATIC_DEVICE_TABLE="system/device_table_dev.txt"
BR2_ROOTFS_SKELETON_DEFAULT=y
BR2_TARGET_GENERIC_ROOT_PASSWD=""
BR2_TARGET_GENERIC_GETTY_PORT="ttyS0"
BR2_TARGET_GENERIC_GETTY_BAUDRATE_115200=y
BR2_TARGET_GENERIC_GETTY_BAUDRATE="115200"
BR2_TARGET_GENERIC_GETTY_TERM="vt100"
BR2_TARGET_GENERIC_REMOUNT_ROOTFS_RW=y
BR2_ROOTFS_OVERLAY=""
BR2_ROOTFS_POST_BUILD_SCRIPT=""
BR2_ROOTFS_POST_IMAGE_SCRIPT=""
BR2_PACKAGE_BUSYBOX=y
BR2_BUSYBOX_VERSION_1_21_X=y
BR2_BUSYBOX_VERSION="1.21.1"
BR2_PACKAGE_BUSYBOX_CONFIG="package/busybox/busybox-1.21.x.config"
BR2_PACKAGE_GIT=y
BR2_PACKAGE_ZLIB=y
BR2_PACKAGE_LIBICONV=y
BR2_TARGET_ROOTFS_TAR=y
BR2_TARGET_ROOTFS_TAR_NONE=y
BR2_TARGET_ROOTFS_TAR_OPTIONS=""


And my new and working config (note: on ubuntu, I needed to install the
32bits glibc package, "libc6-i386" before building)

BR2_HAVE_DOT_CONFIG=y
BR2_HOSTARCH_NEEDS_IA32_LIBS=y
BR2_ARCH_IS_64=y
BR2_x86_64=y
BR2_ARCH="x86_64"
BR2_ENDIAN="LITTLE"
BR2_GCC_TARGET_TUNE="generic"
BR2_x86_generic=y
BR2_WGET="wget --passive-ftp -nd -t 3"
BR2_SVN="svn"
BR2_BZR="bzr"
BR2_GIT="git"
BR2_LOCALFILES="cp"
BR2_SCP="scp"
BR2_SSH="ssh"
BR2_HG="hg"
BR2_ZCAT="gzip -d -c"
BR2_BZCAT="bzcat"
BR2_XZCAT="xzcat"
BR2_TAR_OPTIONS=""
BR2_DEFCONFIG="$(CONFIG_DIR)/defconfig"
BR2_DL_DIR="$(TOPDIR)/dl"
BR2_HOST_DIR="$(BASE_DIR)/host"
BR2_PRIMARY_SITE=""
BR2_BACKUP_SITE="http://sources.buildroot.net/"
BR2_KERNEL_MIRROR="http://www.kernel.org/pub/"
BR2_GNU_MIRROR="http://ftp.gnu.org/pub/gnu"
BR2_DEBIAN_MIRROR="http://ftp.debian.org"
BR2_JLEVEL=0
BR2_STRIP_strip=y
BR2_STRIP_EXCLUDE_FILES=""
BR2_STRIP_EXCLUDE_DIRS=""
BR2_OPTIMIZE_S=y
BR2_PACKAGE_OVERRIDE_FILE="$(TOPDIR)/local.mk"
BR2_GLOBAL_PATCH_DIR=""
BR2_TOOLCHAIN_USES_GLIBC=y
BR2_TOOLCHAIN_EXTERNAL=y
BR2_TOOLCHAIN_EXTERNAL_CODESOURCERY_X86_201209=y
BR2_TOOLCHAIN_EXTERNAL_DOWNLOAD=y
BR2_TOOLCHAIN_EXTERNAL_PREFIX="i686-pc-linux-gnu"
BR2_TOOLCHAIN_EXTERNAL_GLIBC=y
BR2_LARGEFILE=y
BR2_INET_IPV6=y
BR2_TOOLCHAIN_HAS_NATIVE_RPC=y
BR2_USE_WCHAR=y
BR2_ENABLE_LOCALE=y
BR2_INSTALL_LIBSTDCPP=y
BR2_TOOLCHAIN_HAS_THREADS=y
BR2_TOOLCHAIN_HAS_THREADS_DEBUG=y
BR2_TOOLCHAIN_HAS_SHADOW_PASSWORDS=y
BR2_GENERATE_LOCALE=""
BR2_USE_MMU=y
BR2_TARGET_OPTIMIZATION="-pipe"
BR2_TARGET_LDFLAGS=""
BR2_TARGET_GENERIC_HOSTNAME="buildroot"
BR2_TARGET_GENERIC_ISSUE="Welcome to Buildroot"
BR2_TARGET_GENERIC_PASSWD_MD5=y
BR2_TARGET_GENERIC_PASSWD_METHOD="md5"
BR2_ROOTFS_DEVICE_CREATION_STATIC=y
BR2_INIT_BUSYBOX=y
BR2_ROOTFS_DEVICE_TABLE="system/device_table.txt"
BR2_ROOTFS_STATIC_DEVICE_TABLE="system/device_table_dev.txt"
BR2_ROOTFS_SKELETON_DEFAULT=y
BR2_TARGET_GENERIC_ROOT_PASSWD=""
BR2_TARGET_GENERIC_GETTY_PORT="ttyS0"
BR2_TARGET_GENERIC_GETTY_BAUDRATE_115200=y
BR2_TARGET_GENERIC_GETTY_BAUDRATE="115200"
BR2_TARGET_GENERIC_GETTY_TERM="vt100"
BR2_TARGET_GENERIC_REMOUNT_ROOTFS_RW=y
BR2_ROOTFS_OVERLAY=""
BR2_ROOTFS_POST_BUILD_SCRIPT=""
BR2_ROOTFS_POST_IMAGE_SCRIPT=""
BR2_PACKAGE_BUSYBOX=y
BR2_BUSYBOX_VERSION_1_21_X=y
BR2_BUSYBOX_VERSION="1.21.1"
BR2_PACKAGE_BUSYBOX_CONFIG="package/busybox/busybox-1.21.x.config"
BR2_PACKAGE_GIT=y
BR2_PACKAGE_ZLIB=y
BR2_TARGET_ROOTFS_TAR=y
BR2_TARGET_ROOTFS_TAR_NONE=y
BR2_TARGET_ROOTFS_TAR_OPTIONS=""


_______________________________________________
buildroot mailing list
buildroot@
http://lists.busybox.net/mailman/listinfo/buildroot





--
View this message in context: http://buildroot-busybox.2317881.n4.nabble.com/Bug-7016-New-Git-issues-in-resulting-buildroot-tar-tp66457p68708.html
Sent from the Buildroot (busybox) mailing list archive at Nabble.com.

[Buildroot] [Bug 7016] Git issues in resulting buildroot tar

[Thomas De Schampheleire]

Hi Brian,

On Fri, May 2, 2014 at 8:14 AM, brianclements <brian@brianclements.net> wrote:
> On Thu, May 1, 2014 at 10:26 PM, brianclements &lt;brian@&gt; wrote:
> [..]
>>
>>>In the menuconfig, in the toolchain menu, you can specify the toolchain
>>> type. It is set to 'Buildroot' by >default, but you can specify
>>> 'external'
>>> here. I then used the default proposed toolchain (Sourcery)
>>>
>>> So I've found the fix. Selecting the external toolchain "Sourcery" (and
>>> WOW
>>> is it much faster) with my other selections same as before of x86_64 and
>>> git, was all I needed to get git to work. So I know what I need to do now
>>> to
>>> get it to work for my uses. But I'm sure that still means there is a bug
>>> somewhere in the buildroot toolchain. I'm not sure how to go about
>>> finding
>>> it for you, but I'd be more then happy to test and reproduce things along
>>> the way for you if you desire. Let me know,
>
>>Could you send me the buildroot config you were using before?
>>I will build the exact same config this time and see if I can reproduce on
> qemu.
>
> I wasn't sure which you wanted, so I included both. Both were created with
> the command `make menuconfig` and this is with comments and whitespace
> removed.
>

I have now built a system that should be pretty much the same as the
one you mentioned.
The git clone works fine on that rootfs using qemu.

The .config is attached.
I do notice a difference in buildroot baseline: I'm running on top of
buildroot git (recent version) while you are using probably 2014.02. I
don't really think this matters though.

One relevant difference between the external toolchain and the
internal one, is the libc library. The external toolchain is using
glibc, while the internal one is uclibc.
My initial assumption was that your using docker to run this rootfs is
similar to running a rootfs directly on the host system, and that it
is not completely isolated. To be honest, I still think this could be
the case. Are you sure that no single userspace library from the host
system (in particular the c library) is used in docker?

What I suggest is that you build the system with my .config file too,
and see if it works or not in docker.
I would also suggest you try qemu using the exact same system. Setting
it up isn't hard, really.
If you're using a Debian-based system, you can simply apt-get qemu-system.
Then, from the buildroot root:
qemu-system-x86_64 -M pc -kernel output/images/bzImage -drive
file=output/images/rootfs.ext2,if=ide -append root=/dev/sda -net
nic,model=rtl8139 -net user

In the system, login as root, run the dhcp client (udhcpc or dhcpcd),
update /etc/resolv.conf with a nameserver, e.g. 'nameserver 8.8.8.8',
and then run the git clone.

Best regards,
Thomas
-------------- next part --------------
A non-text attachment was scrubbed...
Name: .config
Type: application/octet-stream
Size: 41323 bytes
Desc: not available
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20140505/34623916/attachment-0001.obj>

[Buildroot] [Bug 7016] Git issues in resulting buildroot tar

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=7016

Thomas De Schampheleire <patrickdepinguin+buildroot@gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|---                         |2014.05

-- 
Configure bugmail: https://bugs.busybox.net/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Buildroot] [Bug 7016] Git issues in resulting buildroot tar

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=7016

Thomas De Schampheleire <patrickdepinguin@gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |WORKSFORME

--- Comment #2 from Thomas De Schampheleire <patrickdepinguin@gmail.com> 2014-05-14 08:13:47 UTC ---
Investigation was discussed further on the mailing list, see
http://lists.busybox.net/pipermail/buildroot/2014-April/093307.html
http://lists.busybox.net/pipermail/buildroot/2014-May/095513.html

The issue could not be reproduced using the same configuration in qemu.
My assumption is that the way the rootfs is run through docker is causing
trouble, and thus that it isn't a buildroot problem.
Therefore, I'm going to close this bug.

Brian: do not hesitate to continue the discussion on the mailing list, or to
reopen this bug if you think it effectively is a buildroot bug.

Thanks.

-- 
Configure bugmail: https://bugs.busybox.net/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.