[U-Boot] [ANN] git server, FTP server

[U-Boot] [ANN] git server, FTP server

[Wolfgang Denk]

Hello all,

we're planning some reorganization / updates for our servers:

1. We will move all git repositories to a new, faster machine.
   In this process, the web interface will change (moving from
   gitweb to cgit).  In this process, for a (hopefully) short time,
   access to the git repositories will be impossible.  I will send
   more details when we have decided about the schedule.

2. For some time now, we provide not only the classic FTP server for
   download of the U-Boot release tarballs, but also a public
   directory in the Amazon Cloud Drive [1].  The ACD is supposed to
   provide much better connectivity (especially for non-european
   users) and much higher download rates.

   So far, I have received zero feedback about this.  I would like to
   hear about your experiences (good and bad), and what you think
   about dropping the FTP server completely.

Thanks in advance.

[1] https://www.amazon.com/clouddrive/share/IvfObauv9EsZau4GLr7i6BsLX5Zh8THmHXDiOGSLLkh?ref_=cd_share_link_copy

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
The Buddha, the Godhead, resides quite as comfortably in the circuits
of a digital computer or the gears of a cycle transmission as he does
at the top of a mountain or in the petals of a flower.
            - R.  Pirsig, "Zen and the Art of Motorcycle Maintenance"

[U-Boot] [ANN] git server, FTP server

[Igor Grinberg]

Hi Wolfgang,

On 07/15/15 10:24, Wolfgang Denk wrote:
> Hello all,
> 
> we're planning some reorganization / updates for our servers:
> 
> 1. We will move all git repositories to a new, faster machine.
>    In this process, the web interface will change (moving from
>    gitweb to cgit).  In this process, for a (hopefully) short time,
>    access to the git repositories will be impossible.  I will send
>    more details when we have decided about the schedule.
> 
> 2. For some time now, we provide not only the classic FTP server for
>    download of the U-Boot release tarballs, but also a public
>    directory in the Amazon Cloud Drive [1].  The ACD is supposed to
>    provide much better connectivity (especially for non-european
>    users) and much higher download rates.
> 
>    So far, I have received zero feedback about this.  I would like to
>    hear about your experiences (good and bad), and what you think
>    about dropping the FTP server completely.

Seems very reasonable.
We came to the same conclusion recently for some of our needs.

> 
> Thanks in advance.
> 
> [1] https://www.amazon.com/clouddrive/share/IvfObauv9EsZau4GLr7i6BsLX5Zh8THmHXDiOGSLLkh?ref_=cd_share_link_copy
> 

Works great, although all the files are relatively small (<10MB).
Anyway, it took about ~5-8 seconds to download the u-boot-2015.07.tar.bz2.


-- 
Regards,
Igor.

[U-Boot] [ANN] git server, FTP server

[Bin Meng]

Hi Wolfgang,

On Wed, Jul 15, 2015 at 3:24 PM, Wolfgang Denk <wd@denx.de> wrote:
> Hello all,
>
> we're planning some reorganization / updates for our servers:
>
> 1. We will move all git repositories to a new, faster machine.
>    In this process, the web interface will change (moving from
>    gitweb to cgit).  In this process, for a (hopefully) short time,
>    access to the git repositories will be impossible.  I will send
>    more details when we have decided about the schedule.
>

Is the U-Boot mailing list hosted on the same machine as git? Will
this move also fix the mailing list occasional hiccup issue?

> 2. For some time now, we provide not only the classic FTP server for
>    download of the U-Boot release tarballs, but also a public
>    directory in the Amazon Cloud Drive [1].  The ACD is supposed to
>    provide much better connectivity (especially for non-european
>    users) and much higher download rates.
>
>    So far, I have received zero feedback about this.  I would like to
>    hear about your experiences (good and bad), and what you think
>    about dropping the FTP server completely.
>
> Thanks in advance.
>
> [1] https://www.amazon.com/clouddrive/share/IvfObauv9EsZau4GLr7i6BsLX5Zh8THmHXDiOGSLLkh?ref_=cd_share_link_copy
>

I just downloaded several tarballs from ACD. Looks the download speed
is OK for me (average 200KB/s).

Regards,
Bin

[U-Boot] [ANN] git server, FTP server

[Andy Pont]

Wolfgang wrote...

{snip}

> 2. For some time now, we provide not only the classic FTP server for
>    download of the U-Boot release tarballs, but also a public
>    directory in the Amazon Cloud Drive [1].  The ACD is supposed to
>    provide much better connectivity (especially for non-european
>    users) and much higher download rates.
> 
>    So far, I have received zero feedback about this.  I would like to
>    hear about your experiences (good and bad), and what you think
>    about dropping the FTP server completely.

Today it took about 5 seconds of manual effort to copy the classic FTP URL
from the announce email and use it with wget which then took just under 6
seconds to download it.

So far, two attempts at using the Amazon Cloud web interface have failed.
Clicking on the "Download" button result in the message "Download starting"
but no file appears.  It is probably just Chrome being a PITA but I know
which one currently gets my vote!

Andy.

[U-Boot] [ANN] git server, FTP server

[thomas.langer at lantiq.com]

Hello Wolfgang,

Wolfgang Denk wrote on?2015-07-15:

> 2. For some time now, we provide not only the classic FTP server for
>    download of the U-Boot release tarballs, but also a public
>    directory in the Amazon Cloud Drive [1].  The ACD is supposed to
>    provide much better connectivity (especially for non-european
>    users) and much higher download rates.
>    
>    So far, I have received zero feedback about this.  I would like to
>    hear about your experiences (good and bad), and what you think
>    about dropping the FTP server completely.

In addition to the mail from Andy:

I tried to find a way to download a file with wget or a similar tool, which 
would be used by a distribution builder (like Yocto, Buildroot, OpenWrt, ...).
Up to now I failed, as the download button depends on javascript and the 
generated download link for the file [1] has no indication of the file name
(and because of "templink" inside I am also not sure, how constant this would be).

If somebody can provide a way to download the tarballs by name from a 
command line I would accept the ACD as a replacement.
Otherwise I would prefer a server (like ftp) with some kind of stable URLs.

Best Regards,
Thomas

[1] https://content-na.drive.amazonaws.com/cdproxy/templink/No5u9y7HLlaVIEwnz3VGucNfORmh8RrskPQGcuDGFkkE0Xnc3?download=true

[U-Boot] [ANN] git server, FTP server

[Wolfgang Denk]

Dear Bin,

In message <CAEUhbmUXS5DE_un0fvJo+cAwALFP-1Gu1J+Ra_wpLypJfO0e_Q@mail.gmail.com> you wrote:
> 
> Is the U-Boot mailing list hosted on the same machine as git? Will
> this move also fix the mailing list occasional hiccup issue?

Yes, at the moment this is on the same, old and slow machine.
We will move services one by one, the mailing lists are next after
git.  And yes, this is also supposed to help with the mailing list
issues.

> > [1] https://www.amazon.com/clouddrive/share/IvfObauv9EsZau4GLr7i6BsLX5Zh8THmHXDiOGSLLkh?ref_=cd_share_link_copy
> 
> I just downloaded several tarballs from ACD. Looks the download speed
> is OK for me (average 200KB/s).

Ouch, that's awfully slow.  Are you sure the bottleneck is on the ACD
side?  How fast is your link usually?

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
"I find this a nice feature but it is not according to  the  documen-
tation. Or is it a BUG?"   "Let's call it an accidental feature. :-)"
                       - Larry Wall in <6909@jpl-devvax.JPL.NASA.GOV>

[U-Boot] [ANN] git server, FTP server

[Wolfgang Denk]

Dear Thomas,

In message <593AEF6C47F46446852B067021A273D6FBC788CC@MUCSE037.lantiq.com> you wrote:
> 
> I tried to find a way to download a file with wget or a similar tool, which  
> would be used by a distribution builder (like Yocto, Buildroot, OpenWrt, ...).

Why would any build environment use tarballs?  can you not just
reference the git repository?  This is much more efficient, IMHO.

> Up to now I failed, as the download button depends on javascript and the 
> generated download link for the file [1] has no indication of the file name
> (and because of "templink" inside I am also not sure, how constant this would be).

Sorry, I cannot say much about this.

> If somebody can provide a way to download the tarballs by name from a 
> command line I would accept the ACD as a replacement.

There is tools like acd_cli.py (on github) which also allow a FUSE
mount on ACD data, so there are other options for fancy uses.  But I
don't know if (or how) these would work with shared data.

> Otherwise I would prefer a server (like ftp) with some kind of stable URLs.

Well, the ACD URLs _are_ stable, just not really conveniend to use
from the command line.

Thanks for your input.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
That Microsoft, the Trabant of the operating  system  world,  may  be
glancing  over the Berlin Wall at the Audis and BMWs and Mercedes. In
their own universe Trabants and Ladas were mainstream too...
                                                   -- Evan Leibovitch

[U-Boot] [ANN] git server, FTP server

[Bin Meng]

Hi Wolfgang,

On Wed, Jul 15, 2015 at 8:35 PM, Wolfgang Denk <wd@denx.de> wrote:
> Dear Bin,
>
> In message <CAEUhbmUXS5DE_un0fvJo+cAwALFP-1Gu1J+Ra_wpLypJfO0e_Q@mail.gmail.com> you wrote:
>>
>> Is the U-Boot mailing list hosted on the same machine as git? Will
>> this move also fix the mailing list occasional hiccup issue?
>
> Yes, at the moment this is on the same, old and slow machine.
> We will move services one by one, the mailing lists are next after
> git.  And yes, this is also supposed to help with the mailing list
> issues.
>
>> > [1] https://www.amazon.com/clouddrive/share/IvfObauv9EsZau4GLr7i6BsLX5Zh8THmHXDiOGSLLkh?ref_=cd_share_link_copy
>>
>> I just downloaded several tarballs from ACD. Looks the download speed
>> is OK for me (average 200KB/s).
>
> Ouch, that's awfully slow.  Are you sure the bottleneck is on the ACD
> side?  How fast is your link usually?
>

I have a 15Mbps bandwidth link to the internet. Is the ACD supposed to
choose wherever the nearest download site to the downloader for better
speed? The reason that I felt it is OK for me, is that for the
original ftp.denx.de I could only get average 30KB/s when loading a
tarball. Thus I am thankful for the new ACD :)

Regards,
Bin

[U-Boot] [ANN] git server, FTP server

[Wolfgang Denk]

Dear Bin,

In message <CAEUhbmUYatxaYtbBAaKWrxNopaNfxJiXchw-3Y_NP1FxZ9RWjw@mail.gmail.com> you wrote:
> 
> >> I just downloaded several tarballs from ACD. Looks the download speed
> >> is OK for me (average 200KB/s).
> >
> > Ouch, that's awfully slow.  Are you sure the bottleneck is on the ACD
> > side?  How fast is your link usually?
> 
> I have a 15Mbps bandwidth link to the internet. Is the ACD supposed to
> choose wherever the nearest download site to the downloader for better
> speed?

Yes.

Hm, other prople have reported (ok - for bigger files, say 4+ GiB)
download rates of 11 MiB/s (MegaBYTES per second) on a 100 Mbps
fibre link, which is close to the theoretical limit there.  So you
should get better rates, I think.

> The reason that I felt it is OK for me, is that for the
> original ftp.denx.de I could only get average 30KB/s when loading a
> tarball. Thus I am thankful for the new ACD :)

I'm glad if it helps, even though I can't explain why you don't get
the full bandwidth.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
"It may be that our role on this planet is not to worship God but  to
create him."                                       - Arthur C. Clarke

[U-Boot] [ANN] git server, FTP server

[Andreas Färber]

Dear Wolfgang,

Am 15.07.2015 um 14:51 schrieb Wolfgang Denk:
> In message <593AEF6C47F46446852B067021A273D6FBC788CC@MUCSE037.lantiq.com> you wrote:
>>
>> I tried to find a way to download a file with wget or a similar tool, which  
>> would be used by a distribution builder (like Yocto, Buildroot, OpenWrt, ...).
> 
> Why would any build environment use tarballs?  can you not just
> reference the git repository?  This is much more efficient, IMHO.

Open Build Service uses tarballs, too. We usually specify full URLs in
the RPM .spec file [1] both for documentation of origin and so that the
mirrored tarball can be verified via gpg or checksum where available.
As the actual build is intentionally done offline, at some point a
tarball is needed. The URL could happily be a redirect to some mirror.

When developing, Git is great of course.

Regards,
Andreas

[1]
https://build.opensuse.org/package/view_file/Base:System/u-boot/u-boot.spec.in?expand=1

-- 
SUSE Linux GmbH, Maxfeldstr. 5, 90409 N?rnberg, Germany
GF: Felix Imend?rffer, Jane Smithard, Dilip Upmanyu, Graham Norton; HRB
21284 (AG N?rnberg)

[U-Boot] [ANN] git server, FTP server

[Peter Robinson]

>>> I tried to find a way to download a file with wget or a similar tool, which
>>> would be used by a distribution builder (like Yocto, Buildroot, OpenWrt, ...).
>>
>> Why would any build environment use tarballs?  can you not just
>> reference the git repository?  This is much more efficient, IMHO.
>
> Open Build Service uses tarballs, too. We usually specify full URLs in
> the RPM .spec file [1] both for documentation of origin and so that the
> mirrored tarball can be verified via gpg or checksum where available.
> As the actual build is intentionally done offline, at some point a
> tarball is needed. The URL could happily be a redirect to some mirror.

This is the same in Fedora, the builds are done in a constrained
environment without connectivity to the general internet.

Peter

[U-Boot] [ANN] git server, FTP server

[Wolfgang Denk]

Dear Andreas,

In message <55ABD78C.2060108@suse.de> you wrote:
>
> > Why would any build environment use tarballs?  can you not just
> > reference the git repository?  This is much more efficient, IMHO.
> 
> Open Build Service uses tarballs, too. We usually specify full URLs in
> the RPM .spec file [1] both for documentation of origin and so that the
> mirrored tarball can be verified via gpg or checksum where available.

OK, I get the argument.

Well, technically a git repository is better even in this aspect, as
it is inherently self-verifying :-)

But no worries, we will keep FTP if there are people who find it
useful.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
I wrote my name at the top of the page. I wrote down  the  number  of
the  question  ``1''.  After much reflection I put a bracket round it
thus ``(1)''. But thereafter I could not think of anything  connected
with it that was either relevant or true.
                        - Sir Winston Churchill _My Early Life_ ch. 2

[U-Boot] [ANN] git server, FTP server

[Wolfgang Denk]

Dear Peter,

In message <CALeDE9NJiqCaE3zhdrS80Fe_P4bfxCG2naW8YhzrHM+WqSHWfw@mail.gmail.com> you wrote:
> >>> I tried to find a way to download a file with wget or a similar tool, which
> >>> would be used by a distribution builder (like Yocto, Buildroot, OpenWrt, ...).
> >>
> >> Why would any build environment use tarballs?  can you not just
> >> reference the git repository?  This is much more efficient, IMHO.
> >
> > Open Build Service uses tarballs, too. We usually specify full URLs in
> > the RPM .spec file [1] both for documentation of origin and so that the
> > mirrored tarball can be verified via gpg or checksum where available.
> > As the actual build is intentionally done offline, at some point a
> > tarball is needed. The URL could happily be a redirect to some mirror.
> 
> This is the same in Fedora, the builds are done in a constrained
> environment without connectivity to the general internet.

This is normal.  It's the same in any Yocto based environment.  And
probably in any other buil environment that alloes to reliably
reproduce a specific build.  But you can still use git to fetch your
sources - instead of storing a tarball locally, you store a clone of
the git repo.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
1000 pains                                              = 1 Megahertz