* [PATCH] http.postbuffer: make a size_t
@ 2017-03-30 20:29 David Turner
2017-03-30 20:42 ` Shawn Pearce
` (2 more replies)
0 siblings, 3 replies; 6+ messages in thread
From: David Turner @ 2017-03-30 20:29 UTC (permalink / raw)
To: git; +Cc: David Turner
Unfortunately, in order to push some large repos, the http postbuffer
must sometimes exceed two gigabytes. On a 64-bit system, this is OK:
we just malloc a larger buffer.
Signed-off-by: David Turner <dturner@twosigma.com>
---
cache.h | 1 +
config.c | 17 +++++++++++++++++
http.c | 2 +-
3 files changed, 19 insertions(+), 1 deletion(-)
diff --git a/cache.h b/cache.h
index fbdf7a815a..a8c1b65db0 100644
--- a/cache.h
+++ b/cache.h
@@ -1900,6 +1900,7 @@ extern int git_parse_maybe_bool(const char *);
extern int git_config_int(const char *, const char *);
extern int64_t git_config_int64(const char *, const char *);
extern unsigned long git_config_ulong(const char *, const char *);
+extern size_t git_config_size_t(const char *, const char *);
extern int git_config_bool_or_int(const char *, const char *, int *);
extern int git_config_bool(const char *, const char *);
extern int git_config_maybe_bool(const char *, const char *);
diff --git a/config.c b/config.c
index 1a4d85537b..7b706cf27a 100644
--- a/config.c
+++ b/config.c
@@ -834,6 +834,15 @@ int git_parse_ulong(const char *value, unsigned long *ret)
return 1;
}
+static size_t git_parse_size_t(const char *value, unsigned long *ret)
+{
+ size_t tmp;
+ if (!git_parse_signed(value, &tmp, maximum_unsigned_value_of_type(size_t)))
+ return 0;
+ *ret = tmp;
+ return 1;
+}
+
NORETURN
static void die_bad_number(const char *name, const char *value)
{
@@ -892,6 +901,14 @@ unsigned long git_config_ulong(const char *name, const char *value)
return ret;
}
+size_t git_config_size_t(const char *name, const char *value)
+{
+ unsigned long ret;
+ if (!git_parse_size_t(value, &ret))
+ die_bad_number(name, value);
+ return ret;
+}
+
int git_parse_maybe_bool(const char *value)
{
if (!value)
diff --git a/http.c b/http.c
index 96d84bbed3..ab6080835f 100644
--- a/http.c
+++ b/http.c
@@ -331,7 +331,7 @@ static int http_options(const char *var, const char *value, void *cb)
}
if (!strcmp("http.postbuffer", var)) {
- http_post_buffer = git_config_int(var, value);
+ http_post_buffer = git_config_size_t(var, value);
if (http_post_buffer < LARGE_PACKET_MAX)
http_post_buffer = LARGE_PACKET_MAX;
return 0;
--
2.11.GIT
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH] http.postbuffer: make a size_t
2017-03-30 20:29 [PATCH] http.postbuffer: make a size_t David Turner
@ 2017-03-30 20:42 ` Shawn Pearce
2017-03-30 20:59 ` David Turner
2017-03-30 22:07 ` Junio C Hamano
2017-03-31 4:22 ` Torsten Bögershausen
2 siblings, 1 reply; 6+ messages in thread
From: Shawn Pearce @ 2017-03-30 20:42 UTC (permalink / raw)
To: David Turner; +Cc: git
On Thu, Mar 30, 2017 at 1:29 PM, David Turner <dturner@twosigma.com> wrote:
> Unfortunately, in order to push some large repos, the http postbuffer
> must sometimes exceed two gigabytes. On a 64-bit system, this is OK:
> we just malloc a larger buffer.
I'm slightly curious what server you are pushing to that needs the
entire thing buffered to compute a Content-Length, rather than using
Transfer-Encoding: chunked. Most Git-over-HTTP should be accepting
Transfer-Encoding: chunked when the stream exceeds postBuffer.
^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: [PATCH] http.postbuffer: make a size_t
2017-03-30 20:42 ` Shawn Pearce
@ 2017-03-30 20:59 ` David Turner
0 siblings, 0 replies; 6+ messages in thread
From: David Turner @ 2017-03-30 20:59 UTC (permalink / raw)
To: Shawn Pearce; +Cc: git
GitLab. I can't speak to our particular configuration of it -- but if you have a specific question about what the config is, I can ask our gitlab admins.
> -----Original Message-----
> From: Shawn Pearce [mailto:spearce@spearce.org]
> Sent: Thursday, March 30, 2017 4:42 PM
> To: David Turner <David.Turner@twosigma.com>
> Cc: git <git@vger.kernel.org>
> Subject: Re: [PATCH] http.postbuffer: make a size_t
>
> On Thu, Mar 30, 2017 at 1:29 PM, David Turner <dturner@twosigma.com>
> wrote:
> > Unfortunately, in order to push some large repos, the http postbuffer
> > must sometimes exceed two gigabytes. On a 64-bit system, this is OK:
> > we just malloc a larger buffer.
>
> I'm slightly curious what server you are pushing to that needs the entire thing
> buffered to compute a Content-Length, rather than using
> Transfer-Encoding: chunked. Most Git-over-HTTP should be accepting
> Transfer-Encoding: chunked when the stream exceeds postBuffer.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] http.postbuffer: make a size_t
2017-03-30 20:29 [PATCH] http.postbuffer: make a size_t David Turner
2017-03-30 20:42 ` Shawn Pearce
@ 2017-03-30 22:07 ` Junio C Hamano
2017-03-31 4:22 ` Torsten Bögershausen
2 siblings, 0 replies; 6+ messages in thread
From: Junio C Hamano @ 2017-03-30 22:07 UTC (permalink / raw)
To: David Turner; +Cc: git
David Turner <dturner@twosigma.com> writes:
> Unfortunately, in order to push some large repos, the http postbuffer
> must sometimes exceed two gigabytes. On a 64-bit system, this is OK:
> we just malloc a larger buffer.
>
> Signed-off-by: David Turner <dturner@twosigma.com>
> ---
> cache.h | 1 +
> config.c | 17 +++++++++++++++++
> http.c | 2 +-
> 3 files changed, 19 insertions(+), 1 deletion(-)
>
> diff --git a/cache.h b/cache.h
> index fbdf7a815a..a8c1b65db0 100644
> --- a/cache.h
> +++ b/cache.h
> @@ -1900,6 +1900,7 @@ extern int git_parse_maybe_bool(const char *);
> extern int git_config_int(const char *, const char *);
> extern int64_t git_config_int64(const char *, const char *);
> extern unsigned long git_config_ulong(const char *, const char *);
> +extern size_t git_config_size_t(const char *, const char *);
> extern int git_config_bool_or_int(const char *, const char *, int *);
> extern int git_config_bool(const char *, const char *);
> extern int git_config_maybe_bool(const char *, const char *);
> diff --git a/config.c b/config.c
> index 1a4d85537b..7b706cf27a 100644
> --- a/config.c
> +++ b/config.c
> @@ -834,6 +834,15 @@ int git_parse_ulong(const char *value, unsigned long *ret)
> return 1;
> }
>
> +static size_t git_parse_size_t(const char *value, unsigned long *ret)
> +{
> + size_t tmp;
> + if (!git_parse_signed(value, &tmp, maximum_unsigned_value_of_type(size_t)))
I am getting these:
config.c:840:2: error: pointer targets in passing argument 2 of 'git_parse_signed' differ in signedness [-Werror=pointer-sign]
if (!git_parse_signed(value, &tmp, maximum_unsigned_value_of_type(size_t)))
^
config.c:753:12: note: expected 'intmax_t *' but argument is of type 'size_t *'
static int git_parse_signed(const char *value, intmax_t *ret, intmax_t max)
^
Changing "size_t tmp" to "intmax_t tmp" squelches it but the maximum
unsigned value of size_t type would probably overflow "intmax_t max"
which is signed, so...
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] http.postbuffer: make a size_t
2017-03-30 20:29 [PATCH] http.postbuffer: make a size_t David Turner
2017-03-30 20:42 ` Shawn Pearce
2017-03-30 22:07 ` Junio C Hamano
@ 2017-03-31 4:22 ` Torsten Bögershausen
2017-03-31 15:52 ` David Turner
2 siblings, 1 reply; 6+ messages in thread
From: Torsten Bögershausen @ 2017-03-31 4:22 UTC (permalink / raw)
To: David Turner, git
On 30/03/17 22:29, David Turner wrote:
> Unfortunately, in order to push some large repos, the http postbuffer
> must sometimes exceed two gigabytes. On a 64-bit system, this is OK:
> we just malloc a larger buffer.
>
> Signed-off-by: David Turner <dturner@twosigma.com>
> ---
> cache.h | 1 +
> config.c | 17 +++++++++++++++++
> http.c | 2 +-
> 3 files changed, 19 insertions(+), 1 deletion(-)
>
> diff --git a/cache.h b/cache.h
> index fbdf7a815a..a8c1b65db0 100644
> --- a/cache.h
> +++ b/cache.h
> @@ -1900,6 +1900,7 @@ extern int git_parse_maybe_bool(const char *);
> extern int git_config_int(const char *, const char *);
> extern int64_t git_config_int64(const char *, const char *);
> extern unsigned long git_config_ulong(const char *, const char *);
> +extern size_t git_config_size_t(const char *, const char *);
> extern int git_config_bool_or_int(const char *, const char *, int *);
> extern int git_config_bool(const char *, const char *);
> extern int git_config_maybe_bool(const char *, const char *);
> diff --git a/config.c b/config.c
> index 1a4d85537b..7b706cf27a 100644
> --- a/config.c
> +++ b/config.c
> @@ -834,6 +834,15 @@ int git_parse_ulong(const char *value, unsigned long *ret)
> return 1;
> }
>
> +static size_t git_parse_size_t(const char *value, unsigned long *ret)
> +{
> + size_t tmp;
> + if (!git_parse_signed(value, &tmp, maximum_unsigned_value_of_type(size_t)))
> + return 0;
> + *ret = tmp;
> + return 1;
> +}
What is the return value here ?
Isn't it a size_t we want ?
(There was a recent discussion about "unsigned long" vs "size_t", which
are the same on many systems, but not under Win64)
Would the following work ?
static int git_parse_size_t(const char *value, size_t *ret)
{
if (!git_parse_signed(value, ret, maximum_unsigned_value_of_type(size_t)))
return 0;
return 1;
}
[]
> +size_t git_config_size_t(const char *name, const char *value)
> +{
> + unsigned long ret;
> + if (!git_parse_size_t(value, &ret))
> + die_bad_number(name, value);
> + return ret;
> +}
> +
Same here:
size_t git_config_size_t(const char *name, const char *value)
{
size_t ret;
if (!git_parse_size_t(value, &ret))
die_bad_number(name, value);
return ret;
}
^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: [PATCH] http.postbuffer: make a size_t
2017-03-31 4:22 ` Torsten Bögershausen
@ 2017-03-31 15:52 ` David Turner
0 siblings, 0 replies; 6+ messages in thread
From: David Turner @ 2017-03-31 15:52 UTC (permalink / raw)
To: Torsten Bögershausen; +Cc: git
> -----Original Message-----
> From: Torsten Bögershausen [mailto:tboegi@web.de]
> Sent: Friday, March 31, 2017 12:22 AM
> To: David Turner <David.Turner@twosigma.com>; git@vger.kernel.org
> Subject: Re: [PATCH] http.postbuffer: make a size_t
>
>
>
> On 30/03/17 22:29, David Turner wrote:
> > Unfortunately, in order to push some large repos, the http postbuffer
> > must sometimes exceed two gigabytes. On a 64-bit system, this is OK:
> > we just malloc a larger buffer.
> >
> > Signed-off-by: David Turner <dturner@twosigma.com>
> > ---
> > cache.h | 1 +
> > config.c | 17 +++++++++++++++++
> > http.c | 2 +-
> > 3 files changed, 19 insertions(+), 1 deletion(-)
> >
> > diff --git a/cache.h b/cache.h
> > index fbdf7a815a..a8c1b65db0 100644
> > --- a/cache.h
> > +++ b/cache.h
> > @@ -1900,6 +1900,7 @@ extern int git_parse_maybe_bool(const char *);
> > extern int git_config_int(const char *, const char *); extern int64_t
> > git_config_int64(const char *, const char *); extern unsigned long
> > git_config_ulong(const char *, const char *);
> > +extern size_t git_config_size_t(const char *, const char *);
> > extern int git_config_bool_or_int(const char *, const char *, int *);
> > extern int git_config_bool(const char *, const char *); extern int
> > git_config_maybe_bool(const char *, const char *); diff --git
> > a/config.c b/config.c index 1a4d85537b..7b706cf27a 100644
> > --- a/config.c
> > +++ b/config.c
> > @@ -834,6 +834,15 @@ int git_parse_ulong(const char *value, unsigned
> long *ret)
> > return 1;
> > }
> >
> > +static size_t git_parse_size_t(const char *value, unsigned long *ret)
> > +{
> > + size_t tmp;
> > + if (!git_parse_signed(value, &tmp,
> maximum_unsigned_value_of_type(size_t)))
> > + return 0;
> > + *ret = tmp;
> > + return 1;
> > +}
> What is the return value here ?
> Isn't it a size_t we want ?
Yeah, that should return an int, since it's just "parsed" vs "unparsed".
> (There was a recent discussion about "unsigned long" vs "size_t", which
> are the same on many systems, but not under Win64) Would the following
> work ?
>
> static int git_parse_size_t(const char *value, size_t *ret) {
> if (!git_parse_signed(value, ret,
> maximum_unsigned_value_of_type(size_t)))
> return 0;
> return 1;
> }
>
> []
> > +size_t git_config_size_t(const char *name, const char *value) {
> > + unsigned long ret;
> > + if (!git_parse_size_t(value, &ret))
> > + die_bad_number(name, value);
> > + return ret;
> > +}
> > +
> Same here:
> size_t git_config_size_t(const char *name, const char *value) {
> size_t ret;
> if (!git_parse_size_t(value, &ret))
> die_bad_number(name, value);
> return ret;
> }
In v2 I decided to make this a ssize_t, since it was previously a signed int. I know we're not using negative values right now, but since nobody has 2**63 bytes of ram anyway, it's probably safe to keep it signed to keep our options open.
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2017-03-31 15:57 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-03-30 20:29 [PATCH] http.postbuffer: make a size_t David Turner
2017-03-30 20:42 ` Shawn Pearce
2017-03-30 20:59 ` David Turner
2017-03-30 22:07 ` Junio C Hamano
2017-03-31 4:22 ` Torsten Bögershausen
2017-03-31 15:52 ` David Turner
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.