* [Qemu-devel] [PATCH 0/2] migration/qemu-file: cleanup and refine qemu-file
@ 2019-07-31 14:42 Wei Yang
2019-07-31 14:42 ` [Qemu-devel] [PATCH 1/2] migration/qemu-file: remove check on writev_buffer in qemu_put_compression_data Wei Yang
` (2 more replies)
0 siblings, 3 replies; 13+ messages in thread
From: Wei Yang @ 2019-07-31 14:42 UTC (permalink / raw)
To: qemu-devel; +Cc: Wei Yang, dgilbert, quintela
Two cleanup:
Patch #1 make code consistent on calling add_to_iovec
Patch #2 refine the code to handle the case when buf already flushed
Wei Yang (2):
migration/qemu-file: remove check on writev_buffer in
qemu_put_compression_data
migration/qemu-file: fix potential buf waste for extra buf_index
adjustment
migration/qemu-file.c | 44 +++++++++++++++++++++++++++----------------
1 file changed, 28 insertions(+), 16 deletions(-)
--
2.17.1
^ permalink raw reply [flat|nested] 13+ messages in thread
* [Qemu-devel] [PATCH 1/2] migration/qemu-file: remove check on writev_buffer in qemu_put_compression_data
2019-07-31 14:42 [Qemu-devel] [PATCH 0/2] migration/qemu-file: cleanup and refine qemu-file Wei Yang
@ 2019-07-31 14:42 ` Wei Yang
2019-08-23 10:30 ` Dr. David Alan Gilbert
2019-07-31 14:42 ` [Qemu-devel] [PATCH 2/2] migration/qemu-file: fix potential buf waste for extra buf_index adjustment Wei Yang
2019-08-19 2:36 ` [Qemu-devel] [PATCH 0/2] migration/qemu-file: cleanup and refine qemu-file Wei Yang
2 siblings, 1 reply; 13+ messages in thread
From: Wei Yang @ 2019-07-31 14:42 UTC (permalink / raw)
To: qemu-devel; +Cc: Wei Yang, dgilbert, quintela
The check of writev_buffer is in qemu_fflush, which means it is not
harmful if it is NULL.
And removing it will make the code consistent since all other
add_to_iovec() is called without the check.
Signed-off-by: Wei Yang <richardw.yang@linux.intel.com>
---
migration/qemu-file.c | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git a/migration/qemu-file.c b/migration/qemu-file.c
index 0431585502..35c22605dd 100644
--- a/migration/qemu-file.c
+++ b/migration/qemu-file.c
@@ -717,9 +717,7 @@ ssize_t qemu_put_compression_data(QEMUFile *f, z_stream *stream,
}
qemu_put_be32(f, blen);
- if (f->ops->writev_buffer) {
- add_to_iovec(f, f->buf + f->buf_index, blen, false);
- }
+ add_to_iovec(f, f->buf + f->buf_index, blen, false);
f->buf_index += blen;
if (f->buf_index == IO_BUF_SIZE) {
qemu_fflush(f);
--
2.17.1
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [Qemu-devel] [PATCH 2/2] migration/qemu-file: fix potential buf waste for extra buf_index adjustment
2019-07-31 14:42 [Qemu-devel] [PATCH 0/2] migration/qemu-file: cleanup and refine qemu-file Wei Yang
2019-07-31 14:42 ` [Qemu-devel] [PATCH 1/2] migration/qemu-file: remove check on writev_buffer in qemu_put_compression_data Wei Yang
@ 2019-07-31 14:42 ` Wei Yang
2019-08-23 11:06 ` Dr. David Alan Gilbert
2019-08-19 2:36 ` [Qemu-devel] [PATCH 0/2] migration/qemu-file: cleanup and refine qemu-file Wei Yang
2 siblings, 1 reply; 13+ messages in thread
From: Wei Yang @ 2019-07-31 14:42 UTC (permalink / raw)
To: qemu-devel; +Cc: Wei Yang, dgilbert, quintela
In add_to_iovec(), qemu_fflush() will be called if iovec is full. If
this happens, buf_index is reset. Currently, this is not checked and
buf_index would always been adjust with buf size.
This is not harmful, but will waste some space in file buffer.
This patch make add_to_iovec() return 1 when it has flushed the file.
Then the caller could check the return value to see whether it is
necessary to adjust the buf_index any more.
Signed-off-by: Wei Yang <richardw.yang@linux.intel.com>
---
migration/qemu-file.c | 42 ++++++++++++++++++++++++++++--------------
1 file changed, 28 insertions(+), 14 deletions(-)
diff --git a/migration/qemu-file.c b/migration/qemu-file.c
index 35c22605dd..05d9f42ddb 100644
--- a/migration/qemu-file.c
+++ b/migration/qemu-file.c
@@ -343,8 +343,16 @@ int qemu_fclose(QEMUFile *f)
return ret;
}
-static void add_to_iovec(QEMUFile *f, const uint8_t *buf, size_t size,
- bool may_free)
+/*
+ * Add buf to iovec. Do flush if iovec is full.
+ *
+ * Return values:
+ * 1 iovec is full and flushed
+ * 0 iovec is not flushed
+ *
+ */
+static int add_to_iovec(QEMUFile *f, const uint8_t *buf, size_t size,
+ bool may_free)
{
/* check for adjacent buffer and coalesce them */
if (f->iovcnt > 0 && buf == f->iov[f->iovcnt - 1].iov_base +
@@ -362,7 +370,10 @@ static void add_to_iovec(QEMUFile *f, const uint8_t *buf, size_t size,
if (f->iovcnt >= MAX_IOV_SIZE) {
qemu_fflush(f);
+ return 1;
}
+
+ return 0;
}
void qemu_put_buffer_async(QEMUFile *f, const uint8_t *buf, size_t size,
@@ -391,10 +402,11 @@ void qemu_put_buffer(QEMUFile *f, const uint8_t *buf, size_t size)
}
memcpy(f->buf + f->buf_index, buf, l);
f->bytes_xfer += l;
- add_to_iovec(f, f->buf + f->buf_index, l, false);
- f->buf_index += l;
- if (f->buf_index == IO_BUF_SIZE) {
- qemu_fflush(f);
+ if (!add_to_iovec(f, f->buf + f->buf_index, l, false)) {
+ f->buf_index += l;
+ if (f->buf_index == IO_BUF_SIZE) {
+ qemu_fflush(f);
+ }
}
if (qemu_file_get_error(f)) {
break;
@@ -412,10 +424,11 @@ void qemu_put_byte(QEMUFile *f, int v)
f->buf[f->buf_index] = v;
f->bytes_xfer++;
- add_to_iovec(f, f->buf + f->buf_index, 1, false);
- f->buf_index++;
- if (f->buf_index == IO_BUF_SIZE) {
- qemu_fflush(f);
+ if (!add_to_iovec(f, f->buf + f->buf_index, 1, false)) {
+ f->buf_index++;
+ if (f->buf_index == IO_BUF_SIZE) {
+ qemu_fflush(f);
+ }
}
}
@@ -717,10 +730,11 @@ ssize_t qemu_put_compression_data(QEMUFile *f, z_stream *stream,
}
qemu_put_be32(f, blen);
- add_to_iovec(f, f->buf + f->buf_index, blen, false);
- f->buf_index += blen;
- if (f->buf_index == IO_BUF_SIZE) {
- qemu_fflush(f);
+ if (!add_to_iovec(f, f->buf + f->buf_index, blen, false)) {
+ f->buf_index += blen;
+ if (f->buf_index == IO_BUF_SIZE) {
+ qemu_fflush(f);
+ }
}
return blen + sizeof(int32_t);
}
--
2.17.1
^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: [Qemu-devel] [PATCH 0/2] migration/qemu-file: cleanup and refine qemu-file
2019-07-31 14:42 [Qemu-devel] [PATCH 0/2] migration/qemu-file: cleanup and refine qemu-file Wei Yang
2019-07-31 14:42 ` [Qemu-devel] [PATCH 1/2] migration/qemu-file: remove check on writev_buffer in qemu_put_compression_data Wei Yang
2019-07-31 14:42 ` [Qemu-devel] [PATCH 2/2] migration/qemu-file: fix potential buf waste for extra buf_index adjustment Wei Yang
@ 2019-08-19 2:36 ` Wei Yang
2 siblings, 0 replies; 13+ messages in thread
From: Wei Yang @ 2019-08-19 2:36 UTC (permalink / raw)
To: Wei Yang; +Cc: qemu-devel, dgilbert, quintela
On Wed, Jul 31, 2019 at 10:42:23PM +0800, Wei Yang wrote:
>Two cleanup:
>
>Patch #1 make code consistent on calling add_to_iovec
>Patch #2 refine the code to handle the case when buf already flushed
>
Ping~
>Wei Yang (2):
> migration/qemu-file: remove check on writev_buffer in
> qemu_put_compression_data
> migration/qemu-file: fix potential buf waste for extra buf_index
> adjustment
>
> migration/qemu-file.c | 44 +++++++++++++++++++++++++++----------------
> 1 file changed, 28 insertions(+), 16 deletions(-)
>
>--
>2.17.1
--
Wei Yang
Help you, Help me
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [Qemu-devel] [PATCH 1/2] migration/qemu-file: remove check on writev_buffer in qemu_put_compression_data
2019-07-31 14:42 ` [Qemu-devel] [PATCH 1/2] migration/qemu-file: remove check on writev_buffer in qemu_put_compression_data Wei Yang
@ 2019-08-23 10:30 ` Dr. David Alan Gilbert
0 siblings, 0 replies; 13+ messages in thread
From: Dr. David Alan Gilbert @ 2019-08-23 10:30 UTC (permalink / raw)
To: Wei Yang; +Cc: qemu-devel, quintela
* Wei Yang (richardw.yang@linux.intel.com) wrote:
> The check of writev_buffer is in qemu_fflush, which means it is not
> harmful if it is NULL.
>
> And removing it will make the code consistent since all other
> add_to_iovec() is called without the check.
>
> Signed-off-by: Wei Yang <richardw.yang@linux.intel.com>
So I think the real reason here is commit 11808bb which removed
support for non-iovec writers.
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
> ---
> migration/qemu-file.c | 4 +---
> 1 file changed, 1 insertion(+), 3 deletions(-)
>
> diff --git a/migration/qemu-file.c b/migration/qemu-file.c
> index 0431585502..35c22605dd 100644
> --- a/migration/qemu-file.c
> +++ b/migration/qemu-file.c
> @@ -717,9 +717,7 @@ ssize_t qemu_put_compression_data(QEMUFile *f, z_stream *stream,
> }
>
> qemu_put_be32(f, blen);
> - if (f->ops->writev_buffer) {
> - add_to_iovec(f, f->buf + f->buf_index, blen, false);
> - }
> + add_to_iovec(f, f->buf + f->buf_index, blen, false);
> f->buf_index += blen;
> if (f->buf_index == IO_BUF_SIZE) {
> qemu_fflush(f);
> --
> 2.17.1
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [Qemu-devel] [PATCH 2/2] migration/qemu-file: fix potential buf waste for extra buf_index adjustment
2019-07-31 14:42 ` [Qemu-devel] [PATCH 2/2] migration/qemu-file: fix potential buf waste for extra buf_index adjustment Wei Yang
@ 2019-08-23 11:06 ` Dr. David Alan Gilbert
2019-08-23 11:38 ` Daniel P. Berrangé
2019-08-24 16:15 ` Wei Yang
0 siblings, 2 replies; 13+ messages in thread
From: Dr. David Alan Gilbert @ 2019-08-23 11:06 UTC (permalink / raw)
To: Wei Yang, berrange; +Cc: qemu-devel, quintela
(Copying Dan in)
* Wei Yang (richardw.yang@linux.intel.com) wrote:
> In add_to_iovec(), qemu_fflush() will be called if iovec is full. If
> this happens, buf_index is reset. Currently, this is not checked and
> buf_index would always been adjust with buf size.
>
> This is not harmful, but will waste some space in file buffer.
That's a nice find.
> This patch make add_to_iovec() return 1 when it has flushed the file.
> Then the caller could check the return value to see whether it is
> necessary to adjust the buf_index any more.
>
> Signed-off-by: Wei Yang <richardw.yang@linux.intel.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
(I wonder if there's a way to wrap that little add_to_iovec, check, add
to index, flush in a little wrapper).
Dave
> ---
> migration/qemu-file.c | 42 ++++++++++++++++++++++++++++--------------
> 1 file changed, 28 insertions(+), 14 deletions(-)
>
> diff --git a/migration/qemu-file.c b/migration/qemu-file.c
> index 35c22605dd..05d9f42ddb 100644
> --- a/migration/qemu-file.c
> +++ b/migration/qemu-file.c
> @@ -343,8 +343,16 @@ int qemu_fclose(QEMUFile *f)
> return ret;
> }
>
> -static void add_to_iovec(QEMUFile *f, const uint8_t *buf, size_t size,
> - bool may_free)
> +/*
> + * Add buf to iovec. Do flush if iovec is full.
> + *
> + * Return values:
> + * 1 iovec is full and flushed
> + * 0 iovec is not flushed
> + *
> + */
> +static int add_to_iovec(QEMUFile *f, const uint8_t *buf, size_t size,
> + bool may_free)
> {
> /* check for adjacent buffer and coalesce them */
> if (f->iovcnt > 0 && buf == f->iov[f->iovcnt - 1].iov_base +
> @@ -362,7 +370,10 @@ static void add_to_iovec(QEMUFile *f, const uint8_t *buf, size_t size,
>
> if (f->iovcnt >= MAX_IOV_SIZE) {
> qemu_fflush(f);
> + return 1;
> }
> +
> + return 0;
> }
>
> void qemu_put_buffer_async(QEMUFile *f, const uint8_t *buf, size_t size,
> @@ -391,10 +402,11 @@ void qemu_put_buffer(QEMUFile *f, const uint8_t *buf, size_t size)
> }
> memcpy(f->buf + f->buf_index, buf, l);
> f->bytes_xfer += l;
> - add_to_iovec(f, f->buf + f->buf_index, l, false);
> - f->buf_index += l;
> - if (f->buf_index == IO_BUF_SIZE) {
> - qemu_fflush(f);
> + if (!add_to_iovec(f, f->buf + f->buf_index, l, false)) {
> + f->buf_index += l;
> + if (f->buf_index == IO_BUF_SIZE) {
> + qemu_fflush(f);
> + }
> }
> if (qemu_file_get_error(f)) {
> break;
> @@ -412,10 +424,11 @@ void qemu_put_byte(QEMUFile *f, int v)
>
> f->buf[f->buf_index] = v;
> f->bytes_xfer++;
> - add_to_iovec(f, f->buf + f->buf_index, 1, false);
> - f->buf_index++;
> - if (f->buf_index == IO_BUF_SIZE) {
> - qemu_fflush(f);
> + if (!add_to_iovec(f, f->buf + f->buf_index, 1, false)) {
> + f->buf_index++;
> + if (f->buf_index == IO_BUF_SIZE) {
> + qemu_fflush(f);
> + }
> }
> }
>
> @@ -717,10 +730,11 @@ ssize_t qemu_put_compression_data(QEMUFile *f, z_stream *stream,
> }
>
> qemu_put_be32(f, blen);
> - add_to_iovec(f, f->buf + f->buf_index, blen, false);
> - f->buf_index += blen;
> - if (f->buf_index == IO_BUF_SIZE) {
> - qemu_fflush(f);
> + if (!add_to_iovec(f, f->buf + f->buf_index, blen, false)) {
> + f->buf_index += blen;
> + if (f->buf_index == IO_BUF_SIZE) {
> + qemu_fflush(f);
> + }
> }
> return blen + sizeof(int32_t);
> }
> --
> 2.17.1
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [Qemu-devel] [PATCH 2/2] migration/qemu-file: fix potential buf waste for extra buf_index adjustment
2019-08-23 11:06 ` Dr. David Alan Gilbert
@ 2019-08-23 11:38 ` Daniel P. Berrangé
2019-08-23 13:05 ` Dr. David Alan Gilbert
2019-08-24 16:15 ` Wei Yang
1 sibling, 1 reply; 13+ messages in thread
From: Daniel P. Berrangé @ 2019-08-23 11:38 UTC (permalink / raw)
To: Dr. David Alan Gilbert; +Cc: quintela, Wei Yang, qemu-devel
On Fri, Aug 23, 2019 at 12:06:09PM +0100, Dr. David Alan Gilbert wrote:
> (Copying Dan in)
>
> * Wei Yang (richardw.yang@linux.intel.com) wrote:
> > In add_to_iovec(), qemu_fflush() will be called if iovec is full. If
> > this happens, buf_index is reset. Currently, this is not checked and
> > buf_index would always been adjust with buf size.
> >
> > This is not harmful, but will waste some space in file buffer.
>
> That's a nice find.
>
> > This patch make add_to_iovec() return 1 when it has flushed the file.
> > Then the caller could check the return value to see whether it is
> > necessary to adjust the buf_index any more.
> >
> > Signed-off-by: Wei Yang <richardw.yang@linux.intel.com>
>
> Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
>
> (I wonder if there's a way to wrap that little add_to_iovec, check, add
> to index, flush in a little wrapper).
Given the name "add_to_iovec" I think it is pretty surprising
that it calls "qemu_flush" at all.
It is also pretty wierd that we're checking two different
conditions in two different places.
Right now the code is essentially doing this:
if (f->iovcnt >= MAX_IOV_SIZE) {
qemu_fflush(f);
}
if (f->buf_index == IO_BUF_SIZE) {
qemu_fflush(f);
}
Except that in the qemu_put_buffer_async() case, we're
only doing the first of these two checks. This feels
very odd indeed - I would have thought either it should
do both, or do neither.
Assuming doing both flushs is ok for qemu_put_buffer_async
then I'd suggest renaming 'add_to_iovec' to 'queue_buffer'
and have that method do both of these qemu_fflush() calls.
> > ---
> > migration/qemu-file.c | 42 ++++++++++++++++++++++++++++--------------
> > 1 file changed, 28 insertions(+), 14 deletions(-)
> >
> > diff --git a/migration/qemu-file.c b/migration/qemu-file.c
> > index 35c22605dd..05d9f42ddb 100644
> > --- a/migration/qemu-file.c
> > +++ b/migration/qemu-file.c
> > @@ -343,8 +343,16 @@ int qemu_fclose(QEMUFile *f)
> > return ret;
> > }
> >
> > -static void add_to_iovec(QEMUFile *f, const uint8_t *buf, size_t size,
> > - bool may_free)
> > +/*
> > + * Add buf to iovec. Do flush if iovec is full.
> > + *
> > + * Return values:
> > + * 1 iovec is full and flushed
> > + * 0 iovec is not flushed
> > + *
> > + */
> > +static int add_to_iovec(QEMUFile *f, const uint8_t *buf, size_t size,
> > + bool may_free)
> > {
> > /* check for adjacent buffer and coalesce them */
> > if (f->iovcnt > 0 && buf == f->iov[f->iovcnt - 1].iov_base +
> > @@ -362,7 +370,10 @@ static void add_to_iovec(QEMUFile *f, const uint8_t *buf, size_t size,
> >
> > if (f->iovcnt >= MAX_IOV_SIZE) {
> > qemu_fflush(f);
> > + return 1;
> > }
> > +
> > + return 0;
> > }
> >
> > void qemu_put_buffer_async(QEMUFile *f, const uint8_t *buf, size_t size,
> > @@ -391,10 +402,11 @@ void qemu_put_buffer(QEMUFile *f, const uint8_t *buf, size_t size)
> > }
> > memcpy(f->buf + f->buf_index, buf, l);
> > f->bytes_xfer += l;
> > - add_to_iovec(f, f->buf + f->buf_index, l, false);
> > - f->buf_index += l;
> > - if (f->buf_index == IO_BUF_SIZE) {
> > - qemu_fflush(f);
> > + if (!add_to_iovec(f, f->buf + f->buf_index, l, false)) {
> > + f->buf_index += l;
> > + if (f->buf_index == IO_BUF_SIZE) {
> > + qemu_fflush(f);
> > + }
> > }
> > if (qemu_file_get_error(f)) {
> > break;
> > @@ -412,10 +424,11 @@ void qemu_put_byte(QEMUFile *f, int v)
> >
> > f->buf[f->buf_index] = v;
> > f->bytes_xfer++;
> > - add_to_iovec(f, f->buf + f->buf_index, 1, false);
> > - f->buf_index++;
> > - if (f->buf_index == IO_BUF_SIZE) {
> > - qemu_fflush(f);
> > + if (!add_to_iovec(f, f->buf + f->buf_index, 1, false)) {
> > + f->buf_index++;
> > + if (f->buf_index == IO_BUF_SIZE) {
> > + qemu_fflush(f);
> > + }
> > }
> > }
> >
> > @@ -717,10 +730,11 @@ ssize_t qemu_put_compression_data(QEMUFile *f, z_stream *stream,
> > }
> >
> > qemu_put_be32(f, blen);
> > - add_to_iovec(f, f->buf + f->buf_index, blen, false);
> > - f->buf_index += blen;
> > - if (f->buf_index == IO_BUF_SIZE) {
> > - qemu_fflush(f);
> > + if (!add_to_iovec(f, f->buf + f->buf_index, blen, false)) {
> > + f->buf_index += blen;
> > + if (f->buf_index == IO_BUF_SIZE) {
> > + qemu_fflush(f);
> > + }
> > }
> > return blen + sizeof(int32_t);
> > }
> > --
> > 2.17.1
> >
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [Qemu-devel] [PATCH 2/2] migration/qemu-file: fix potential buf waste for extra buf_index adjustment
2019-08-23 11:38 ` Daniel P. Berrangé
@ 2019-08-23 13:05 ` Dr. David Alan Gilbert
2019-08-24 16:22 ` Wei Yang
0 siblings, 1 reply; 13+ messages in thread
From: Dr. David Alan Gilbert @ 2019-08-23 13:05 UTC (permalink / raw)
To: Daniel P. Berrangé; +Cc: quintela, Wei Yang, qemu-devel
* Daniel P. Berrangé (berrange@redhat.com) wrote:
> On Fri, Aug 23, 2019 at 12:06:09PM +0100, Dr. David Alan Gilbert wrote:
> > (Copying Dan in)
> >
> > * Wei Yang (richardw.yang@linux.intel.com) wrote:
> > > In add_to_iovec(), qemu_fflush() will be called if iovec is full. If
> > > this happens, buf_index is reset. Currently, this is not checked and
> > > buf_index would always been adjust with buf size.
> > >
> > > This is not harmful, but will waste some space in file buffer.
> >
> > That's a nice find.
> >
> > > This patch make add_to_iovec() return 1 when it has flushed the file.
> > > Then the caller could check the return value to see whether it is
> > > necessary to adjust the buf_index any more.
> > >
> > > Signed-off-by: Wei Yang <richardw.yang@linux.intel.com>
> >
> > Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
> >
> > (I wonder if there's a way to wrap that little add_to_iovec, check, add
> > to index, flush in a little wrapper).
>
> Given the name "add_to_iovec" I think it is pretty surprising
> that it calls "qemu_flush" at all.
>
> It is also pretty wierd that we're checking two different
> conditions in two different places.
>
> Right now the code is essentially doing this:
>
> if (f->iovcnt >= MAX_IOV_SIZE) {
> qemu_fflush(f);
> }
> if (f->buf_index == IO_BUF_SIZE) {
> qemu_fflush(f);
> }
>
> Except that in the qemu_put_buffer_async() case, we're
> only doing the first of these two checks. This feels
> very odd indeed - I would have thought either it should
> do both, or do neither.
No, there's two separate types of buffers.
There's f->buf which is a single allocated buffer in the QEMUFile
with an offset buf_index, and there are arbitrary RAM pages
added typically via qemu_put_buffer_async.
The check for >= IO_BUF_SIZE is only done when adding to the f->buf,
where as the check on f->iovcnt is done when you add an element to
the iovec and that can happen potentially in either case.
Dave
> Assuming doing both flushs is ok for qemu_put_buffer_async
> then I'd suggest renaming 'add_to_iovec' to 'queue_buffer'
> and have that method do both of these qemu_fflush() calls.
>
> > > ---
> > > migration/qemu-file.c | 42 ++++++++++++++++++++++++++++--------------
> > > 1 file changed, 28 insertions(+), 14 deletions(-)
> > >
> > > diff --git a/migration/qemu-file.c b/migration/qemu-file.c
> > > index 35c22605dd..05d9f42ddb 100644
> > > --- a/migration/qemu-file.c
> > > +++ b/migration/qemu-file.c
> > > @@ -343,8 +343,16 @@ int qemu_fclose(QEMUFile *f)
> > > return ret;
> > > }
> > >
> > > -static void add_to_iovec(QEMUFile *f, const uint8_t *buf, size_t size,
> > > - bool may_free)
> > > +/*
> > > + * Add buf to iovec. Do flush if iovec is full.
> > > + *
> > > + * Return values:
> > > + * 1 iovec is full and flushed
> > > + * 0 iovec is not flushed
> > > + *
> > > + */
> > > +static int add_to_iovec(QEMUFile *f, const uint8_t *buf, size_t size,
> > > + bool may_free)
> > > {
> > > /* check for adjacent buffer and coalesce them */
> > > if (f->iovcnt > 0 && buf == f->iov[f->iovcnt - 1].iov_base +
> > > @@ -362,7 +370,10 @@ static void add_to_iovec(QEMUFile *f, const uint8_t *buf, size_t size,
> > >
> > > if (f->iovcnt >= MAX_IOV_SIZE) {
> > > qemu_fflush(f);
> > > + return 1;
> > > }
> > > +
> > > + return 0;
> > > }
> > >
> > > void qemu_put_buffer_async(QEMUFile *f, const uint8_t *buf, size_t size,
> > > @@ -391,10 +402,11 @@ void qemu_put_buffer(QEMUFile *f, const uint8_t *buf, size_t size)
> > > }
> > > memcpy(f->buf + f->buf_index, buf, l);
> > > f->bytes_xfer += l;
> > > - add_to_iovec(f, f->buf + f->buf_index, l, false);
> > > - f->buf_index += l;
> > > - if (f->buf_index == IO_BUF_SIZE) {
> > > - qemu_fflush(f);
> > > + if (!add_to_iovec(f, f->buf + f->buf_index, l, false)) {
> > > + f->buf_index += l;
> > > + if (f->buf_index == IO_BUF_SIZE) {
> > > + qemu_fflush(f);
> > > + }
> > > }
> > > if (qemu_file_get_error(f)) {
> > > break;
> > > @@ -412,10 +424,11 @@ void qemu_put_byte(QEMUFile *f, int v)
> > >
> > > f->buf[f->buf_index] = v;
> > > f->bytes_xfer++;
> > > - add_to_iovec(f, f->buf + f->buf_index, 1, false);
> > > - f->buf_index++;
> > > - if (f->buf_index == IO_BUF_SIZE) {
> > > - qemu_fflush(f);
> > > + if (!add_to_iovec(f, f->buf + f->buf_index, 1, false)) {
> > > + f->buf_index++;
> > > + if (f->buf_index == IO_BUF_SIZE) {
> > > + qemu_fflush(f);
> > > + }
> > > }
> > > }
> > >
> > > @@ -717,10 +730,11 @@ ssize_t qemu_put_compression_data(QEMUFile *f, z_stream *stream,
> > > }
> > >
> > > qemu_put_be32(f, blen);
> > > - add_to_iovec(f, f->buf + f->buf_index, blen, false);
> > > - f->buf_index += blen;
> > > - if (f->buf_index == IO_BUF_SIZE) {
> > > - qemu_fflush(f);
> > > + if (!add_to_iovec(f, f->buf + f->buf_index, blen, false)) {
> > > + f->buf_index += blen;
> > > + if (f->buf_index == IO_BUF_SIZE) {
> > > + qemu_fflush(f);
> > > + }
> > > }
> > > return blen + sizeof(int32_t);
> > > }
> > > --
> > > 2.17.1
> > >
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> >
>
> Regards,
> Daniel
> --
> |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org -o- https://fstop138.berrange.com :|
> |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [Qemu-devel] [PATCH 2/2] migration/qemu-file: fix potential buf waste for extra buf_index adjustment
2019-08-23 11:06 ` Dr. David Alan Gilbert
2019-08-23 11:38 ` Daniel P. Berrangé
@ 2019-08-24 16:15 ` Wei Yang
2019-09-03 18:43 ` Dr. David Alan Gilbert
1 sibling, 1 reply; 13+ messages in thread
From: Wei Yang @ 2019-08-24 16:15 UTC (permalink / raw)
To: Dr. David Alan Gilbert; +Cc: quintela, berrange, Wei Yang, qemu-devel
On Fri, Aug 23, 2019 at 12:06:09PM +0100, Dr. David Alan Gilbert wrote:
>(Copying Dan in)
>
>* Wei Yang (richardw.yang@linux.intel.com) wrote:
>> In add_to_iovec(), qemu_fflush() will be called if iovec is full. If
>> this happens, buf_index is reset. Currently, this is not checked and
>> buf_index would always been adjust with buf size.
>>
>> This is not harmful, but will waste some space in file buffer.
>
>That's a nice find.
>
>> This patch make add_to_iovec() return 1 when it has flushed the file.
>> Then the caller could check the return value to see whether it is
>> necessary to adjust the buf_index any more.
>>
>> Signed-off-by: Wei Yang <richardw.yang@linux.intel.com>
>
>Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
>
>(I wonder if there's a way to wrap that little add_to_iovec, check, add
>to index, flush in a little wrapper).
>
>Dave
>
>> ---
>> migration/qemu-file.c | 42 ++++++++++++++++++++++++++++--------------
>> 1 file changed, 28 insertions(+), 14 deletions(-)
>>
>> diff --git a/migration/qemu-file.c b/migration/qemu-file.c
>> index 35c22605dd..05d9f42ddb 100644
>> --- a/migration/qemu-file.c
>> +++ b/migration/qemu-file.c
>> @@ -343,8 +343,16 @@ int qemu_fclose(QEMUFile *f)
>> return ret;
>> }
>>
>> -static void add_to_iovec(QEMUFile *f, const uint8_t *buf, size_t size,
>> - bool may_free)
>> +/*
>> + * Add buf to iovec. Do flush if iovec is full.
>> + *
>> + * Return values:
>> + * 1 iovec is full and flushed
>> + * 0 iovec is not flushed
>> + *
>> + */
>> +static int add_to_iovec(QEMUFile *f, const uint8_t *buf, size_t size,
>> + bool may_free)
>> {
>> /* check for adjacent buffer and coalesce them */
>> if (f->iovcnt > 0 && buf == f->iov[f->iovcnt - 1].iov_base +
>> @@ -362,7 +370,10 @@ static void add_to_iovec(QEMUFile *f, const uint8_t *buf, size_t size,
>>
>> if (f->iovcnt >= MAX_IOV_SIZE) {
>> qemu_fflush(f);
>> + return 1;
>> }
>> +
>> + return 0;
>> }
>>
>> void qemu_put_buffer_async(QEMUFile *f, const uint8_t *buf, size_t size,
>> @@ -391,10 +402,11 @@ void qemu_put_buffer(QEMUFile *f, const uint8_t *buf, size_t size)
>> }
>> memcpy(f->buf + f->buf_index, buf, l);
>> f->bytes_xfer += l;
>> - add_to_iovec(f, f->buf + f->buf_index, l, false);
>> - f->buf_index += l;
>> - if (f->buf_index == IO_BUF_SIZE) {
>> - qemu_fflush(f);
>> + if (!add_to_iovec(f, f->buf + f->buf_index, l, false)) {
>> + f->buf_index += l;
>> + if (f->buf_index == IO_BUF_SIZE) {
>> + qemu_fflush(f);
>> + }
You mean put these four lines into a wrapper?
Name it as add_buf_to_iovec?
>> }
>> if (qemu_file_get_error(f)) {
>> break;
>> @@ -412,10 +424,11 @@ void qemu_put_byte(QEMUFile *f, int v)
>>
>> f->buf[f->buf_index] = v;
>> f->bytes_xfer++;
>> - add_to_iovec(f, f->buf + f->buf_index, 1, false);
>> - f->buf_index++;
>> - if (f->buf_index == IO_BUF_SIZE) {
>> - qemu_fflush(f);
>> + if (!add_to_iovec(f, f->buf + f->buf_index, 1, false)) {
>> + f->buf_index++;
>> + if (f->buf_index == IO_BUF_SIZE) {
>> + qemu_fflush(f);
>> + }
>> }
>> }
>>
>> @@ -717,10 +730,11 @@ ssize_t qemu_put_compression_data(QEMUFile *f, z_stream *stream,
>> }
>>
>> qemu_put_be32(f, blen);
>> - add_to_iovec(f, f->buf + f->buf_index, blen, false);
>> - f->buf_index += blen;
>> - if (f->buf_index == IO_BUF_SIZE) {
>> - qemu_fflush(f);
>> + if (!add_to_iovec(f, f->buf + f->buf_index, blen, false)) {
>> + f->buf_index += blen;
>> + if (f->buf_index == IO_BUF_SIZE) {
>> + qemu_fflush(f);
>> + }
>> }
>> return blen + sizeof(int32_t);
>> }
>> --
>> 2.17.1
>>
>--
>Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
--
Wei Yang
Help you, Help me
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [Qemu-devel] [PATCH 2/2] migration/qemu-file: fix potential buf waste for extra buf_index adjustment
2019-08-23 13:05 ` Dr. David Alan Gilbert
@ 2019-08-24 16:22 ` Wei Yang
2019-09-03 13:22 ` Dr. David Alan Gilbert
0 siblings, 1 reply; 13+ messages in thread
From: Wei Yang @ 2019-08-24 16:22 UTC (permalink / raw)
To: Dr. David Alan Gilbert; +Cc: qemu-devel, Daniel P. Berrang?, Wei Yang, quintela
On Fri, Aug 23, 2019 at 02:05:02PM +0100, Dr. David Alan Gilbert wrote:
>* Daniel P. Berrang? (berrange@redhat.com) wrote:
>> On Fri, Aug 23, 2019 at 12:06:09PM +0100, Dr. David Alan Gilbert wrote:
>> > (Copying Dan in)
>> >
>> > * Wei Yang (richardw.yang@linux.intel.com) wrote:
>> > > In add_to_iovec(), qemu_fflush() will be called if iovec is full. If
>> > > this happens, buf_index is reset. Currently, this is not checked and
>> > > buf_index would always been adjust with buf size.
>> > >
>> > > This is not harmful, but will waste some space in file buffer.
>> >
>> > That's a nice find.
>> >
>> > > This patch make add_to_iovec() return 1 when it has flushed the file.
>> > > Then the caller could check the return value to see whether it is
>> > > necessary to adjust the buf_index any more.
>> > >
>> > > Signed-off-by: Wei Yang <richardw.yang@linux.intel.com>
>> >
>> > Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
>> >
>> > (I wonder if there's a way to wrap that little add_to_iovec, check, add
>> > to index, flush in a little wrapper).
>>
>> Given the name "add_to_iovec" I think it is pretty surprising
>> that it calls "qemu_flush" at all.
>>
>> It is also pretty wierd that we're checking two different
>> conditions in two different places.
>>
>> Right now the code is essentially doing this:
>>
>> if (f->iovcnt >= MAX_IOV_SIZE) {
>> qemu_fflush(f);
>> }
>> if (f->buf_index == IO_BUF_SIZE) {
>> qemu_fflush(f);
>> }
>>
>> Except that in the qemu_put_buffer_async() case, we're
>> only doing the first of these two checks. This feels
>> very odd indeed - I would have thought either it should
>> do both, or do neither.
>
>No, there's two separate types of buffers.
>
>There's f->buf which is a single allocated buffer in the QEMUFile
>with an offset buf_index, and there are arbitrary RAM pages
>added typically via qemu_put_buffer_async.
>
qemu_put_buffer_async is the only one which put a range not in f->buf into the
iovec.
And one thing confused me is even its name is async, add_to_iovec still would
call qemu_fflush when iovec is full. So it is not a always async function.
From the function name, it is a little difficult to differentiate
qemu_put_buffer and qemu_put_buffer_async.
>The check for >= IO_BUF_SIZE is only done when adding to the f->buf,
>where as the check on f->iovcnt is done when you add an element to
>the iovec and that can happen potentially in either case.
>
>Dave
>
>> Assuming doing both flushs is ok for qemu_put_buffer_async
>> then I'd suggest renaming 'add_to_iovec' to 'queue_buffer'
>> and have that method do both of these qemu_fflush() calls.
>>
>> > > ---
>> > > migration/qemu-file.c | 42 ++++++++++++++++++++++++++++--------------
>> > > 1 file changed, 28 insertions(+), 14 deletions(-)
>> > >
>> > > diff --git a/migration/qemu-file.c b/migration/qemu-file.c
>> > > index 35c22605dd..05d9f42ddb 100644
>> > > --- a/migration/qemu-file.c
>> > > +++ b/migration/qemu-file.c
>> > > @@ -343,8 +343,16 @@ int qemu_fclose(QEMUFile *f)
>> > > return ret;
>> > > }
>> > >
>> > > -static void add_to_iovec(QEMUFile *f, const uint8_t *buf, size_t size,
>> > > - bool may_free)
>> > > +/*
>> > > + * Add buf to iovec. Do flush if iovec is full.
>> > > + *
>> > > + * Return values:
>> > > + * 1 iovec is full and flushed
>> > > + * 0 iovec is not flushed
>> > > + *
>> > > + */
>> > > +static int add_to_iovec(QEMUFile *f, const uint8_t *buf, size_t size,
>> > > + bool may_free)
>> > > {
>> > > /* check for adjacent buffer and coalesce them */
>> > > if (f->iovcnt > 0 && buf == f->iov[f->iovcnt - 1].iov_base +
>> > > @@ -362,7 +370,10 @@ static void add_to_iovec(QEMUFile *f, const uint8_t *buf, size_t size,
>> > >
>> > > if (f->iovcnt >= MAX_IOV_SIZE) {
>> > > qemu_fflush(f);
>> > > + return 1;
>> > > }
>> > > +
>> > > + return 0;
>> > > }
>> > >
>> > > void qemu_put_buffer_async(QEMUFile *f, const uint8_t *buf, size_t size,
>> > > @@ -391,10 +402,11 @@ void qemu_put_buffer(QEMUFile *f, const uint8_t *buf, size_t size)
>> > > }
>> > > memcpy(f->buf + f->buf_index, buf, l);
>> > > f->bytes_xfer += l;
>> > > - add_to_iovec(f, f->buf + f->buf_index, l, false);
>> > > - f->buf_index += l;
>> > > - if (f->buf_index == IO_BUF_SIZE) {
>> > > - qemu_fflush(f);
>> > > + if (!add_to_iovec(f, f->buf + f->buf_index, l, false)) {
>> > > + f->buf_index += l;
>> > > + if (f->buf_index == IO_BUF_SIZE) {
>> > > + qemu_fflush(f);
>> > > + }
>> > > }
>> > > if (qemu_file_get_error(f)) {
>> > > break;
>> > > @@ -412,10 +424,11 @@ void qemu_put_byte(QEMUFile *f, int v)
>> > >
>> > > f->buf[f->buf_index] = v;
>> > > f->bytes_xfer++;
>> > > - add_to_iovec(f, f->buf + f->buf_index, 1, false);
>> > > - f->buf_index++;
>> > > - if (f->buf_index == IO_BUF_SIZE) {
>> > > - qemu_fflush(f);
>> > > + if (!add_to_iovec(f, f->buf + f->buf_index, 1, false)) {
>> > > + f->buf_index++;
>> > > + if (f->buf_index == IO_BUF_SIZE) {
>> > > + qemu_fflush(f);
>> > > + }
>> > > }
>> > > }
>> > >
>> > > @@ -717,10 +730,11 @@ ssize_t qemu_put_compression_data(QEMUFile *f, z_stream *stream,
>> > > }
>> > >
>> > > qemu_put_be32(f, blen);
>> > > - add_to_iovec(f, f->buf + f->buf_index, blen, false);
>> > > - f->buf_index += blen;
>> > > - if (f->buf_index == IO_BUF_SIZE) {
>> > > - qemu_fflush(f);
>> > > + if (!add_to_iovec(f, f->buf + f->buf_index, blen, false)) {
>> > > + f->buf_index += blen;
>> > > + if (f->buf_index == IO_BUF_SIZE) {
>> > > + qemu_fflush(f);
>> > > + }
>> > > }
>> > > return blen + sizeof(int32_t);
>> > > }
>> > > --
>> > > 2.17.1
>> > >
>> > --
>> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>> >
>>
>> Regards,
>> Daniel
>> --
>> |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
>> |: https://libvirt.org -o- https://fstop138.berrange.com :|
>> |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
>--
>Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
--
Wei Yang
Help you, Help me
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [Qemu-devel] [PATCH 2/2] migration/qemu-file: fix potential buf waste for extra buf_index adjustment
2019-08-24 16:22 ` Wei Yang
@ 2019-09-03 13:22 ` Dr. David Alan Gilbert
0 siblings, 0 replies; 13+ messages in thread
From: Dr. David Alan Gilbert @ 2019-09-03 13:22 UTC (permalink / raw)
To: Wei Yang; +Cc: qemu-devel, Daniel P. Berrang?, Wei Yang, quintela
* Wei Yang (richard.weiyang@gmail.com) wrote:
> On Fri, Aug 23, 2019 at 02:05:02PM +0100, Dr. David Alan Gilbert wrote:
> >* Daniel P. Berrang? (berrange@redhat.com) wrote:
> >> On Fri, Aug 23, 2019 at 12:06:09PM +0100, Dr. David Alan Gilbert wrote:
> >> > (Copying Dan in)
> >> >
> >> > * Wei Yang (richardw.yang@linux.intel.com) wrote:
> >> > > In add_to_iovec(), qemu_fflush() will be called if iovec is full. If
> >> > > this happens, buf_index is reset. Currently, this is not checked and
> >> > > buf_index would always been adjust with buf size.
> >> > >
> >> > > This is not harmful, but will waste some space in file buffer.
> >> >
> >> > That's a nice find.
> >> >
> >> > > This patch make add_to_iovec() return 1 when it has flushed the file.
> >> > > Then the caller could check the return value to see whether it is
> >> > > necessary to adjust the buf_index any more.
> >> > >
> >> > > Signed-off-by: Wei Yang <richardw.yang@linux.intel.com>
> >> >
> >> > Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
> >> >
> >> > (I wonder if there's a way to wrap that little add_to_iovec, check, add
> >> > to index, flush in a little wrapper).
> >>
> >> Given the name "add_to_iovec" I think it is pretty surprising
> >> that it calls "qemu_flush" at all.
> >>
> >> It is also pretty wierd that we're checking two different
> >> conditions in two different places.
> >>
> >> Right now the code is essentially doing this:
> >>
> >> if (f->iovcnt >= MAX_IOV_SIZE) {
> >> qemu_fflush(f);
> >> }
> >> if (f->buf_index == IO_BUF_SIZE) {
> >> qemu_fflush(f);
> >> }
> >>
> >> Except that in the qemu_put_buffer_async() case, we're
> >> only doing the first of these two checks. This feels
> >> very odd indeed - I would have thought either it should
> >> do both, or do neither.
> >
> >No, there's two separate types of buffers.
> >
> >There's f->buf which is a single allocated buffer in the QEMUFile
> >with an offset buf_index, and there are arbitrary RAM pages
> >added typically via qemu_put_buffer_async.
> >
>
> qemu_put_buffer_async is the only one which put a range not in f->buf into the
> iovec.
>
> And one thing confused me is even its name is async, add_to_iovec still would
> call qemu_fflush when iovec is full. So it is not a always async function.
> From the function name, it is a little difficult to differentiate
> qemu_put_buffer and qemu_put_buffer_async.
It's a bit confusing, but I think it is the right name; the 'async'
refers to the fact that the data is read from the buffer we pass
sometime later, not synchronously before the return of the
qemu_put_buffer_async call.
Dave
> >The check for >= IO_BUF_SIZE is only done when adding to the f->buf,
> >where as the check on f->iovcnt is done when you add an element to
> >the iovec and that can happen potentially in either case.
> >
> >Dave
> >
> >> Assuming doing both flushs is ok for qemu_put_buffer_async
> >> then I'd suggest renaming 'add_to_iovec' to 'queue_buffer'
> >> and have that method do both of these qemu_fflush() calls.
> >>
> >> > > ---
> >> > > migration/qemu-file.c | 42 ++++++++++++++++++++++++++++--------------
> >> > > 1 file changed, 28 insertions(+), 14 deletions(-)
> >> > >
> >> > > diff --git a/migration/qemu-file.c b/migration/qemu-file.c
> >> > > index 35c22605dd..05d9f42ddb 100644
> >> > > --- a/migration/qemu-file.c
> >> > > +++ b/migration/qemu-file.c
> >> > > @@ -343,8 +343,16 @@ int qemu_fclose(QEMUFile *f)
> >> > > return ret;
> >> > > }
> >> > >
> >> > > -static void add_to_iovec(QEMUFile *f, const uint8_t *buf, size_t size,
> >> > > - bool may_free)
> >> > > +/*
> >> > > + * Add buf to iovec. Do flush if iovec is full.
> >> > > + *
> >> > > + * Return values:
> >> > > + * 1 iovec is full and flushed
> >> > > + * 0 iovec is not flushed
> >> > > + *
> >> > > + */
> >> > > +static int add_to_iovec(QEMUFile *f, const uint8_t *buf, size_t size,
> >> > > + bool may_free)
> >> > > {
> >> > > /* check for adjacent buffer and coalesce them */
> >> > > if (f->iovcnt > 0 && buf == f->iov[f->iovcnt - 1].iov_base +
> >> > > @@ -362,7 +370,10 @@ static void add_to_iovec(QEMUFile *f, const uint8_t *buf, size_t size,
> >> > >
> >> > > if (f->iovcnt >= MAX_IOV_SIZE) {
> >> > > qemu_fflush(f);
> >> > > + return 1;
> >> > > }
> >> > > +
> >> > > + return 0;
> >> > > }
> >> > >
> >> > > void qemu_put_buffer_async(QEMUFile *f, const uint8_t *buf, size_t size,
> >> > > @@ -391,10 +402,11 @@ void qemu_put_buffer(QEMUFile *f, const uint8_t *buf, size_t size)
> >> > > }
> >> > > memcpy(f->buf + f->buf_index, buf, l);
> >> > > f->bytes_xfer += l;
> >> > > - add_to_iovec(f, f->buf + f->buf_index, l, false);
> >> > > - f->buf_index += l;
> >> > > - if (f->buf_index == IO_BUF_SIZE) {
> >> > > - qemu_fflush(f);
> >> > > + if (!add_to_iovec(f, f->buf + f->buf_index, l, false)) {
> >> > > + f->buf_index += l;
> >> > > + if (f->buf_index == IO_BUF_SIZE) {
> >> > > + qemu_fflush(f);
> >> > > + }
> >> > > }
> >> > > if (qemu_file_get_error(f)) {
> >> > > break;
> >> > > @@ -412,10 +424,11 @@ void qemu_put_byte(QEMUFile *f, int v)
> >> > >
> >> > > f->buf[f->buf_index] = v;
> >> > > f->bytes_xfer++;
> >> > > - add_to_iovec(f, f->buf + f->buf_index, 1, false);
> >> > > - f->buf_index++;
> >> > > - if (f->buf_index == IO_BUF_SIZE) {
> >> > > - qemu_fflush(f);
> >> > > + if (!add_to_iovec(f, f->buf + f->buf_index, 1, false)) {
> >> > > + f->buf_index++;
> >> > > + if (f->buf_index == IO_BUF_SIZE) {
> >> > > + qemu_fflush(f);
> >> > > + }
> >> > > }
> >> > > }
> >> > >
> >> > > @@ -717,10 +730,11 @@ ssize_t qemu_put_compression_data(QEMUFile *f, z_stream *stream,
> >> > > }
> >> > >
> >> > > qemu_put_be32(f, blen);
> >> > > - add_to_iovec(f, f->buf + f->buf_index, blen, false);
> >> > > - f->buf_index += blen;
> >> > > - if (f->buf_index == IO_BUF_SIZE) {
> >> > > - qemu_fflush(f);
> >> > > + if (!add_to_iovec(f, f->buf + f->buf_index, blen, false)) {
> >> > > + f->buf_index += blen;
> >> > > + if (f->buf_index == IO_BUF_SIZE) {
> >> > > + qemu_fflush(f);
> >> > > + }
> >> > > }
> >> > > return blen + sizeof(int32_t);
> >> > > }
> >> > > --
> >> > > 2.17.1
> >> > >
> >> > --
> >> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> >> >
> >>
> >> Regards,
> >> Daniel
> >> --
> >> |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
> >> |: https://libvirt.org -o- https://fstop138.berrange.com :|
> >> |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
> >--
> >Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
> --
> Wei Yang
> Help you, Help me
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [Qemu-devel] [PATCH 2/2] migration/qemu-file: fix potential buf waste for extra buf_index adjustment
2019-08-24 16:15 ` Wei Yang
@ 2019-09-03 18:43 ` Dr. David Alan Gilbert
2019-09-11 12:26 ` Wei Yang
0 siblings, 1 reply; 13+ messages in thread
From: Dr. David Alan Gilbert @ 2019-09-03 18:43 UTC (permalink / raw)
To: Wei Yang; +Cc: quintela, berrange, Wei Yang, qemu-devel
* Wei Yang (richard.weiyang@gmail.com) wrote:
> On Fri, Aug 23, 2019 at 12:06:09PM +0100, Dr. David Alan Gilbert wrote:
> >(Copying Dan in)
> >
> >* Wei Yang (richardw.yang@linux.intel.com) wrote:
> >> In add_to_iovec(), qemu_fflush() will be called if iovec is full. If
> >> this happens, buf_index is reset. Currently, this is not checked and
> >> buf_index would always been adjust with buf size.
> >>
> >> This is not harmful, but will waste some space in file buffer.
> >
> >That's a nice find.
> >
> >> This patch make add_to_iovec() return 1 when it has flushed the file.
> >> Then the caller could check the return value to see whether it is
> >> necessary to adjust the buf_index any more.
> >>
> >> Signed-off-by: Wei Yang <richardw.yang@linux.intel.com>
> >
> >Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
> >
> >(I wonder if there's a way to wrap that little add_to_iovec, check, add
> >to index, flush in a little wrapper).
> >
> >Dave
> >
> >> ---
> >> migration/qemu-file.c | 42 ++++++++++++++++++++++++++++--------------
> >> 1 file changed, 28 insertions(+), 14 deletions(-)
> >>
> >> diff --git a/migration/qemu-file.c b/migration/qemu-file.c
> >> index 35c22605dd..05d9f42ddb 100644
> >> --- a/migration/qemu-file.c
> >> +++ b/migration/qemu-file.c
> >> @@ -343,8 +343,16 @@ int qemu_fclose(QEMUFile *f)
> >> return ret;
> >> }
> >>
> >> -static void add_to_iovec(QEMUFile *f, const uint8_t *buf, size_t size,
> >> - bool may_free)
> >> +/*
> >> + * Add buf to iovec. Do flush if iovec is full.
> >> + *
> >> + * Return values:
> >> + * 1 iovec is full and flushed
> >> + * 0 iovec is not flushed
> >> + *
> >> + */
> >> +static int add_to_iovec(QEMUFile *f, const uint8_t *buf, size_t size,
> >> + bool may_free)
> >> {
> >> /* check for adjacent buffer and coalesce them */
> >> if (f->iovcnt > 0 && buf == f->iov[f->iovcnt - 1].iov_base +
> >> @@ -362,7 +370,10 @@ static void add_to_iovec(QEMUFile *f, const uint8_t *buf, size_t size,
> >>
> >> if (f->iovcnt >= MAX_IOV_SIZE) {
> >> qemu_fflush(f);
> >> + return 1;
> >> }
> >> +
> >> + return 0;
> >> }
> >>
> >> void qemu_put_buffer_async(QEMUFile *f, const uint8_t *buf, size_t size,
> >> @@ -391,10 +402,11 @@ void qemu_put_buffer(QEMUFile *f, const uint8_t *buf, size_t size)
> >> }
> >> memcpy(f->buf + f->buf_index, buf, l);
> >> f->bytes_xfer += l;
> >> - add_to_iovec(f, f->buf + f->buf_index, l, false);
> >> - f->buf_index += l;
> >> - if (f->buf_index == IO_BUF_SIZE) {
> >> - qemu_fflush(f);
> >> + if (!add_to_iovec(f, f->buf + f->buf_index, l, false)) {
> >> + f->buf_index += l;
> >> + if (f->buf_index == IO_BUF_SIZE) {
> >> + qemu_fflush(f);
> >> + }
>
> You mean put these four lines into a wrapper?
>
> Name it as add_buf_to_iovec?
Yes.
Dave
> >> }
> >> if (qemu_file_get_error(f)) {
> >> break;
> >> @@ -412,10 +424,11 @@ void qemu_put_byte(QEMUFile *f, int v)
> >>
> >> f->buf[f->buf_index] = v;
> >> f->bytes_xfer++;
> >> - add_to_iovec(f, f->buf + f->buf_index, 1, false);
> >> - f->buf_index++;
> >> - if (f->buf_index == IO_BUF_SIZE) {
> >> - qemu_fflush(f);
> >> + if (!add_to_iovec(f, f->buf + f->buf_index, 1, false)) {
> >> + f->buf_index++;
> >> + if (f->buf_index == IO_BUF_SIZE) {
> >> + qemu_fflush(f);
> >> + }
> >> }
> >> }
> >>
> >> @@ -717,10 +730,11 @@ ssize_t qemu_put_compression_data(QEMUFile *f, z_stream *stream,
> >> }
> >>
> >> qemu_put_be32(f, blen);
> >> - add_to_iovec(f, f->buf + f->buf_index, blen, false);
> >> - f->buf_index += blen;
> >> - if (f->buf_index == IO_BUF_SIZE) {
> >> - qemu_fflush(f);
> >> + if (!add_to_iovec(f, f->buf + f->buf_index, blen, false)) {
> >> + f->buf_index += blen;
> >> + if (f->buf_index == IO_BUF_SIZE) {
> >> + qemu_fflush(f);
> >> + }
> >> }
> >> return blen + sizeof(int32_t);
> >> }
> >> --
> >> 2.17.1
> >>
> >--
> >Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
> --
> Wei Yang
> Help you, Help me
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [Qemu-devel] [PATCH 2/2] migration/qemu-file: fix potential buf waste for extra buf_index adjustment
2019-09-03 18:43 ` Dr. David Alan Gilbert
@ 2019-09-11 12:26 ` Wei Yang
0 siblings, 0 replies; 13+ messages in thread
From: Wei Yang @ 2019-09-11 12:26 UTC (permalink / raw)
To: Dr. David Alan Gilbert; +Cc: qemu-devel, quintela, berrange, Wei Yang, Wei Yang
On Tue, Sep 03, 2019 at 07:43:24PM +0100, Dr. David Alan Gilbert wrote:
>* Wei Yang (richard.weiyang@gmail.com) wrote:
>> On Fri, Aug 23, 2019 at 12:06:09PM +0100, Dr. David Alan Gilbert wrote:
>> >(Copying Dan in)
>> >
>> >* Wei Yang (richardw.yang@linux.intel.com) wrote:
>> >> In add_to_iovec(), qemu_fflush() will be called if iovec is full. If
>> >> this happens, buf_index is reset. Currently, this is not checked and
>> >> buf_index would always been adjust with buf size.
>> >>
>> >> This is not harmful, but will waste some space in file buffer.
>> >
>> >That's a nice find.
>> >
>> >> This patch make add_to_iovec() return 1 when it has flushed the file.
>> >> Then the caller could check the return value to see whether it is
>> >> necessary to adjust the buf_index any more.
>> >>
>> >> Signed-off-by: Wei Yang <richardw.yang@linux.intel.com>
>> >
>> >Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
>> >
>> >(I wonder if there's a way to wrap that little add_to_iovec, check, add
>> >to index, flush in a little wrapper).
>> >
>> >Dave
>> >
>> >> ---
>> >> migration/qemu-file.c | 42 ++++++++++++++++++++++++++++--------------
>> >> 1 file changed, 28 insertions(+), 14 deletions(-)
>> >>
>> >> diff --git a/migration/qemu-file.c b/migration/qemu-file.c
>> >> index 35c22605dd..05d9f42ddb 100644
>> >> --- a/migration/qemu-file.c
>> >> +++ b/migration/qemu-file.c
>> >> @@ -343,8 +343,16 @@ int qemu_fclose(QEMUFile *f)
>> >> return ret;
>> >> }
>> >>
>> >> -static void add_to_iovec(QEMUFile *f, const uint8_t *buf, size_t size,
>> >> - bool may_free)
>> >> +/*
>> >> + * Add buf to iovec. Do flush if iovec is full.
>> >> + *
>> >> + * Return values:
>> >> + * 1 iovec is full and flushed
>> >> + * 0 iovec is not flushed
>> >> + *
>> >> + */
>> >> +static int add_to_iovec(QEMUFile *f, const uint8_t *buf, size_t size,
>> >> + bool may_free)
>> >> {
>> >> /* check for adjacent buffer and coalesce them */
>> >> if (f->iovcnt > 0 && buf == f->iov[f->iovcnt - 1].iov_base +
>> >> @@ -362,7 +370,10 @@ static void add_to_iovec(QEMUFile *f, const uint8_t *buf, size_t size,
>> >>
>> >> if (f->iovcnt >= MAX_IOV_SIZE) {
>> >> qemu_fflush(f);
>> >> + return 1;
>> >> }
>> >> +
>> >> + return 0;
>> >> }
>> >>
>> >> void qemu_put_buffer_async(QEMUFile *f, const uint8_t *buf, size_t size,
>> >> @@ -391,10 +402,11 @@ void qemu_put_buffer(QEMUFile *f, const uint8_t *buf, size_t size)
>> >> }
>> >> memcpy(f->buf + f->buf_index, buf, l);
>> >> f->bytes_xfer += l;
>> >> - add_to_iovec(f, f->buf + f->buf_index, l, false);
>> >> - f->buf_index += l;
>> >> - if (f->buf_index == IO_BUF_SIZE) {
>> >> - qemu_fflush(f);
>> >> + if (!add_to_iovec(f, f->buf + f->buf_index, l, false)) {
>> >> + f->buf_index += l;
>> >> + if (f->buf_index == IO_BUF_SIZE) {
>> >> + qemu_fflush(f);
>> >> + }
>>
>> You mean put these four lines into a wrapper?
>>
>> Name it as add_buf_to_iovec?
>
>Yes.
Sure, Let me prepare v2 with this.
--
Wei Yang
Help you, Help me
^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2019-09-11 12:27 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-07-31 14:42 [Qemu-devel] [PATCH 0/2] migration/qemu-file: cleanup and refine qemu-file Wei Yang
2019-07-31 14:42 ` [Qemu-devel] [PATCH 1/2] migration/qemu-file: remove check on writev_buffer in qemu_put_compression_data Wei Yang
2019-08-23 10:30 ` Dr. David Alan Gilbert
2019-07-31 14:42 ` [Qemu-devel] [PATCH 2/2] migration/qemu-file: fix potential buf waste for extra buf_index adjustment Wei Yang
2019-08-23 11:06 ` Dr. David Alan Gilbert
2019-08-23 11:38 ` Daniel P. Berrangé
2019-08-23 13:05 ` Dr. David Alan Gilbert
2019-08-24 16:22 ` Wei Yang
2019-09-03 13:22 ` Dr. David Alan Gilbert
2019-08-24 16:15 ` Wei Yang
2019-09-03 18:43 ` Dr. David Alan Gilbert
2019-09-11 12:26 ` Wei Yang
2019-08-19 2:36 ` [Qemu-devel] [PATCH 0/2] migration/qemu-file: cleanup and refine qemu-file Wei Yang
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).