[Qemu-devel] [PATCH 00/19] buffer/vnc: improve vnc buffer hsndling

[Qemu-devel] [PATCH 00/19] buffer/vnc: improve vnc buffer hsndling

[Gerd Hoffmann]

  Hi,

This series has a bunch of improvements in the vnc buffer handling,
to reduce the memory footprint.  Some of the changes are applied to
the buffer helper functions which Daniel separated out of the vnc code
recently.

Most patches have been on the list before, based on a older version of
Daniel's "separate out buffer code" patches.  Now I finally managed to
rebase and adapt the changes to the latest version which landed in
master meanwhile.  I don't expect major issues showing up here and plan
to have a pull request with this in time for 2.5-rc0.

Peter, if you have anything pending not yet in here please rebase and
resend.

please review,
  Gerd

Gerd Hoffmann (14):
  buffer: add buffer_init
  buffer: add buffer_move_empty
  buffer: add buffer_move
  buffer: add buffer_shrink
  buffer: add tracing
  vnc: attach names to buffers
  vnc: kill jobs queue buffer
  vnc-jobs: move buffer reset, use new buffer move
  vnc: zap dead code
  vnc: add vnc_width+vnc_height helpers
  vnc: factor out vnc_update_server_surface
  vnc: use vnc_{width,height} in vnc_set_area_dirty
  vnc: only alloc server surface with clients connected
  vnc: fix local state init

Peter Lieven (5):
  buffer: make the Buffer capacity increase in powers of two
  vnc: recycle empty vs->output buffer
  buffer: factor out buffer_req_size
  buffer: factor out buffer_adj_size
  buffer: allow a buffer to shrink gracefully

 include/qemu/buffer.h |  43 ++++++++++++++++++++
 trace-events          |   6 +++
 ui/vnc-jobs.c         |  34 +++++++++-------
 ui/vnc.c              |  92 ++++++++++++++++++++++++++++++++-----------
 util/buffer.c         | 107 +++++++++++++++++++++++++++++++++++++++++++++++++-
 5 files changed, 243 insertions(+), 39 deletions(-)

-- 
1.8.3.1

[Qemu-devel] [PATCH 01/19] buffer: make the Buffer capacity increase in powers of two

[Gerd Hoffmann]

From: Peter Lieven <pl@kamp.de>

This makes sure the number of reallocs is in O(log N).

Signed-off-by: Peter Lieven <pl@kamp.de>

[ rebased to util/buffer.c ]

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 util/buffer.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/util/buffer.c b/util/buffer.c
index cedd055..7ddd693 100644
--- a/util/buffer.c
+++ b/util/buffer.c
@@ -20,10 +20,13 @@
 
 #include "qemu/buffer.h"
 
+#define BUFFER_MIN_INIT_SIZE 4096
+
 void buffer_reserve(Buffer *buffer, size_t len)
 {
     if ((buffer->capacity - buffer->offset) < len) {
-        buffer->capacity += (len + 1024);
+        buffer->capacity = pow2ceil(buffer->offset + len);
+        buffer->capacity = MAX(buffer->capacity, BUFFER_MIN_INIT_SIZE);
         buffer->buffer = g_realloc(buffer->buffer, buffer->capacity);
     }
 }
-- 
1.8.3.1

[Qemu-devel] [PATCH 02/19] buffer: add buffer_init

[Gerd Hoffmann]

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Reviewed-by: Peter Lieven <pl@kamp.de>
---
 include/qemu/buffer.h | 12 ++++++++++++
 util/buffer.c         | 11 +++++++++++
 2 files changed, 23 insertions(+)

diff --git a/include/qemu/buffer.h b/include/qemu/buffer.h
index b380cec..0710e16 100644
--- a/include/qemu/buffer.h
+++ b/include/qemu/buffer.h
@@ -34,12 +34,24 @@ typedef struct Buffer Buffer;
  */
 
 struct Buffer {
+    char *name;
     size_t capacity;
     size_t offset;
     uint8_t *buffer;
 };
 
 /**
+ * buffer_init:
+ * @buffer: the buffer object
+ * @name: buffer name
+ *
+ * Optionally attach a name to the buffer, to make it easier
+ * to identify in debug traces.
+ */
+void buffer_init(Buffer *buffer, const char *name, ...)
+        GCC_FMT_ATTR(2, 3);
+
+/**
  * buffer_reserve:
  * @buffer: the buffer object
  * @len: the minimum required free space
diff --git a/util/buffer.c b/util/buffer.c
index 7ddd693..12bf2d7 100644
--- a/util/buffer.c
+++ b/util/buffer.c
@@ -22,6 +22,15 @@
 
 #define BUFFER_MIN_INIT_SIZE 4096
 
+void buffer_init(Buffer *buffer, const char *name, ...)
+{
+    va_list ap;
+
+    va_start(ap, name);
+    buffer->name = g_strdup_vprintf(name, ap);
+    va_end(ap);
+}
+
 void buffer_reserve(Buffer *buffer, size_t len)
 {
     if ((buffer->capacity - buffer->offset) < len) {
@@ -49,9 +58,11 @@ void buffer_reset(Buffer *buffer)
 void buffer_free(Buffer *buffer)
 {
     g_free(buffer->buffer);
+    g_free(buffer->name);
     buffer->offset = 0;
     buffer->capacity = 0;
     buffer->buffer = NULL;
+    buffer->name = NULL;
 }
 
 void buffer_append(Buffer *buffer, const void *data, size_t len)
-- 
1.8.3.1

[Qemu-devel] [PATCH 03/19] buffer: add buffer_move_empty

[Gerd Hoffmann]

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Reviewed-by: Peter Lieven <pl@kamp.de>
---
 include/qemu/buffer.h | 10 ++++++++++
 util/buffer.c         | 14 ++++++++++++++
 2 files changed, 24 insertions(+)

diff --git a/include/qemu/buffer.h b/include/qemu/buffer.h
index 0710e16..f53ee9e 100644
--- a/include/qemu/buffer.h
+++ b/include/qemu/buffer.h
@@ -127,4 +127,14 @@ uint8_t *buffer_end(Buffer *buffer);
  */
 gboolean buffer_empty(Buffer *buffer);
 
+/**
+ * buffer_move_empty:
+ * @to: destination buffer object
+ * @from: source buffer object
+ *
+ * Moves buffer, without copying data.  'to' buffer must be empty.
+ * 'from' buffer is empty and zero-sized on return.
+ */
+void buffer_move_empty(Buffer *to, Buffer *from);
+
 #endif /* QEMU_BUFFER_H__ */
diff --git a/util/buffer.c b/util/buffer.c
index 12bf2d7..c7a39ec 100644
--- a/util/buffer.c
+++ b/util/buffer.c
@@ -77,3 +77,17 @@ void buffer_advance(Buffer *buffer, size_t len)
             (buffer->offset - len));
     buffer->offset -= len;
 }
+
+void buffer_move_empty(Buffer *to, Buffer *from)
+{
+    assert(to->offset == 0);
+
+    g_free(to->buffer);
+    to->offset = from->offset;
+    to->capacity = from->capacity;
+    to->buffer = from->buffer;
+
+    from->offset = 0;
+    from->capacity = 0;
+    from->buffer = NULL;
+}
-- 
1.8.3.1

[Qemu-devel] [PATCH 04/19] buffer: add buffer_move

[Gerd Hoffmann]

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Reviewed-by: Peter Lieven <pl@kamp.de>
---
 include/qemu/buffer.h | 10 ++++++++++
 util/buffer.c         | 16 ++++++++++++++++
 2 files changed, 26 insertions(+)

diff --git a/include/qemu/buffer.h b/include/qemu/buffer.h
index f53ee9e..1358df1 100644
--- a/include/qemu/buffer.h
+++ b/include/qemu/buffer.h
@@ -137,4 +137,14 @@ gboolean buffer_empty(Buffer *buffer);
  */
 void buffer_move_empty(Buffer *to, Buffer *from);
 
+/**
+ * buffer_move:
+ * @to: destination buffer object
+ * @from: source buffer object
+ *
+ * Moves buffer, copying data (unless 'to' buffer happens to be empty).
+ * 'from' buffer is empty and zero-sized on return.
+ */
+void buffer_move(Buffer *to, Buffer *from);
+
 #endif /* QEMU_BUFFER_H__ */
diff --git a/util/buffer.c b/util/buffer.c
index c7a39ec..e8f798e 100644
--- a/util/buffer.c
+++ b/util/buffer.c
@@ -91,3 +91,19 @@ void buffer_move_empty(Buffer *to, Buffer *from)
     from->capacity = 0;
     from->buffer = NULL;
 }
+
+void buffer_move(Buffer *to, Buffer *from)
+{
+    if (to->offset == 0) {
+        buffer_move_empty(to, from);
+        return;
+    }
+
+    buffer_reserve(to, from->offset);
+    buffer_append(to, from->buffer, from->offset);
+
+    g_free(from->buffer);
+    from->offset = 0;
+    from->capacity = 0;
+    from->buffer = NULL;
+}
-- 
1.8.3.1

[Qemu-devel] [PATCH 05/19] buffer: add buffer_shrink

[Gerd Hoffmann]

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 include/qemu/buffer.h | 10 ++++++++++
 util/buffer.c         | 20 +++++++++++++++++++-
 2 files changed, 29 insertions(+), 1 deletion(-)

diff --git a/include/qemu/buffer.h b/include/qemu/buffer.h
index 1358df1..0a69b3a 100644
--- a/include/qemu/buffer.h
+++ b/include/qemu/buffer.h
@@ -52,6 +52,16 @@ void buffer_init(Buffer *buffer, const char *name, ...)
         GCC_FMT_ATTR(2, 3);
 
 /**
+ * buffer_shrink:
+ * @buffer: the buffer object
+ *
+ * Try to shrink the buffer.  Checks current buffer capacity and size
+ * and reduces capacity in case only a fraction of the buffer is
+ * actually used.
+ */
+void buffer_shrink(Buffer *buffer);
+
+/**
  * buffer_reserve:
  * @buffer: the buffer object
  * @len: the minimum required free space
diff --git a/util/buffer.c b/util/buffer.c
index e8f798e..234e33d 100644
--- a/util/buffer.c
+++ b/util/buffer.c
@@ -20,7 +20,8 @@
 
 #include "qemu/buffer.h"
 
-#define BUFFER_MIN_INIT_SIZE 4096
+#define BUFFER_MIN_INIT_SIZE     4096
+#define BUFFER_MIN_SHRINK_SIZE  65536
 
 void buffer_init(Buffer *buffer, const char *name, ...)
 {
@@ -31,6 +32,23 @@ void buffer_init(Buffer *buffer, const char *name, ...)
     va_end(ap);
 }
 
+void buffer_shrink(Buffer *buffer)
+{
+    /*
+     * Only shrink in case the used size is *much* smaller than the
+     * capacity, to avoid bumping up & down the buffers all the time.
+     * realloc() isn't exactly cheap ...
+     */
+    if (buffer->offset < (buffer->capacity >> 3) &&
+        buffer->capacity > BUFFER_MIN_SHRINK_SIZE) {
+        return;
+    }
+
+    buffer->capacity = pow2ceil(buffer->offset);
+    buffer->capacity = MAX(buffer->capacity, BUFFER_MIN_SHRINK_SIZE);
+    buffer->buffer = g_realloc(buffer->buffer, buffer->capacity);
+}
+
 void buffer_reserve(Buffer *buffer, size_t len)
 {
     if ((buffer->capacity - buffer->offset) < len) {
-- 
1.8.3.1

[Qemu-devel] [PATCH 06/19] buffer: add tracing

[Gerd Hoffmann]

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 trace-events  |  6 ++++++
 util/buffer.c | 18 ++++++++++++++++++
 2 files changed, 24 insertions(+)

diff --git a/trace-events b/trace-events
index bdfe79f..050d45b 100644
--- a/trace-events
+++ b/trace-events
@@ -1387,6 +1387,12 @@ ppc_tb_adjust(uint64_t offs1, uint64_t offs2, int64_t diff, int64_t seconds) "ad
 prep_io_800_writeb(uint32_t addr, uint32_t val) "0x%08" PRIx32 " => 0x%02" PRIx32
 prep_io_800_readb(uint32_t addr, uint32_t retval) "0x%08" PRIx32 " <= 0x%02" PRIx32
 
+# io/buffer.c
+buffer_resize(const char *buf, size_t olen, size_t len) "%s: old %zd, new %zd"
+buffer_move_empty(const char *buf, size_t len, const char *from) "%s: %zd bytes from %s"
+buffer_move(const char *buf, size_t len, const char *from) "%s: %zd bytes from %s"
+buffer_free(const char *buf, size_t len) "%s: capacity %zd"
+
 # util/hbitmap.c
 hbitmap_iter_skip_words(const void *hb, void *hbi, uint64_t pos, unsigned long cur) "hb %p hbi %p pos %"PRId64" cur 0x%lx"
 hbitmap_reset(void *hb, uint64_t start, uint64_t count, uint64_t sbit, uint64_t ebit) "hb %p items %"PRIu64",%"PRIu64" bits %"PRIu64"..%"PRIu64
diff --git a/util/buffer.c b/util/buffer.c
index 234e33d..ae2907e 100644
--- a/util/buffer.c
+++ b/util/buffer.c
@@ -19,6 +19,7 @@
  */
 
 #include "qemu/buffer.h"
+#include "trace.h"
 
 #define BUFFER_MIN_INIT_SIZE     4096
 #define BUFFER_MIN_SHRINK_SIZE  65536
@@ -34,6 +35,8 @@ void buffer_init(Buffer *buffer, const char *name, ...)
 
 void buffer_shrink(Buffer *buffer)
 {
+    size_t old;
+
     /*
      * Only shrink in case the used size is *much* smaller than the
      * capacity, to avoid bumping up & down the buffers all the time.
@@ -44,17 +47,25 @@ void buffer_shrink(Buffer *buffer)
         return;
     }
 
+    old = buffer->capacity;
     buffer->capacity = pow2ceil(buffer->offset);
     buffer->capacity = MAX(buffer->capacity, BUFFER_MIN_SHRINK_SIZE);
     buffer->buffer = g_realloc(buffer->buffer, buffer->capacity);
+    trace_buffer_resize(buffer->name ?: "unnamed",
+                        old, buffer->capacity);
 }
 
 void buffer_reserve(Buffer *buffer, size_t len)
 {
+    size_t old;
+
     if ((buffer->capacity - buffer->offset) < len) {
+        old = buffer->capacity;
         buffer->capacity = pow2ceil(buffer->offset + len);
         buffer->capacity = MAX(buffer->capacity, BUFFER_MIN_INIT_SIZE);
         buffer->buffer = g_realloc(buffer->buffer, buffer->capacity);
+        trace_buffer_resize(buffer->name ?: "unnamed",
+                            old, buffer->capacity);
     }
 }
 
@@ -75,6 +86,7 @@ void buffer_reset(Buffer *buffer)
 
 void buffer_free(Buffer *buffer)
 {
+    trace_buffer_free(buffer->name ?: "unnamed", buffer->capacity);
     g_free(buffer->buffer);
     g_free(buffer->name);
     buffer->offset = 0;
@@ -98,6 +110,9 @@ void buffer_advance(Buffer *buffer, size_t len)
 
 void buffer_move_empty(Buffer *to, Buffer *from)
 {
+    trace_buffer_move_empty(to->name ?: "unnamed",
+                            from->offset,
+                            from->name ?: "unnamed");
     assert(to->offset == 0);
 
     g_free(to->buffer);
@@ -117,6 +132,9 @@ void buffer_move(Buffer *to, Buffer *from)
         return;
     }
 
+    trace_buffer_move(to->name ?: "unnamed",
+                      from->offset,
+                      from->name ?: "unnamed");
     buffer_reserve(to, from->offset);
     buffer_append(to, from->buffer, from->offset);
 
-- 
1.8.3.1

[Qemu-devel] [PATCH 07/19] vnc: attach names to buffers

[Gerd Hoffmann]

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 ui/vnc-jobs.c |  3 +++
 ui/vnc.c      | 20 ++++++++++++++++++++
 2 files changed, 23 insertions(+)

diff --git a/ui/vnc-jobs.c b/ui/vnc-jobs.c
index 22c9abc..2e6c15f 100644
--- a/ui/vnc-jobs.c
+++ b/ui/vnc-jobs.c
@@ -218,6 +218,8 @@ static int vnc_worker_thread_loop(VncJobQueue *queue)
     int n_rectangles;
     int saved_offset;
 
+    buffer_init(&vs.output, "vnc-worker-output");
+
     vnc_lock_queue(queue);
     while (QTAILQ_EMPTY(&queue->jobs) && !queue->exit) {
         qemu_cond_wait(&queue->cond, &queue->mutex);
@@ -302,6 +304,7 @@ static VncJobQueue *vnc_queue_init(void)
 
     qemu_cond_init(&queue->cond);
     qemu_mutex_init(&queue->mutex);
+    buffer_init(&queue->buffer, "vnc-job-queue");
     QTAILQ_INIT(&queue->jobs);
     return queue;
 }
diff --git a/ui/vnc.c b/ui/vnc.c
index dd4f581..49e8f3a 100644
--- a/ui/vnc.c
+++ b/ui/vnc.c
@@ -2978,6 +2978,26 @@ static void vnc_connect(VncDisplay *vd, int csock,
     vs->csock = csock;
     vs->vd = vd;
 
+    buffer_init(&vs->input,          "vnc-input/%d", csock);
+    buffer_init(&vs->output,         "vnc-output/%d", csock);
+    buffer_init(&vs->ws_input,       "vnc-ws_input/%d", csock);
+    buffer_init(&vs->ws_output,      "vnc-ws_output/%d", csock);
+    buffer_init(&vs->jobs_buffer,    "vnc-jobs_buffer/%d", csock);
+
+    buffer_init(&vs->tight.tight,    "vnc-tight/%d", csock);
+    buffer_init(&vs->tight.zlib,     "vnc-tight-zlib/%d", csock);
+    buffer_init(&vs->tight.gradient, "vnc-tight-gradient/%d", csock);
+#ifdef CONFIG_VNC_JPEG
+    buffer_init(&vs->tight.jpeg,     "vnc-tight-jpeg/%d", csock);
+#endif
+#ifdef CONFIG_VNC_PNG
+    buffer_init(&vs->tight.png,      "vnc-tight-png/%d", csock);
+#endif
+    buffer_init(&vs->zlib.zlib,      "vnc-zlib/%d", csock);
+    buffer_init(&vs->zrle.zrle,      "vnc-zrle/%d", csock);
+    buffer_init(&vs->zrle.fb,        "vnc-zrle-fb/%d", csock);
+    buffer_init(&vs->zrle.zlib,      "vnc-zrle-zlib/%d", csock);
+
     if (skipauth) {
 	vs->auth = VNC_AUTH_NONE;
 	vs->subauth = VNC_AUTH_INVALID;
-- 
1.8.3.1

[Qemu-devel] [PATCH 08/19] vnc: kill jobs queue buffer

[Gerd Hoffmann]

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Reviewed-by: Peter Lieven <pl@kamp.de>
---
 ui/vnc-jobs.c | 6 ------
 1 file changed, 6 deletions(-)

diff --git a/ui/vnc-jobs.c b/ui/vnc-jobs.c
index 2e6c15f..329d13e 100644
--- a/ui/vnc-jobs.c
+++ b/ui/vnc-jobs.c
@@ -54,7 +54,6 @@ struct VncJobQueue {
     QemuCond cond;
     QemuMutex mutex;
     QemuThread thread;
-    Buffer buffer;
     bool exit;
     QTAILQ_HEAD(, VncJob) jobs;
 };
@@ -193,7 +192,6 @@ static void vnc_async_encoding_start(VncState *orig, VncState *local)
     local->zlib = orig->zlib;
     local->hextile = orig->hextile;
     local->zrle = orig->zrle;
-    local->output =  queue->buffer;
     local->csock = -1; /* Don't do any network work on this thread */
 
     buffer_reset(&local->output);
@@ -206,8 +204,6 @@ static void vnc_async_encoding_end(VncState *orig, VncState *local)
     orig->hextile = local->hextile;
     orig->zrle = local->zrle;
     orig->lossy_rect = local->lossy_rect;
-
-    queue->buffer = local->output;
 }
 
 static int vnc_worker_thread_loop(VncJobQueue *queue)
@@ -304,7 +300,6 @@ static VncJobQueue *vnc_queue_init(void)
 
     qemu_cond_init(&queue->cond);
     qemu_mutex_init(&queue->mutex);
-    buffer_init(&queue->buffer, "vnc-job-queue");
     QTAILQ_INIT(&queue->jobs);
     return queue;
 }
@@ -313,7 +308,6 @@ static void vnc_queue_clear(VncJobQueue *q)
 {
     qemu_cond_destroy(&queue->cond);
     qemu_mutex_destroy(&queue->mutex);
-    buffer_free(&queue->buffer);
     g_free(q);
     queue = NULL; /* Unset global queue */
 }
-- 
1.8.3.1

[Qemu-devel] [PATCH 09/19] vnc-jobs: move buffer reset, use new buffer move

[Gerd Hoffmann]

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 ui/vnc-jobs.c | 15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)

diff --git a/ui/vnc-jobs.c b/ui/vnc-jobs.c
index 329d13e..fd9ed39 100644
--- a/ui/vnc-jobs.c
+++ b/ui/vnc-jobs.c
@@ -29,6 +29,7 @@
 #include "vnc.h"
 #include "vnc-jobs.h"
 #include "qemu/sockets.h"
+#include "qemu/main-loop.h"
 #include "block/aio.h"
 
 /*
@@ -165,8 +166,11 @@ void vnc_jobs_consume_buffer(VncState *vs)
 
     vnc_lock_output(vs);
     if (vs->jobs_buffer.offset) {
-        vnc_write(vs, vs->jobs_buffer.buffer, vs->jobs_buffer.offset);
-        buffer_reset(&vs->jobs_buffer);
+        if (vs->csock != -1 && buffer_empty(&vs->output)) {
+            qemu_set_fd_handler(vs->csock, vnc_client_read,
+                                vnc_client_write, vs);
+        }
+        buffer_move(&vs->output, &vs->jobs_buffer);
     }
     flush = vs->csock != -1 && vs->abort != true;
     vnc_unlock_output(vs);
@@ -193,8 +197,6 @@ static void vnc_async_encoding_start(VncState *orig, VncState *local)
     local->hextile = orig->hextile;
     local->zrle = orig->zrle;
     local->csock = -1; /* Don't do any network work on this thread */
-
-    buffer_reset(&local->output);
 }
 
 static void vnc_async_encoding_end(VncState *orig, VncState *local)
@@ -272,14 +274,13 @@ static int vnc_worker_thread_loop(VncJobQueue *queue)
 
     vnc_lock_output(job->vs);
     if (job->vs->csock != -1) {
-        buffer_reserve(&job->vs->jobs_buffer, vs.output.offset);
-        buffer_append(&job->vs->jobs_buffer, vs.output.buffer,
-                      vs.output.offset);
+        buffer_move(&job->vs->jobs_buffer, &vs.output);
         /* Copy persistent encoding data */
         vnc_async_encoding_end(job->vs, &vs);
 
 	qemu_bh_schedule(job->vs->bh);
     }  else {
+        buffer_reset(&vs.output);
         /* Copy persistent encoding data */
         vnc_async_encoding_end(job->vs, &vs);
     }
-- 
1.8.3.1

[Qemu-devel] [PATCH 10/19] vnc: zap dead code

[Gerd Hoffmann]

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 ui/vnc.c | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/ui/vnc.c b/ui/vnc.c
index 49e8f3a..cb1954c 100644
--- a/ui/vnc.c
+++ b/ui/vnc.c
@@ -722,10 +722,6 @@ static void vnc_dpy_switch(DisplayChangeListener *dcl,
                                           width, height, NULL, 0);
 
     /* guest surface */
-#if 0 /* FIXME */
-    if (ds_get_bytes_per_pixel(ds) != vd->guest.ds->pf.bytes_per_pixel)
-        console_color_init(ds);
-#endif
     qemu_pixman_image_unref(vd->guest.fb);
     vd->guest.fb = pixman_image_ref(surface->image);
     vd->guest.format = surface->format;
-- 
1.8.3.1

[Qemu-devel] [PATCH 11/19] vnc: add vnc_width+vnc_height helpers

[Gerd Hoffmann]

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 ui/vnc.c | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)

diff --git a/ui/vnc.c b/ui/vnc.c
index cb1954c..283b93c 100644
--- a/ui/vnc.c
+++ b/ui/vnc.c
@@ -605,6 +605,17 @@ static void framebuffer_update_request(VncState *vs, int incremental,
 static void vnc_refresh(DisplayChangeListener *dcl);
 static int vnc_refresh_server_surface(VncDisplay *vd);
 
+static int vnc_width(VncDisplay *vd)
+{
+    return MIN(VNC_MAX_WIDTH, ROUND_UP(surface_width(vd->ds),
+                                       VNC_DIRTY_PIXELS_PER_BIT));
+}
+
+static int vnc_height(VncDisplay *vd)
+{
+    return MIN(VNC_MAX_HEIGHT, surface_height(vd->ds));
+}
+
 static void vnc_set_area_dirty(DECLARE_BITMAP(dirty[VNC_MAX_HEIGHT],
                                VNC_MAX_WIDTH / VNC_DIRTY_PIXELS_PER_BIT),
                                int width, int height,
@@ -715,9 +726,8 @@ static void vnc_dpy_switch(DisplayChangeListener *dcl,
     /* server surface */
     qemu_pixman_image_unref(vd->server);
     vd->ds = surface;
-    width = MIN(VNC_MAX_WIDTH, ROUND_UP(surface_width(vd->ds),
-                                        VNC_DIRTY_PIXELS_PER_BIT));
-    height = MIN(VNC_MAX_HEIGHT, surface_height(vd->ds));
+    width = vnc_width(vd);
+    height = vnc_height(vd);
     vd->server = pixman_image_create_bits(VNC_SERVER_FB_FORMAT,
                                           width, height, NULL, 0);
 
-- 
1.8.3.1

[Qemu-devel] [PATCH 12/19] vnc: factor out vnc_update_server_surface

[Gerd Hoffmann]

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 ui/vnc.c | 23 ++++++++++++++++-------
 1 file changed, 16 insertions(+), 7 deletions(-)

diff --git a/ui/vnc.c b/ui/vnc.c
index 283b93c..61a8f2c 100644
--- a/ui/vnc.c
+++ b/ui/vnc.c
@@ -714,6 +714,17 @@ void *vnc_server_fb_ptr(VncDisplay *vd, int x, int y)
     return ptr;
 }
 
+static void vnc_update_server_surface(VncDisplay *vd)
+{
+    qemu_pixman_image_unref(vd->server);
+    vd->server = NULL;
+
+    vd->server = pixman_image_create_bits(VNC_SERVER_FB_FORMAT,
+                                          vnc_width(vd),
+                                          vnc_height(vd),
+                                          NULL, 0);
+}
+
 static void vnc_dpy_switch(DisplayChangeListener *dcl,
                            DisplaySurface *surface)
 {
@@ -722,19 +733,17 @@ static void vnc_dpy_switch(DisplayChangeListener *dcl,
     int width, height;
 
     vnc_abort_display_jobs(vd);
-
-    /* server surface */
-    qemu_pixman_image_unref(vd->server);
     vd->ds = surface;
-    width = vnc_width(vd);
-    height = vnc_height(vd);
-    vd->server = pixman_image_create_bits(VNC_SERVER_FB_FORMAT,
-                                          width, height, NULL, 0);
+
+    /* server surface */
+    vnc_update_server_surface(vd);
 
     /* guest surface */
     qemu_pixman_image_unref(vd->guest.fb);
     vd->guest.fb = pixman_image_ref(surface->image);
     vd->guest.format = surface->format;
+    width = vnc_width(vd);
+    height = vnc_height(vd);
     memset(vd->guest.dirty, 0x00, sizeof(vd->guest.dirty));
     vnc_set_area_dirty(vd->guest.dirty, width, height, 0, 0,
                        width, height);
-- 
1.8.3.1

[Qemu-devel] [PATCH 13/19] vnc: use vnc_{width, height} in vnc_set_area_dirty

[Gerd Hoffmann]

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 ui/vnc.c | 21 ++++++++++-----------
 1 file changed, 10 insertions(+), 11 deletions(-)

diff --git a/ui/vnc.c b/ui/vnc.c
index 61a8f2c..8ee1266 100644
--- a/ui/vnc.c
+++ b/ui/vnc.c
@@ -618,8 +618,12 @@ static int vnc_height(VncDisplay *vd)
 
 static void vnc_set_area_dirty(DECLARE_BITMAP(dirty[VNC_MAX_HEIGHT],
                                VNC_MAX_WIDTH / VNC_DIRTY_PIXELS_PER_BIT),
-                               int width, int height,
-                               int x, int y, int w, int h) {
+                               VncDisplay *vd,
+                               int x, int y, int w, int h)
+{
+    int width = vnc_width(vd);
+    int height = vnc_height(vd);
+
     /* this is needed this to ensure we updated all affected
      * blocks if x % VNC_DIRTY_PIXELS_PER_BIT != 0 */
     w += (x % VNC_DIRTY_PIXELS_PER_BIT);
@@ -641,10 +645,8 @@ static void vnc_dpy_update(DisplayChangeListener *dcl,
 {
     VncDisplay *vd = container_of(dcl, VncDisplay, dcl);
     struct VncSurface *s = &vd->guest;
-    int width = pixman_image_get_width(vd->server);
-    int height = pixman_image_get_height(vd->server);
 
-    vnc_set_area_dirty(s->dirty, width, height, x, y, w, h);
+    vnc_set_area_dirty(s->dirty, vd, x, y, w, h);
 }
 
 void vnc_framebuffer_update(VncState *vs, int x, int y, int w, int h,
@@ -745,7 +747,7 @@ static void vnc_dpy_switch(DisplayChangeListener *dcl,
     width = vnc_width(vd);
     height = vnc_height(vd);
     memset(vd->guest.dirty, 0x00, sizeof(vd->guest.dirty));
-    vnc_set_area_dirty(vd->guest.dirty, width, height, 0, 0,
+    vnc_set_area_dirty(vd->guest.dirty, vd, 0, 0,
                        width, height);
 
     QTAILQ_FOREACH(vs, &vd->clients, next) {
@@ -755,7 +757,7 @@ static void vnc_dpy_switch(DisplayChangeListener *dcl,
             vnc_cursor_define(vs);
         }
         memset(vs->dirty, 0x00, sizeof(vs->dirty));
-        vnc_set_area_dirty(vs->dirty, width, height, 0, 0,
+        vnc_set_area_dirty(vs->dirty, vd, 0, 0,
                            width, height);
     }
 }
@@ -2011,9 +2013,6 @@ static void ext_key_event(VncState *vs, int down,
 static void framebuffer_update_request(VncState *vs, int incremental,
                                        int x, int y, int w, int h)
 {
-    int width = pixman_image_get_width(vs->vd->server);
-    int height = pixman_image_get_height(vs->vd->server);
-
     vs->need_update = 1;
 
     if (incremental) {
@@ -2021,7 +2020,7 @@ static void framebuffer_update_request(VncState *vs, int incremental,
     }
 
     vs->force_update = 1;
-    vnc_set_area_dirty(vs->dirty, width, height, x, y, w, h);
+    vnc_set_area_dirty(vs->dirty, vs->vd, x, y, w, h);
 }
 
 static void send_ext_key_event_ack(VncState *vs)
-- 
1.8.3.1

[Qemu-devel] [PATCH 14/19] vnc: only alloc server surface with clients connected

[Gerd Hoffmann]

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 ui/vnc.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/ui/vnc.c b/ui/vnc.c
index 8ee1266..c5bef47 100644
--- a/ui/vnc.c
+++ b/ui/vnc.c
@@ -721,6 +721,10 @@ static void vnc_update_server_surface(VncDisplay *vd)
     qemu_pixman_image_unref(vd->server);
     vd->server = NULL;
 
+    if (QTAILQ_EMPTY(&vd->clients)) {
+        return;
+    }
+
     vd->server = pixman_image_create_bits(VNC_SERVER_FB_FORMAT,
                                           vnc_width(vd),
                                           vnc_height(vd),
@@ -1231,6 +1235,10 @@ void vnc_disconnect_finish(VncState *vs)
     if (vs->initialized) {
         QTAILQ_REMOVE(&vs->vd->clients, vs, next);
         qemu_remove_mouse_mode_change_notifier(&vs->mouse_mode_notifier);
+        if (QTAILQ_EMPTY(&vs->vd->clients)) {
+            /* last client gone */
+            vnc_update_server_surface(vs->vd);
+        }
     }
 
     if (vs->vd->lock_key_sync)
@@ -3069,6 +3077,7 @@ void vnc_init_state(VncState *vs)
 {
     vs->initialized = true;
     VncDisplay *vd = vs->vd;
+    bool first_client = QTAILQ_EMPTY(&vd->clients);
 
     vs->last_x = -1;
     vs->last_y = -1;
@@ -3082,6 +3091,9 @@ void vnc_init_state(VncState *vs)
     vs->bh = qemu_bh_new(vnc_jobs_bh, vs);
 
     QTAILQ_INSERT_TAIL(&vd->clients, vs, next);
+    if (first_client) {
+        vnc_update_server_surface(vd);
+    }
 
     graphic_hw_update(vd->dcl.con);
 
-- 
1.8.3.1

[Qemu-devel] [PATCH 15/19] vnc: fix local state init

[Gerd Hoffmann]

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 ui/vnc-jobs.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/ui/vnc-jobs.c b/ui/vnc-jobs.c
index fd9ed39..12389cc 100644
--- a/ui/vnc-jobs.c
+++ b/ui/vnc-jobs.c
@@ -185,6 +185,9 @@ void vnc_jobs_consume_buffer(VncState *vs)
  */
 static void vnc_async_encoding_start(VncState *orig, VncState *local)
 {
+    buffer_init(&local->output, "vnc-worker-output");
+    local->csock = -1; /* Don't do any network work on this thread */
+
     local->vnc_encoding = orig->vnc_encoding;
     local->features = orig->features;
     local->vd = orig->vd;
@@ -196,7 +199,6 @@ static void vnc_async_encoding_start(VncState *orig, VncState *local)
     local->zlib = orig->zlib;
     local->hextile = orig->hextile;
     local->zrle = orig->zrle;
-    local->csock = -1; /* Don't do any network work on this thread */
 }
 
 static void vnc_async_encoding_end(VncState *orig, VncState *local)
@@ -212,12 +214,10 @@ static int vnc_worker_thread_loop(VncJobQueue *queue)
 {
     VncJob *job;
     VncRectEntry *entry, *tmp;
-    VncState vs;
+    VncState vs = {};
     int n_rectangles;
     int saved_offset;
 
-    buffer_init(&vs.output, "vnc-worker-output");
-
     vnc_lock_queue(queue);
     while (QTAILQ_EMPTY(&queue->jobs) && !queue->exit) {
         qemu_cond_wait(&queue->cond, &queue->mutex);
-- 
1.8.3.1

[Qemu-devel] [PATCH 16/19] vnc: recycle empty vs->output buffer

[Gerd Hoffmann]

From: Peter Lieven <pl@kamp.de>

If the vs->output buffer is empty it will be dropped
by the next qio_buffer_move_empty in vnc_jobs_consume_buffer
anyway. So reuse the allocated buffer from this buffer
in the worker thread where we otherwise would start with
an empty (unallocated buffer).

Signed-off-by: Peter Lieven <pl@kamp.de>

[ added a comment describing the non-obvious optimization ]

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 ui/vnc-jobs.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/ui/vnc-jobs.c b/ui/vnc-jobs.c
index 12389cc..08f0163 100644
--- a/ui/vnc-jobs.c
+++ b/ui/vnc-jobs.c
@@ -235,6 +235,14 @@ static int vnc_worker_thread_loop(VncJobQueue *queue)
         vnc_unlock_output(job->vs);
         goto disconnected;
     }
+    if (buffer_empty(&job->vs->output)) {
+        /*
+         * Looks like a NOP as it obviously moves no data.  But it
+         * moves the empty buffer, so we don't have to malloc a new
+         * one for vs.output
+         */
+        buffer_move_empty(&vs.output, &job->vs->output);
+    }
     vnc_unlock_output(job->vs);
 
     /* Make a local copy of vs and switch output buffers */
-- 
1.8.3.1

[Qemu-devel] [PATCH 17/19] buffer: factor out buffer_req_size

[Gerd Hoffmann]

From: Peter Lieven <pl@kamp.de>

Signed-off-by: Peter Lieven <pl@kamp.de>
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 util/buffer.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/util/buffer.c b/util/buffer.c
index ae2907e..31f1d9f 100644
--- a/util/buffer.c
+++ b/util/buffer.c
@@ -24,6 +24,13 @@
 #define BUFFER_MIN_INIT_SIZE     4096
 #define BUFFER_MIN_SHRINK_SIZE  65536
 
+static size_t buffer_req_size(Buffer *buffer, size_t len)
+{
+    return MAX(BUFFER_MIN_INIT_SIZE,
+               pow2ceil(buffer->offset + len));
+}
+
+
 void buffer_init(Buffer *buffer, const char *name, ...)
 {
     va_list ap;
@@ -61,8 +68,7 @@ void buffer_reserve(Buffer *buffer, size_t len)
 
     if ((buffer->capacity - buffer->offset) < len) {
         old = buffer->capacity;
-        buffer->capacity = pow2ceil(buffer->offset + len);
-        buffer->capacity = MAX(buffer->capacity, BUFFER_MIN_INIT_SIZE);
+        buffer->capacity = buffer_req_size(buffer, len);
         buffer->buffer = g_realloc(buffer->buffer, buffer->capacity);
         trace_buffer_resize(buffer->name ?: "unnamed",
                             old, buffer->capacity);
-- 
1.8.3.1

[Qemu-devel] [PATCH 18/19] buffer: factor out buffer_adj_size

[Gerd Hoffmann]

From: Peter Lieven <pl@kamp.de>

Signed-off-by: Peter Lieven <pl@kamp.de>
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 util/buffer.c | 25 ++++++++++---------------
 1 file changed, 10 insertions(+), 15 deletions(-)

diff --git a/util/buffer.c b/util/buffer.c
index 31f1d9f..fe5a44e 100644
--- a/util/buffer.c
+++ b/util/buffer.c
@@ -30,6 +30,14 @@ static size_t buffer_req_size(Buffer *buffer, size_t len)
                pow2ceil(buffer->offset + len));
 }
 
+static void buffer_adj_size(Buffer *buffer, size_t len)
+{
+    size_t old = buffer->capacity;
+    buffer->capacity = buffer_req_size(buffer, len);
+    buffer->buffer = g_realloc(buffer->buffer, buffer->capacity);
+    trace_buffer_resize(buffer->name ?: "unnamed",
+                        old, buffer->capacity);
+}
 
 void buffer_init(Buffer *buffer, const char *name, ...)
 {
@@ -42,8 +50,6 @@ void buffer_init(Buffer *buffer, const char *name, ...)
 
 void buffer_shrink(Buffer *buffer)
 {
-    size_t old;
-
     /*
      * Only shrink in case the used size is *much* smaller than the
      * capacity, to avoid bumping up & down the buffers all the time.
@@ -54,24 +60,13 @@ void buffer_shrink(Buffer *buffer)
         return;
     }
 
-    old = buffer->capacity;
-    buffer->capacity = pow2ceil(buffer->offset);
-    buffer->capacity = MAX(buffer->capacity, BUFFER_MIN_SHRINK_SIZE);
-    buffer->buffer = g_realloc(buffer->buffer, buffer->capacity);
-    trace_buffer_resize(buffer->name ?: "unnamed",
-                        old, buffer->capacity);
+    buffer_adj_size(buffer, 0);
 }
 
 void buffer_reserve(Buffer *buffer, size_t len)
 {
-    size_t old;
-
     if ((buffer->capacity - buffer->offset) < len) {
-        old = buffer->capacity;
-        buffer->capacity = buffer_req_size(buffer, len);
-        buffer->buffer = g_realloc(buffer->buffer, buffer->capacity);
-        trace_buffer_resize(buffer->name ?: "unnamed",
-                            old, buffer->capacity);
+        buffer_adj_size(buffer, len);
     }
 }
 
-- 
1.8.3.1

[Qemu-devel] [PATCH 19/19] buffer: allow a buffer to shrink gracefully

[Gerd Hoffmann]

From: Peter Lieven <pl@kamp.de>

the idea behind this patch is to allow the buffer to shrink, but
make this a seldom operation. The buffers average size is measured
exponentionally smoothed with am alpha of 1/128.

Signed-off-by: Peter Lieven <pl@kamp.de>
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 include/qemu/buffer.h |  1 +
 util/buffer.c         | 34 ++++++++++++++++++++++++++++------
 2 files changed, 29 insertions(+), 6 deletions(-)

diff --git a/include/qemu/buffer.h b/include/qemu/buffer.h
index 0a69b3a..dead9b7 100644
--- a/include/qemu/buffer.h
+++ b/include/qemu/buffer.h
@@ -37,6 +37,7 @@ struct Buffer {
     char *name;
     size_t capacity;
     size_t offset;
+    uint64_t avg_size;
     uint8_t *buffer;
 };
 
diff --git a/util/buffer.c b/util/buffer.c
index fe5a44e..5461f86 100644
--- a/util/buffer.c
+++ b/util/buffer.c
@@ -23,6 +23,7 @@
 
 #define BUFFER_MIN_INIT_SIZE     4096
 #define BUFFER_MIN_SHRINK_SIZE  65536
+#define BUFFER_AVG_SIZE_SHIFT       7
 
 static size_t buffer_req_size(Buffer *buffer, size_t len)
 {
@@ -37,6 +38,11 @@ static void buffer_adj_size(Buffer *buffer, size_t len)
     buffer->buffer = g_realloc(buffer->buffer, buffer->capacity);
     trace_buffer_resize(buffer->name ?: "unnamed",
                         old, buffer->capacity);
+
+    /* make it even harder for the buffer to shrink, reset average size
+     * to currenty capacity if it is larger than the average. */
+    buffer->avg_size = MAX(buffer->avg_size,
+                           buffer->capacity << BUFFER_AVG_SIZE_SHIFT);
 }
 
 void buffer_init(Buffer *buffer, const char *name, ...)
@@ -48,16 +54,30 @@ void buffer_init(Buffer *buffer, const char *name, ...)
     va_end(ap);
 }
 
+static uint64_t buffer_get_avg_size(Buffer *buffer)
+{
+    return buffer->avg_size >> BUFFER_AVG_SIZE_SHIFT;
+}
+
 void buffer_shrink(Buffer *buffer)
 {
-    /*
-     * Only shrink in case the used size is *much* smaller than the
-     * capacity, to avoid bumping up & down the buffers all the time.
+    size_t new;
+
+    /* Calculate the average size of the buffer as
+     * avg_size = avg_size * ( 1 - a ) + required_size * a
+     * where a is 1 / 2 ^ QIO_BUFFER_AVG_SIZE_SHIFT. */
+    buffer->avg_size *= (1 << BUFFER_AVG_SIZE_SHIFT) - 1;
+    buffer->avg_size >>= BUFFER_AVG_SIZE_SHIFT;
+    buffer->avg_size += buffer_req_size(buffer, 0);
+
+    /* And then only shrink if the average size of the buffer is much
+     * too big, to avoid bumping up & down the buffers all the time.
      * realloc() isn't exactly cheap ...
      */
-    if (buffer->offset < (buffer->capacity >> 3) &&
-        buffer->capacity > BUFFER_MIN_SHRINK_SIZE) {
-        return;
+    new = buffer_req_size(buffer, buffer_get_avg_size(buffer));
+    if (new < buffer->capacity >> 3 &&
+        new >= BUFFER_MIN_SHRINK_SIZE) {
+        buffer_adj_size(buffer, buffer_get_avg_size(buffer));
     }
 
     buffer_adj_size(buffer, 0);
@@ -83,6 +103,7 @@ uint8_t *buffer_end(Buffer *buffer)
 void buffer_reset(Buffer *buffer)
 {
     buffer->offset = 0;
+    buffer_shrink(buffer);
 }
 
 void buffer_free(Buffer *buffer)
@@ -107,6 +128,7 @@ void buffer_advance(Buffer *buffer, size_t len)
     memmove(buffer->buffer, buffer->buffer + len,
             (buffer->offset - len));
     buffer->offset -= len;
+    buffer_shrink(buffer);
 }
 
 void buffer_move_empty(Buffer *to, Buffer *from)
-- 
1.8.3.1

Re: [Qemu-devel] [PATCH 01/19] buffer: make the Buffer capacity increase in powers of two

[Daniel P. Berrange]

On Fri, Oct 30, 2015 at 12:09:56PM +0100, Gerd Hoffmann wrote:
> From: Peter Lieven <pl@kamp.de>
> 
> This makes sure the number of reallocs is in O(log N).
> 
> Signed-off-by: Peter Lieven <pl@kamp.de>
> 
> [ rebased to util/buffer.c ]
> 
> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>

Reviewed-by: Daniel P. Berrange <berrange@redhat.com>

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [PATCH 02/19] buffer: add buffer_init

[Daniel P. Berrange]

On Fri, Oct 30, 2015 at 12:09:57PM +0100, Gerd Hoffmann wrote:
> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
> Reviewed-by: Peter Lieven <pl@kamp.de>
> ---
>  include/qemu/buffer.h | 12 ++++++++++++
>  util/buffer.c         | 11 +++++++++++
>  2 files changed, 23 insertions(+)

Reviewed-by: Daniel P. Berrange <berrange@redhat.com>


Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [PATCH 03/19] buffer: add buffer_move_empty

[Daniel P. Berrange]

On Fri, Oct 30, 2015 at 12:09:58PM +0100, Gerd Hoffmann wrote:
> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
> Reviewed-by: Peter Lieven <pl@kamp.de>
> ---
>  include/qemu/buffer.h | 10 ++++++++++
>  util/buffer.c         | 14 ++++++++++++++
>  2 files changed, 24 insertions(+)
> 
> diff --git a/include/qemu/buffer.h b/include/qemu/buffer.h
> index 0710e16..f53ee9e 100644
> --- a/include/qemu/buffer.h
> +++ b/include/qemu/buffer.h
> @@ -127,4 +127,14 @@ uint8_t *buffer_end(Buffer *buffer);
>   */
>  gboolean buffer_empty(Buffer *buffer);
>  
> +/**
> + * buffer_move_empty:
> + * @to: destination buffer object
> + * @from: source buffer object
> + *
> + * Moves buffer, without copying data.  'to' buffer must be empty.

Hmm, on the one hand the code do 'assert(to->offset) == 0', but on
the other hand it does 'g_free(to->buffer)' as if it expected the
buffer to have existing data.

If you just remove the 'assert', then there's no real requirement
for 'to' to be empty, as we'd do the right thin in free'ing the
data. Then we can just change this API doc to say

    "Any existing data in "to" will be freed"

> + * 'from' buffer is empty and zero-sized on return.
> + */
> +void buffer_move_empty(Buffer *to, Buffer *from);
> +
>  #endif /* QEMU_BUFFER_H__ */
> diff --git a/util/buffer.c b/util/buffer.c
> index 12bf2d7..c7a39ec 100644
> --- a/util/buffer.c
> +++ b/util/buffer.c
> @@ -77,3 +77,17 @@ void buffer_advance(Buffer *buffer, size_t len)
>              (buffer->offset - len));
>      buffer->offset -= len;
>  }
> +
> +void buffer_move_empty(Buffer *to, Buffer *from)
> +{
> +    assert(to->offset == 0);
> +
> +    g_free(to->buffer);
> +    to->offset = from->offset;
> +    to->capacity = from->capacity;
> +    to->buffer = from->buffer;
> +
> +    from->offset = 0;
> +    from->capacity = 0;
> +    from->buffer = NULL;
> +}

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [PATCH 04/19] buffer: add buffer_move

[Daniel P. Berrange]

On Fri, Oct 30, 2015 at 12:09:59PM +0100, Gerd Hoffmann wrote:
> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
> Reviewed-by: Peter Lieven <pl@kamp.de>
> ---
>  include/qemu/buffer.h | 10 ++++++++++
>  util/buffer.c         | 16 ++++++++++++++++
>  2 files changed, 26 insertions(+)

Reviewed-by: Daniel P. Berrange <berrange@redhat.com>


Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [PATCH 05/19] buffer: add buffer_shrink

[Daniel P. Berrange]

On Fri, Oct 30, 2015 at 12:10:00PM +0100, Gerd Hoffmann wrote:
> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
> ---
>  include/qemu/buffer.h | 10 ++++++++++
>  util/buffer.c         | 20 +++++++++++++++++++-
>  2 files changed, 29 insertions(+), 1 deletion(-)

Reviewed-by: Daniel P. Berrange <berrange@redhat.com>

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [PATCH 06/19] buffer: add tracing

[Daniel P. Berrange]

On Fri, Oct 30, 2015 at 12:10:01PM +0100, Gerd Hoffmann wrote:
> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
> ---
>  trace-events  |  6 ++++++
>  util/buffer.c | 18 ++++++++++++++++++
>  2 files changed, 24 insertions(+)

Reviewed-by: Daniel P. Berrange <berrange@redhat.com>


Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [PATCH 07/19] vnc: attach names to buffers

[Daniel P. Berrange]

On Fri, Oct 30, 2015 at 12:10:02PM +0100, Gerd Hoffmann wrote:
> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
> ---
>  ui/vnc-jobs.c |  3 +++
>  ui/vnc.c      | 20 ++++++++++++++++++++
>  2 files changed, 23 insertions(+)

Reviewed-by: Daniel P. Berrange <berrange@redhat.com>

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [PATCH 08/19] vnc: kill jobs queue buffer

[Daniel P. Berrange]

On Fri, Oct 30, 2015 at 12:10:03PM +0100, Gerd Hoffmann wrote:
> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
> Reviewed-by: Peter Lieven <pl@kamp.de>
> ---
>  ui/vnc-jobs.c | 6 ------
>  1 file changed, 6 deletions(-)

Reviewed-by: Daniel P. Berrange <berrange@redhat.com>

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [PATCH 09/19] vnc-jobs: move buffer reset, use new buffer move

[Daniel P. Berrange]

On Fri, Oct 30, 2015 at 12:10:04PM +0100, Gerd Hoffmann wrote:
> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
> ---
>  ui/vnc-jobs.c | 15 ++++++++-------
>  1 file changed, 8 insertions(+), 7 deletions(-)

Reviewed-by: Daniel P. Berrange <berrange@redhat.com>

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [PATCH 10/19] vnc: zap dead code

[Daniel P. Berrange]

On Fri, Oct 30, 2015 at 12:10:05PM +0100, Gerd Hoffmann wrote:
> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
> ---
>  ui/vnc.c | 4 ----
>  1 file changed, 4 deletions(-)

Reviewed-by: Daniel P. Berrange <berrange@redhat.com>

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [PATCH 11/19] vnc: add vnc_width+vnc_height helpers

[Daniel P. Berrange]

On Fri, Oct 30, 2015 at 12:10:06PM +0100, Gerd Hoffmann wrote:
> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
> ---
>  ui/vnc.c | 16 +++++++++++++---
>  1 file changed, 13 insertions(+), 3 deletions(-)

Reviewed-by: Daniel P. Berrange <berrange@redhat.com>


Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [PATCH 12/19] vnc: factor out vnc_update_server_surface

[Daniel P. Berrange]

On Fri, Oct 30, 2015 at 12:10:07PM +0100, Gerd Hoffmann wrote:
> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
> ---
>  ui/vnc.c | 23 ++++++++++++++++-------
>  1 file changed, 16 insertions(+), 7 deletions(-)

Reviewed-by: Daniel P. Berrange <berrange@redhat.com>

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [PATCH 13/19] vnc: use vnc_{width, height} in vnc_set_area_dirty

[Daniel P. Berrange]

On Fri, Oct 30, 2015 at 12:10:08PM +0100, Gerd Hoffmann wrote:
> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
> ---
>  ui/vnc.c | 21 ++++++++++-----------
>  1 file changed, 10 insertions(+), 11 deletions(-)

Reviewed-by: Daniel P. Berrange <berrange@redhat.com>


Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [PATCH 14/19] vnc: only alloc server surface with clients connected

[Daniel P. Berrange]

On Fri, Oct 30, 2015 at 12:10:09PM +0100, Gerd Hoffmann wrote:
> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
> ---
>  ui/vnc.c | 12 ++++++++++++
>  1 file changed, 12 insertions(+)

Reviewed-by: Daniel P. Berrange <berrange@redhat.com>

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [PATCH 15/19] vnc: fix local state init

[Daniel P. Berrange]

On Fri, Oct 30, 2015 at 12:10:10PM +0100, Gerd Hoffmann wrote:

It isn't entirely obvious what bug is being fixed here, perhaps
a little more info in the commit message would help.

> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
> ---
>  ui/vnc-jobs.c | 8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)
> 
> diff --git a/ui/vnc-jobs.c b/ui/vnc-jobs.c
> index fd9ed39..12389cc 100644
> --- a/ui/vnc-jobs.c
> +++ b/ui/vnc-jobs.c
> @@ -185,6 +185,9 @@ void vnc_jobs_consume_buffer(VncState *vs)
>   */
>  static void vnc_async_encoding_start(VncState *orig, VncState *local)
>  {
> +    buffer_init(&local->output, "vnc-worker-output");
> +    local->csock = -1; /* Don't do any network work on this thread */
> +
>      local->vnc_encoding = orig->vnc_encoding;
>      local->features = orig->features;
>      local->vd = orig->vd;
> @@ -196,7 +199,6 @@ static void vnc_async_encoding_start(VncState *orig, VncState *local)
>      local->zlib = orig->zlib;
>      local->hextile = orig->hextile;
>      local->zrle = orig->zrle;
> -    local->csock = -1; /* Don't do any network work on this thread */
>  }
>  
>  static void vnc_async_encoding_end(VncState *orig, VncState *local)
> @@ -212,12 +214,10 @@ static int vnc_worker_thread_loop(VncJobQueue *queue)
>  {
>      VncJob *job;
>      VncRectEntry *entry, *tmp;
> -    VncState vs;
> +    VncState vs = {};

Ok, so we'd not initialized memory in VncState to all zeros before.

>      int n_rectangles;
>      int saved_offset;
>  
> -    buffer_init(&vs.output, "vnc-worker-output");
> -
>      vnc_lock_queue(queue);
>      while (QTAILQ_EMPTY(&queue->jobs) && !queue->exit) {
>          qemu_cond_wait(&queue->cond, &queue->mutex);

Reviewed-by: Daniel P. Berrange <berrange@redhat.com>

> -- 
> 1.8.3.1
> 

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [PATCH 16/19] vnc: recycle empty vs->output buffer

[Daniel P. Berrange]

On Fri, Oct 30, 2015 at 12:10:11PM +0100, Gerd Hoffmann wrote:
> From: Peter Lieven <pl@kamp.de>
> 
> If the vs->output buffer is empty it will be dropped
> by the next qio_buffer_move_empty in vnc_jobs_consume_buffer
> anyway. So reuse the allocated buffer from this buffer
> in the worker thread where we otherwise would start with
> an empty (unallocated buffer).
> 
> Signed-off-by: Peter Lieven <pl@kamp.de>
> 
> [ added a comment describing the non-obvious optimization ]
> 
> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
> ---
>  ui/vnc-jobs.c | 8 ++++++++
>  1 file changed, 8 insertions(+)

Reviewed-by: Daniel P. Berrange <berrange@redhat.com>

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [PATCH 17/19] buffer: factor out buffer_req_size

[Daniel P. Berrange]

On Fri, Oct 30, 2015 at 12:10:12PM +0100, Gerd Hoffmann wrote:
> From: Peter Lieven <pl@kamp.de>
> 
> Signed-off-by: Peter Lieven <pl@kamp.de>
> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
> ---
>  util/buffer.c | 10 ++++++++--
>  1 file changed, 8 insertions(+), 2 deletions(-)

Reviewed-by: Daniel P. Berrange <berrange@redhat.com>


Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [PATCH 18/19] buffer: factor out buffer_adj_size

[Daniel P. Berrange]

On Fri, Oct 30, 2015 at 12:10:13PM +0100, Gerd Hoffmann wrote:
> From: Peter Lieven <pl@kamp.de>
> 
> Signed-off-by: Peter Lieven <pl@kamp.de>
> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
> ---
>  util/buffer.c | 25 ++++++++++---------------
>  1 file changed, 10 insertions(+), 15 deletions(-)

Reviewed-by: Daniel P. Berrange <berrange@redhat.com>

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [PATCH 19/19] buffer: allow a buffer to shrink gracefully

[Daniel P. Berrange]

On Fri, Oct 30, 2015 at 12:10:14PM +0100, Gerd Hoffmann wrote:
> From: Peter Lieven <pl@kamp.de>
> 
> the idea behind this patch is to allow the buffer to shrink, but
> make this a seldom operation. The buffers average size is measured
> exponentionally smoothed with am alpha of 1/128.

s/am/an/

> 
> Signed-off-by: Peter Lieven <pl@kamp.de>
> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
> ---
>  include/qemu/buffer.h |  1 +
>  util/buffer.c         | 34 ++++++++++++++++++++++++++++------
>  2 files changed, 29 insertions(+), 6 deletions(-)


> diff --git a/util/buffer.c b/util/buffer.c
> index fe5a44e..5461f86 100644
> --- a/util/buffer.c
> +++ b/util/buffer.c
> @@ -23,6 +23,7 @@
>  
>  #define BUFFER_MIN_INIT_SIZE     4096
>  #define BUFFER_MIN_SHRINK_SIZE  65536
> +#define BUFFER_AVG_SIZE_SHIFT       7
>  
>  static size_t buffer_req_size(Buffer *buffer, size_t len)
>  {
> @@ -37,6 +38,11 @@ static void buffer_adj_size(Buffer *buffer, size_t len)
>      buffer->buffer = g_realloc(buffer->buffer, buffer->capacity);
>      trace_buffer_resize(buffer->name ?: "unnamed",
>                          old, buffer->capacity);
> +
> +    /* make it even harder for the buffer to shrink, reset average size
> +     * to currenty capacity if it is larger than the average. */

s/currenty/current/

Reviewed-by: Daniel P. Berrange <berrange@redhat.com>


Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [PATCH 03/19] buffer: add buffer_move_empty

[Daniel P. Berrange]

On Fri, Oct 30, 2015 at 09:11:01PM +0900, Daniel P. Berrange wrote:
> On Fri, Oct 30, 2015 at 12:09:58PM +0100, Gerd Hoffmann wrote:
> > Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
> > Reviewed-by: Peter Lieven <pl@kamp.de>
> > ---
> >  include/qemu/buffer.h | 10 ++++++++++
> >  util/buffer.c         | 14 ++++++++++++++
> >  2 files changed, 24 insertions(+)
> > 
> > diff --git a/include/qemu/buffer.h b/include/qemu/buffer.h
> > index 0710e16..f53ee9e 100644
> > --- a/include/qemu/buffer.h
> > +++ b/include/qemu/buffer.h
> > @@ -127,4 +127,14 @@ uint8_t *buffer_end(Buffer *buffer);
> >   */
> >  gboolean buffer_empty(Buffer *buffer);
> >  
> > +/**
> > + * buffer_move_empty:
> > + * @to: destination buffer object
> > + * @from: source buffer object
> > + *
> > + * Moves buffer, without copying data.  'to' buffer must be empty.
> 
> Hmm, on the one hand the code do 'assert(to->offset) == 0', but on
> the other hand it does 'g_free(to->buffer)' as if it expected the
> buffer to have existing data.
> 
> If you just remove the 'assert', then there's no real requirement
> for 'to' to be empty, as we'd do the right thin in free'ing the
> data. Then we can just change this API doc to say
> 
>     "Any existing data in "to" will be freed"

Ignore this comment. I forget that we have an optimization to keep
buffer around when buffer_reset(), as distinct from buffer_free().

Reviewed-by: Daniel Berrange <berrange@redhat.com>

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [PATCH 00/19] buffer/vnc: improve vnc buffer hsndling

[Daniel P. Berrange]

On Fri, Oct 30, 2015 at 12:09:55PM +0100, Gerd Hoffmann wrote:
>   Hi,
> 
> This series has a bunch of improvements in the vnc buffer handling,
> to reduce the memory footprint.  Some of the changes are applied to
> the buffer helper functions which Daniel separated out of the vnc code
> recently.
> 
> Most patches have been on the list before, based on a older version of
> Daniel's "separate out buffer code" patches.  Now I finally managed to
> rebase and adapt the changes to the latest version which landed in
> master meanwhile.  I don't expect major issues showing up here and plan
> to have a pull request with this in time for 2.5-rc0.

I think this series looks good for a merge in 2.5

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [PATCH 19/19] buffer: allow a buffer to shrink gracefully

[Peter Lieven]

Am 30.10.2015 um 12:10 schrieb Gerd Hoffmann:
> From: Peter Lieven <pl@kamp.de>
>
> the idea behind this patch is to allow the buffer to shrink, but
> make this a seldom operation. The buffers average size is measured
> exponentionally smoothed with am alpha of 1/128.
>
> Signed-off-by: Peter Lieven <pl@kamp.de>
> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
> ---
>   include/qemu/buffer.h |  1 +
>   util/buffer.c         | 34 ++++++++++++++++++++++++++++------
>   2 files changed, 29 insertions(+), 6 deletions(-)
>
> diff --git a/include/qemu/buffer.h b/include/qemu/buffer.h
> index 0a69b3a..dead9b7 100644
> --- a/include/qemu/buffer.h
> +++ b/include/qemu/buffer.h
> @@ -37,6 +37,7 @@ struct Buffer {
>       char *name;
>       size_t capacity;
>       size_t offset;
> +    uint64_t avg_size;
>       uint8_t *buffer;
>   };
>   
> diff --git a/util/buffer.c b/util/buffer.c
> index fe5a44e..5461f86 100644
> --- a/util/buffer.c
> +++ b/util/buffer.c
> @@ -23,6 +23,7 @@
>   
>   #define BUFFER_MIN_INIT_SIZE     4096
>   #define BUFFER_MIN_SHRINK_SIZE  65536
> +#define BUFFER_AVG_SIZE_SHIFT       7
>   
>   static size_t buffer_req_size(Buffer *buffer, size_t len)
>   {
> @@ -37,6 +38,11 @@ static void buffer_adj_size(Buffer *buffer, size_t len)
>       buffer->buffer = g_realloc(buffer->buffer, buffer->capacity);
>       trace_buffer_resize(buffer->name ?: "unnamed",
>                           old, buffer->capacity);
> +
> +    /* make it even harder for the buffer to shrink, reset average size
> +     * to currenty capacity if it is larger than the average. */
> +    buffer->avg_size = MAX(buffer->avg_size,
> +                           buffer->capacity << BUFFER_AVG_SIZE_SHIFT);
>   }
>   
>   void buffer_init(Buffer *buffer, const char *name, ...)
> @@ -48,16 +54,30 @@ void buffer_init(Buffer *buffer, const char *name, ...)
>       va_end(ap);
>   }
>   
> +static uint64_t buffer_get_avg_size(Buffer *buffer)
> +{
> +    return buffer->avg_size >> BUFFER_AVG_SIZE_SHIFT;
> +}
> +
>   void buffer_shrink(Buffer *buffer)
>   {
> -    /*
> -     * Only shrink in case the used size is *much* smaller than the
> -     * capacity, to avoid bumping up & down the buffers all the time.
> +    size_t new;
> +
> +    /* Calculate the average size of the buffer as
> +     * avg_size = avg_size * ( 1 - a ) + required_size * a
> +     * where a is 1 / 2 ^ QIO_BUFFER_AVG_SIZE_SHIFT. */
> +    buffer->avg_size *= (1 << BUFFER_AVG_SIZE_SHIFT) - 1;
> +    buffer->avg_size >>= BUFFER_AVG_SIZE_SHIFT;
> +    buffer->avg_size += buffer_req_size(buffer, 0);
> +
> +    /* And then only shrink if the average size of the buffer is much
> +     * too big, to avoid bumping up & down the buffers all the time.
>        * realloc() isn't exactly cheap ...
>        */
> -    if (buffer->offset < (buffer->capacity >> 3) &&
> -        buffer->capacity > BUFFER_MIN_SHRINK_SIZE) {
> -        return;
> +    new = buffer_req_size(buffer, buffer_get_avg_size(buffer));
> +    if (new < buffer->capacity >> 3 &&
> +        new >= BUFFER_MIN_SHRINK_SIZE) {
> +        buffer_adj_size(buffer, buffer_get_avg_size(buffer));
>       }
>   
>       buffer_adj_size(buffer, 0);
> @@ -83,6 +103,7 @@ uint8_t *buffer_end(Buffer *buffer)
>   void buffer_reset(Buffer *buffer)
>   {
>       buffer->offset = 0;
> +    buffer_shrink(buffer);
>   }
>   
>   void buffer_free(Buffer *buffer)
> @@ -107,6 +128,7 @@ void buffer_advance(Buffer *buffer, size_t len)
>       memmove(buffer->buffer, buffer->buffer + len,
>               (buffer->offset - len));
>       buffer->offset -= len;
> +    buffer_shrink(buffer);
>   }
>   
>   void buffer_move_empty(Buffer *to, Buffer *from)

This isn't the last version of this patch. Please have a look at:

https://github.com/plieven/qemu/commit/e599748ab1ef381d4b1c88bf1ea1454dd89353fb

Peter

Re: [Qemu-devel] [PATCH 00/19] buffer/vnc: improve vnc buffer hsndling

[Peter Lieven]

Am 30.10.2015 um 12:09 schrieb Gerd Hoffmann:
>    Hi,
>
> This series has a bunch of improvements in the vnc buffer handling,
> to reduce the memory footprint.  Some of the changes are applied to
> the buffer helper functions which Daniel separated out of the vnc code
> recently.
>
> Most patches have been on the list before, based on a older version of
> Daniel's "separate out buffer code" patches.  Now I finally managed to
> rebase and adapt the changes to the latest version which landed in
> master meanwhile.  I don't expect major issues showing up here and plan
> to have a pull request with this in time for 2.5-rc0.
>
> Peter, if you have anything pending not yet in here please rebase and
> resend.
>
> please review,
>    Gerd
>
> Gerd Hoffmann (14):
>    buffer: add buffer_init
>    buffer: add buffer_move_empty
>    buffer: add buffer_move
>    buffer: add buffer_shrink
>    buffer: add tracing
>    vnc: attach names to buffers
>    vnc: kill jobs queue buffer
>    vnc-jobs: move buffer reset, use new buffer move
>    vnc: zap dead code
>    vnc: add vnc_width+vnc_height helpers
>    vnc: factor out vnc_update_server_surface
>    vnc: use vnc_{width,height} in vnc_set_area_dirty
>    vnc: only alloc server surface with clients connected
>    vnc: fix local state init
All above: Reviewed-by: Peter Lieven <pl@kamp.de>

>
> Peter Lieven (5):
>    buffer: make the Buffer capacity increase in powers of two
>    vnc: recycle empty vs->output buffer
>    buffer: factor out buffer_req_size
>    buffer: factor out buffer_adj_size
>    buffer: allow a buffer to shrink gracefully

The last Patch isn't the latest version. I have one with improved comments here:

https://github.com/plieven/qemu/commit/e599748ab1ef381d4b1c88bf1ea1454dd89353fb

I also had another improvement:

https://github.com/plieven/qemu/commit/2b4180a5f4ec29a59de692e9aa512b7b4d8023e7

which limits the number of memmove operation in qio_buffer_advance.

Peter

Re: [Qemu-devel] [PATCH 00/19] buffer/vnc: improve vnc buffer hsndling

[Gerd Hoffmann]

> >    buffer: allow a buffer to shrink gracefully
> 
> The last Patch isn't the latest version. I have one with improved comments here:
> 
> https://github.com/plieven/qemu/commit/e599748ab1ef381d4b1c88bf1ea1454dd89353fb
> 
> I also had another improvement:
> 
> https://github.com/plieven/qemu/commit/2b4180a5f4ec29a59de692e9aa512b7b4d8023e7
> 
> which limits the number of memmove operation in qio_buffer_advance.

Can you git-send-email them to the list for review?

thanks,
  Gerd

Re: [Qemu-devel] [PATCH 00/19] buffer/vnc: improve vnc buffer hsndling

[Peter Lieven]

Am 03.11.2015 um 09:23 schrieb Gerd Hoffmann:
>>>     buffer: allow a buffer to shrink gracefully
>> The last Patch isn't the latest version. I have one with improved comments here:
>>
>> https://github.com/plieven/qemu/commit/e599748ab1ef381d4b1c88bf1ea1454dd89353fb
>>
>> I also had another improvement:
>>
>> https://github.com/plieven/qemu/commit/2b4180a5f4ec29a59de692e9aa512b7b4d8023e7
>>
>> which limits the number of memmove operation in qio_buffer_advance.
> Can you git-send-email them to the list for review?

done

Peter