From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5B5F7C433F5 for ; Mon, 29 Nov 2021 18:02:52 +0000 (UTC) Received: from localhost ([::1]:54336 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mrkzL-0001Zt-6C for qemu-devel@archiver.kernel.org; Mon, 29 Nov 2021 13:02:51 -0500 Received: from eggs.gnu.org ([209.51.188.92]:37818) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mrkpd-0002nc-Hj for qemu-devel@nongnu.org; Mon, 29 Nov 2021 12:52:49 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:24800) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mrkpa-0006Vo-5C for qemu-devel@nongnu.org; Mon, 29 Nov 2021 12:52:48 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1638208364; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=SvZiFmFGZ11cJ6iPrJyJMgbZ8A+xZlxsE0AlRe1LzDg=; b=GVAUej+PPRA0vQ4TYjx3eYqiAZLzry7RDBfN7zGy5P+NST1vHywPQHaXKo2nv/kZsRsOUE g1yWlLZIkR6qUX8E56f0lCORgi0BN29u58okOPhuyNI+G5D9FA+6X4trqpv0+yFoUyG8dA /wMAe8GzrQ8f2azNb4FL05gEVl4eFJM= Received: from mail-wr1-f70.google.com (mail-wr1-f70.google.com [209.85.221.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-15-m8BrxPjfODuW1TO-EtjCnQ-1; Mon, 29 Nov 2021 12:52:43 -0500 X-MC-Unique: m8BrxPjfODuW1TO-EtjCnQ-1 Received: by mail-wr1-f70.google.com with SMTP id y4-20020adfd084000000b00186b16950f3so3082448wrh.14 for ; Mon, 29 Nov 2021 09:52:43 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=SvZiFmFGZ11cJ6iPrJyJMgbZ8A+xZlxsE0AlRe1LzDg=; b=WFjsyUFjH/P4kjpN9UBFvDhoIHQIOgI87pU9HUECtClK2vauR1v7mTehT/G9C1YqVs +FRajMbKWUp0Q4fUZlHV/3CE3aYTOIZmeTtKUeDd6u1uaDYU+fpxMc/ttHBO12MH3XvQ trSuESgQIsPLdQpMCdC0XkMtkDWmQpe84RS3gV2WCJQJeTExjv1ppDaxL8HIcbOjxD0/ QsGx6G6/3x4CikNh4+0VusztttZiX7kUNb/cSGXm2zt+r31WnnJs0ucPICu7JwFD8r4J pTTK311clLitMDdwzoqYNBpoSTLTiEVIDIm92u9AwsYhQexFcG8+/hDZfpzA24I2OnHn cvqw== X-Gm-Message-State: AOAM532D1P4EfgO58UP9IdVB6MYXgvDWO+nfMFGzECj5fRdt5W0yi1fs 7Oai5Z6c7QekF/Xi8XogoExXQc7qbwlI7Q7+VRWEVzAJMcNaLlKxyUMV2gHw/pkEUZO1ux1u9iO dKfr4FdxloG5suc0= X-Received: by 2002:a5d:5151:: with SMTP id u17mr36560124wrt.126.1638208362198; Mon, 29 Nov 2021 09:52:42 -0800 (PST) X-Google-Smtp-Source: ABdhPJzWaZtY/zY8+G+Q3InDYF7ca86JZfBErTsdAdgqCkdLsYvRHSnvfLARHb1ROYIv8esBRNiStw== X-Received: by 2002:a5d:5151:: with SMTP id u17mr36560096wrt.126.1638208361847; Mon, 29 Nov 2021 09:52:41 -0800 (PST) Received: from work-vm (cpc109025-salf6-2-0-cust480.10-2.cable.virginm.net. [82.30.61.225]) by smtp.gmail.com with ESMTPSA id m7sm14575063wml.38.2021.11.29.09.52.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 29 Nov 2021 09:52:41 -0800 (PST) Date: Mon, 29 Nov 2021 17:52:39 +0000 From: "Dr. David Alan Gilbert" To: Juan Quintela Subject: Re: [PATCH v3 11/23] multifd: Move iov from pages to params Message-ID: References: <20211124100617.19786-1-quintela@redhat.com> <20211124100617.19786-12-quintela@redhat.com> MIME-Version: 1.0 In-Reply-To: <20211124100617.19786-12-quintela@redhat.com> User-Agent: Mutt/2.1.3 (2021-09-10) Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=dgilbert@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Received-SPF: pass client-ip=170.10.129.124; envelope-from=dgilbert@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -34 X-Spam_score: -3.5 X-Spam_bar: --- X-Spam_report: (-3.5 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.717, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Leonardo Bras , qemu-devel@nongnu.org, Peter Xu Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" * Juan Quintela (quintela@redhat.com) wrote: > This will allow us to reduce the number of system calls on the next patch. > > Signed-off-by: Juan Quintela Leo: Does this make your zerocopy any harder? Dave > --- > migration/multifd.h | 8 ++++++-- > migration/multifd.c | 34 ++++++++++++++++++++++++---------- > 2 files changed, 30 insertions(+), 12 deletions(-) > > diff --git a/migration/multifd.h b/migration/multifd.h > index e57adc783b..c3f18af364 100644 > --- a/migration/multifd.h > +++ b/migration/multifd.h > @@ -62,8 +62,6 @@ typedef struct { > uint64_t packet_num; > /* offset of each page */ > ram_addr_t *offset; > - /* pointer to each page */ > - struct iovec *iov; > RAMBlock *block; > } MultiFDPages_t; > > @@ -110,6 +108,10 @@ typedef struct { > uint64_t num_pages; > /* syncs main thread and channels */ > QemuSemaphore sem_sync; > + /* buffers to send */ > + struct iovec *iov; > + /* number of iovs used */ > + uint32_t iovs_num; > /* used for compression methods */ > void *data; > } MultiFDSendParams; > @@ -149,6 +151,8 @@ typedef struct { > uint64_t num_pages; > /* syncs main thread and channels */ > QemuSemaphore sem_sync; > + /* buffers to recv */ > + struct iovec *iov; > /* used for de-compression methods */ > void *data; > } MultiFDRecvParams; > diff --git a/migration/multifd.c b/migration/multifd.c > index 0533da154a..37487fd01c 100644 > --- a/migration/multifd.c > +++ b/migration/multifd.c > @@ -86,7 +86,16 @@ static void nocomp_send_cleanup(MultiFDSendParams *p, Error **errp) > */ > static int nocomp_send_prepare(MultiFDSendParams *p, Error **errp) > { > - p->next_packet_size = p->pages->num * qemu_target_page_size(); > + MultiFDPages_t *pages = p->pages; > + size_t page_size = qemu_target_page_size(); > + > + for (int i = 0; i < p->pages->num; i++) { > + p->iov[p->iovs_num].iov_base = pages->block->host + pages->offset[i]; > + p->iov[p->iovs_num].iov_len = page_size; > + p->iovs_num++; > + } > + > + p->next_packet_size = p->pages->num * page_size; > p->flags |= MULTIFD_FLAG_NOCOMP; > return 0; > } > @@ -104,7 +113,7 @@ static int nocomp_send_prepare(MultiFDSendParams *p, Error **errp) > */ > static int nocomp_send_write(MultiFDSendParams *p, uint32_t used, Error **errp) > { > - return qio_channel_writev_all(p->c, p->pages->iov, used, errp); > + return qio_channel_writev_all(p->c, p->iov, p->iovs_num, errp); > } > > /** > @@ -146,13 +155,18 @@ static void nocomp_recv_cleanup(MultiFDRecvParams *p) > static int nocomp_recv_pages(MultiFDRecvParams *p, Error **errp) > { > uint32_t flags = p->flags & MULTIFD_FLAG_COMPRESSION_MASK; > + size_t page_size = qemu_target_page_size(); > > if (flags != MULTIFD_FLAG_NOCOMP) { > error_setg(errp, "multifd %d: flags received %x flags expected %x", > p->id, flags, MULTIFD_FLAG_NOCOMP); > return -1; > } > - return qio_channel_readv_all(p->c, p->pages->iov, p->pages->num, errp); > + for (int i = 0; i < p->pages->num; i++) { > + p->iov[i].iov_base = p->pages->block->host + p->pages->offset[i]; > + p->iov[i].iov_len = page_size; > + } > + return qio_channel_readv_all(p->c, p->iov, p->pages->num, errp); > } > > static MultiFDMethods multifd_nocomp_ops = { > @@ -242,7 +256,6 @@ static MultiFDPages_t *multifd_pages_init(size_t size) > MultiFDPages_t *pages = g_new0(MultiFDPages_t, 1); > > pages->allocated = size; > - pages->iov = g_new0(struct iovec, size); > pages->offset = g_new0(ram_addr_t, size); > > return pages; > @@ -254,8 +267,6 @@ static void multifd_pages_clear(MultiFDPages_t *pages) > pages->allocated = 0; > pages->packet_num = 0; > pages->block = NULL; > - g_free(pages->iov); > - pages->iov = NULL; > g_free(pages->offset); > pages->offset = NULL; > g_free(pages); > @@ -365,8 +376,6 @@ static int multifd_recv_unfill_packet(MultiFDRecvParams *p, Error **errp) > return -1; > } > p->pages->offset[i] = offset; > - p->pages->iov[i].iov_base = block->host + offset; > - p->pages->iov[i].iov_len = page_size; > } > > return 0; > @@ -470,8 +479,6 @@ int multifd_queue_page(QEMUFile *f, RAMBlock *block, ram_addr_t offset) > > if (pages->block == block) { > pages->offset[pages->num] = offset; > - pages->iov[pages->num].iov_base = block->host + offset; > - pages->iov[pages->num].iov_len = qemu_target_page_size(); > pages->num++; > > if (pages->num < pages->allocated) { > @@ -564,6 +571,8 @@ void multifd_save_cleanup(void) > p->packet_len = 0; > g_free(p->packet); > p->packet = NULL; > + g_free(p->iov); > + p->iov = NULL; > multifd_send_state->ops->send_cleanup(p, &local_err); > if (local_err) { > migrate_set_error(migrate_get_current(), local_err); > @@ -651,6 +660,7 @@ static void *multifd_send_thread(void *opaque) > uint32_t used = p->pages->num; > uint64_t packet_num = p->packet_num; > uint32_t flags = p->flags; > + p->iovs_num = 0; > > if (used) { > ret = multifd_send_state->ops->send_prepare(p, &local_err); > @@ -919,6 +929,7 @@ int multifd_save_setup(Error **errp) > p->packet->version = cpu_to_be32(MULTIFD_VERSION); > p->name = g_strdup_printf("multifdsend_%d", i); > p->tls_hostname = g_strdup(s->hostname); > + p->iov = g_new0(struct iovec, page_count); > socket_send_channel_create(multifd_new_send_channel_async, p); > } > > @@ -1018,6 +1029,8 @@ int multifd_load_cleanup(Error **errp) > p->packet_len = 0; > g_free(p->packet); > p->packet = NULL; > + g_free(p->iov); > + p->iov = NULL; > multifd_recv_state->ops->recv_cleanup(p); > } > qemu_sem_destroy(&multifd_recv_state->sem_sync); > @@ -1158,6 +1171,7 @@ int multifd_load_setup(Error **errp) > + sizeof(uint64_t) * page_count; > p->packet = g_malloc0(p->packet_len); > p->name = g_strdup_printf("multifdrecv_%d", i); > + p->iov = g_new0(struct iovec, page_count); > } > > for (i = 0; i < thread_count; i++) { > -- > 2.33.1 > -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK