From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:33941) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YaFvS-0004Kb-VU for qemu-devel@nongnu.org; Mon, 23 Mar 2015 23:54:51 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1YaFvO-0006A5-Od for qemu-devel@nongnu.org; Mon, 23 Mar 2015 23:54:46 -0400 Received: from ozlabs.org ([103.22.144.67]:52327) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YaFvO-00064y-4I for qemu-devel@nongnu.org; Mon, 23 Mar 2015 23:54:42 -0400 Date: Tue, 24 Mar 2015 13:33:14 +1100 From: David Gibson Message-ID: <20150324023314.GZ25043@voom.fritz.box> References: <1424883128-9841-1-git-send-email-dgilbert@redhat.com> <1424883128-9841-36-git-send-email-dgilbert@redhat.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="rYRijwbmzeJBcrGE" Content-Disposition: inline In-Reply-To: <1424883128-9841-36-git-send-email-dgilbert@redhat.com> Subject: Re: [Qemu-devel] [PATCH v5 35/45] postcopy_ram.c: place_page and helpers List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Dr. David Alan Gilbert (git)" Cc: aarcange@redhat.com, yamahata@private.email.ne.jp, quintela@redhat.com, qemu-devel@nongnu.org, amit.shah@redhat.com, pbonzini@redhat.com, yanghy@cn.fujitsu.com --rYRijwbmzeJBcrGE Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Feb 25, 2015 at 04:51:58PM +0000, Dr. David Alan Gilbert (git) wrot= e: > From: "Dr. David Alan Gilbert" >=20 > postcopy_place_page (etc) provide a way for postcopy to place a page > into guests memory atomically (using the copy ioctl on the ufd). >=20 > Signed-off-by: Dr. David Alan Gilbert > --- > include/migration/migration.h | 2 + > include/migration/postcopy-ram.h | 16 ++++++ > migration/postcopy-ram.c | 113 +++++++++++++++++++++++++++++++++= +++++- > trace-events | 1 + > 4 files changed, 130 insertions(+), 2 deletions(-) >=20 > diff --git a/include/migration/migration.h b/include/migration/migration.h > index b1c7cad..139bb1b 100644 > --- a/include/migration/migration.h > +++ b/include/migration/migration.h > @@ -94,6 +94,8 @@ struct MigrationIncomingState { > QEMUFile *return_path; > QemuMutex rp_mutex; /* We send replies from multiple threads= */ > PostcopyPMI postcopy_pmi; > + void *postcopy_tmp_page; > + long postcopy_place_skipped; /* Check for incorrect place = ops */ > }; > =20 > MigrationIncomingState *migration_incoming_get_current(void); > diff --git a/include/migration/postcopy-ram.h b/include/migration/postcop= y-ram.h > index fbb2a93..3d30280 100644 > --- a/include/migration/postcopy-ram.h > +++ b/include/migration/postcopy-ram.h > @@ -80,4 +80,20 @@ void postcopy_discard_send_chunk(MigrationState *ms, P= ostcopyDiscardState *pds, > void postcopy_discard_send_finish(MigrationState *ms, > PostcopyDiscardState *pds); > =20 > +/* > + * Place a page (from) at (host) efficiently > + * There are restrictions on how 'from' must be mapped, in general be= st > + * to use other postcopy_ routines to allocate. > + * returns 0 on success > + */ > +int postcopy_place_page(MigrationIncomingState *mis, void *host, void *f= rom, > + long bitmap_offset, bool all_zero); > + > +/* > + * Allocate a page of memory that can be mapped at a later point in time > + * using postcopy_place_page > + * Returns: Pointer to allocated page > + */ > +void *postcopy_get_tmp_page(MigrationIncomingState *mis); > + > #endif > diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c > index 33dd332..86fa5a0 100644 > --- a/migration/postcopy-ram.c > +++ b/migration/postcopy-ram.c > @@ -197,7 +197,6 @@ static PostcopyPMIState postcopy_pmi_get_state_nolock( > } > =20 > /* Retrieve the state of the given page */ > -__attribute__ (( unused )) /* Until later in patch series */ > static PostcopyPMIState postcopy_pmi_get_state(MigrationIncomingState *m= is, > size_t bitmap_index) > { > @@ -213,7 +212,6 @@ static PostcopyPMIState postcopy_pmi_get_state(Migrat= ionIncomingState *mis, > * Set the page state to the given state if the previous state was as ex= pected > * Return the actual previous state. > */ > -__attribute__ (( unused )) /* Until later in patch series */ > static PostcopyPMIState postcopy_pmi_change_state(MigrationIncomingState= *mis, > size_t bitmap_index, > PostcopyPMIState expected_sta= te, > @@ -477,6 +475,7 @@ static int cleanup_area(const char *block_name, void = *host_addr, > int postcopy_ram_incoming_init(MigrationIncomingState *mis, size_t ram_p= ages) > { > postcopy_pmi_init(mis, ram_pages); > + mis->postcopy_place_skipped =3D -1; > =20 > if (qemu_ram_foreach_block(init_area, mis)) { > return -1; > @@ -495,6 +494,10 @@ int postcopy_ram_incoming_cleanup(MigrationIncomingS= tate *mis) > return -1; > } > =20 > + if (mis->postcopy_tmp_page) { > + munmap(mis->postcopy_tmp_page, getpagesize()); > + mis->postcopy_tmp_page =3D NULL; > + } > return 0; > } > =20 > @@ -561,6 +564,100 @@ int postcopy_ram_enable_notify(MigrationIncomingSta= te *mis) > return 0; > } > =20 > +/* > + * Place a host page (from) at (host) tomically s/tomically/atomically/ > + * There are restrictions on how 'from' must be mapped, in general be= st > + * to use other postcopy_ routines to allocate. > + * all_zero: Hint that the page being placed is 0 throughout > + * returns 0 on success > + * bitmap_offset: Index into the migration bitmaps > + * > + * State changes: > + * none -> received > + * requested -> received (ack) > + * > + * Note the UF thread is also updating the state, and maybe none->reques= ted > + * at the same time. Hrm.. these facts do tend me towards thinking that separate and explicit requested and received bits will be clearer than treating it as an enum state variable > + */ > +int postcopy_place_page(MigrationIncomingState *mis, void *host, void *f= rom, > + long bitmap_offset, bool all_zero) > +{ > + PostcopyPMIState old_state, tmp_state, new_state; > + > + if (!all_zero) { > + struct uffdio_copy copy_struct; > + > + copy_struct.dst =3D (uint64_t)(uintptr_t)host; > + copy_struct.src =3D (uint64_t)(uintptr_t)from; > + copy_struct.len =3D getpagesize(); > + copy_struct.mode =3D 0; > + > + /* copy also acks to the kernel waking the stalled thread up > + * TODO: We can inhibit that ack and only do it if it was reques= ted > + * which would be slightly cheaper, but we'd have to be careful > + * of the order of updating our page state. > + */ > + if (ioctl(mis->userfault_fd, UFFDIO_COPY, ©_struct)) { > + int e =3D errno; > + error_report("%s: %s copy host: %p from: %p pmi=3D%d", > + __func__, strerror(e), host, from, > + postcopy_pmi_get_state(mis, bitmap_offset)); > + > + return -e; > + } > + } else { > + struct uffdio_zeropage zero_struct; > + > + zero_struct.range.start =3D (uint64_t)(uintptr_t)host; > + zero_struct.range.len =3D getpagesize(); > + zero_struct.mode =3D 0; > + > + if (ioctl(mis->userfault_fd, UFFDIO_ZEROPAGE, &zero_struct)) { > + int e =3D errno; > + error_report("%s: %s zero host: %p from: %p pmi=3D%d", > + __func__, strerror(e), host, from, > + postcopy_pmi_get_state(mis, bitmap_offset)); > + > + return -e; > + } > + } > + > + bitmap_offset &=3D ~(mis->postcopy_pmi.host_bits-1); > + new_state =3D POSTCOPY_PMI_RECEIVED; > + tmp_state =3D postcopy_pmi_get_state(mis, bitmap_offset); > + do { > + old_state =3D tmp_state; > + tmp_state =3D postcopy_pmi_change_state(mis, bitmap_offset, old_= state, > + new_state); > + } while (old_state !=3D tmp_state); Yeah.. see treating the state as two separate booleans, here you'd just need to update received, without caring about whether requested has changed. > + trace_postcopy_place_page(bitmap_offset, host, all_zero, old_state); > + > + return 0; > +} > + > +/* > + * Returns a target page of memory that can be mapped at a later point i= n time > + * using postcopy_place_page > + * The same address is used repeatedly, postcopy_place_page just takes t= he > + * backing page away. > + * Returns: Pointer to allocated page > + * > + */ > +void *postcopy_get_tmp_page(MigrationIncomingState *mis) > +{ > + if (!mis->postcopy_tmp_page) { > + mis->postcopy_tmp_page =3D mmap(NULL, getpagesize(), > + PROT_READ | PROT_WRITE, MAP_PRIVATE | > + MAP_ANONYMOUS, -1, 0); > + if (!mis->postcopy_tmp_page) { > + perror("mapping postcopy tmp page"); > + return NULL; > + } > + } > + > + return mis->postcopy_tmp_page; > +} > + > #else > /* No target OS support, stubs just fail */ > bool postcopy_ram_supported_by_host(void) > @@ -608,6 +705,18 @@ int postcopy_ram_enable_notify(MigrationIncomingStat= e *mis) > { > assert(0); > } > + > +int postcopy_place_page(MigrationIncomingState *mis, void *host, void *f= rom, > + long bitmap_offset, bool all_zero) > +{ > + assert(0); > +} > + > +void *postcopy_get_tmp_page(MigrationIncomingState *mis) > +{ > + assert(0); > +} > + > #endif > =20 > /* ---------------------------------------------------------------------= ---- */ > diff --git a/trace-events b/trace-events > index 781cf5c..16a91d9 100644 > --- a/trace-events > +++ b/trace-events > @@ -1497,6 +1497,7 @@ rdma_start_outgoing_migration_after_rdma_source_ini= t(void) "" > postcopy_discard_send_finish(const char *ramblock, int nwords, int ncmds= ) "%s mask words sent=3D%d in %d commands" > postcopy_cleanup_area(const char *ramblock, void *host_addr, size_t offs= et, size_t length) "%s: %p offset=3D%zx length=3D%zx" > postcopy_init_area(const char *ramblock, void *host_addr, size_t offset,= size_t length) "%s: %p offset=3D%zx length=3D%zx" > +postcopy_place_page(unsigned long offset, void *host_addr, bool all_zero= , int old_state) "offset=3D%lx host=3D%p all_zero=3D%d old_state=3D%d" > =20 > # kvm-all.c > kvm_ioctl(int type, void *arg) "type 0x%x, arg %p" --=20 David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson --rYRijwbmzeJBcrGE Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBAgAGBQJVEMzqAAoJEGw4ysog2bOSkHUP/1KTyshA1rolQKGoyupwFh4o voSE0ObljOVPoJS8INcma/tVBREntLm5oPdMgAmCnMujEdSVJTNNkR/Xaaus1CgD wJm7ZBghX9aUIcOJo0S1kaurGFfJOx+VmSdXHeLm6nalr4GVhFFqEGQeyP7S/2eB CiU7Ae3DlIo028cbeTTBSMQSwcMdz45iZdlpD6M/BO4adedElucwdy3LfpYCUhs3 RT1GQDKs9pUn191pSKiq4n3bUhPSwmgsoGisP2/GuYb1KK5HDAcIEjdoEJN6kJO2 pefNJlruEDOwi1VJAaJDLzbsewI/TaJKFjLBbYgCjP/zsrI7ckOOmozQBbRwp3UI l9YSX2hR+dPlmydUetAWJ8hDPJF6n/zKmKuE1Hk+3/ejBAsi29F2ZSrb/Wp59aKf 0fJJZl5MpAI/g/nM4xiT6pDayEKPIbYDqazXbommOtSfhQmeykFO8X+U47Jmwkyc 7O8vFNCyRDkM9QaAcYxK+nejRlNcQ2pvdcPkG5KkuU8SV9WuILaFx9tsOEDWI8CL Xjk7Ma9tVvLuGq4ADrKsG06fPeCEhDbR6IR0pQXtkDbtYbIVhVckbmrMobiMledU QfRb7Qw1vhTE4Cs9zv7ix8TSFvAnZmBv4XvB9+LL1adhOTcAGzsNKykSRQwhuPdr gaefSZ4qTtsYbdtB4ytE =3UA1 -----END PGP SIGNATURE----- --rYRijwbmzeJBcrGE--