* [Qemu-devel] Can qemu reopen image files? @ 2016-12-18 23:52 Christopher Pereira 2016-12-19 1:07 ` Fam Zheng 0 siblings, 1 reply; 11+ messages in thread From: Christopher Pereira @ 2016-12-18 23:52 UTC (permalink / raw) To: qemu-devel Hi, We are doing a "qemu-img convert" operation (qcow2, with compression) to shorten the backing-chain (in the middle of the backing-chain). In order to force qemu to reopen files, we do a save and restore operation. Is there a faster way to reopen image files using virsh or qemu? Best regards, Christopher ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Qemu-devel] Can qemu reopen image files? 2016-12-18 23:52 [Qemu-devel] Can qemu reopen image files? Christopher Pereira @ 2016-12-19 1:07 ` Fam Zheng 2016-12-19 13:55 ` Stefan Hajnoczi 0 siblings, 1 reply; 11+ messages in thread From: Fam Zheng @ 2016-12-19 1:07 UTC (permalink / raw) To: Christopher Pereira; +Cc: qemu-devel On Sun, 12/18 20:52, Christopher Pereira wrote: > Hi, > > We are doing a "qemu-img convert" operation (qcow2, with compression) to > shorten the backing-chain (in the middle of the backing-chain). > In order to force qemu to reopen files, we do a save and restore operation. > Is there a faster way to reopen image files using virsh or qemu? No, don't use qemu-img when the image is in use by QEMU. You want to use "block-commit" command provided by QMP. Fam ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Qemu-devel] Can qemu reopen image files? 2016-12-19 1:07 ` Fam Zheng @ 2016-12-19 13:55 ` Stefan Hajnoczi 2016-12-19 15:03 ` Christopher Pereira 0 siblings, 1 reply; 11+ messages in thread From: Stefan Hajnoczi @ 2016-12-19 13:55 UTC (permalink / raw) To: Fam Zheng; +Cc: Christopher Pereira, qemu-devel [-- Attachment #1: Type: text/plain, Size: 695 bytes --] On Mon, Dec 19, 2016 at 09:07:43AM +0800, Fam Zheng wrote: > On Sun, 12/18 20:52, Christopher Pereira wrote: > > Hi, > > > > We are doing a "qemu-img convert" operation (qcow2, with compression) to > > shorten the backing-chain (in the middle of the backing-chain). > > In order to force qemu to reopen files, we do a save and restore operation. > > Is there a faster way to reopen image files using virsh or qemu? > > No, don't use qemu-img when the image is in use by QEMU. You want to use > "block-commit" command provided by QMP. It's worth being more explicit here: You will corrupt the image file if you access it with another program while QEMU is using it! Stefan [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 455 bytes --] ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Qemu-devel] Can qemu reopen image files? 2016-12-19 13:55 ` Stefan Hajnoczi @ 2016-12-19 15:03 ` Christopher Pereira 2016-12-19 15:48 ` Eric Blake 0 siblings, 1 reply; 11+ messages in thread From: Christopher Pereira @ 2016-12-19 15:03 UTC (permalink / raw) To: Stefan Hajnoczi, Fam Zheng; +Cc: qemu-devel Hi Fam, Stefan, Thanks for answering. We use "qemu-img convert" to convert a image in the middle of the chain, not the active one. Those images (and the previous ones in the chain) are read-only and there should be no risk in converting them: E.g.: for the following chain: base --> snap1 ---> snap2 ---> snap3 (active) we do "qemu-img convert" on snap2 (readonly), generating a snap2' with the same content as snap2. Then we do the rebase while the VM is suspended to make sure the image files are reopened. Please confirm if I'm missing something here. We are not using block-commit since we want to have more control (keep the base snapshot unmodified, use compression, etc). Best regards, Christopher On 19-Dec-16 10:55, Stefan Hajnoczi wrote: > On Mon, Dec 19, 2016 at 09:07:43AM +0800, Fam Zheng wrote: >> On Sun, 12/18 20:52, Christopher Pereira wrote: >>> Hi, >>> >>> We are doing a "qemu-img convert" operation (qcow2, with compression) to >>> shorten the backing-chain (in the middle of the backing-chain). >>> In order to force qemu to reopen files, we do a save and restore operation. >>> Is there a faster way to reopen image files using virsh or qemu? >> No, don't use qemu-img when the image is in use by QEMU. You want to use >> "block-commit" command provided by QMP. > It's worth being more explicit here: > > You will corrupt the image file if you access it with another program > while QEMU is using it! > > Stefan ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Qemu-devel] Can qemu reopen image files? 2016-12-19 15:03 ` Christopher Pereira @ 2016-12-19 15:48 ` Eric Blake 2016-12-19 16:24 ` Christopher Pereira 0 siblings, 1 reply; 11+ messages in thread From: Eric Blake @ 2016-12-19 15:48 UTC (permalink / raw) To: Christopher Pereira, Stefan Hajnoczi, Fam Zheng; +Cc: qemu-devel [-- Attachment #1: Type: text/plain, Size: 2970 bytes --] On 12/19/2016 09:03 AM, Christopher Pereira wrote: > Hi Fam, Stefan, > > Thanks for answering. > > We use "qemu-img convert" to convert a image in the middle of the chain, > not the active one. > Those images (and the previous ones in the chain) are read-only and > there should be no risk in converting them: > > E.g.: for the following chain: > > base --> snap1 ---> snap2 ---> snap3 (active) We typically write that as: base <- snap1 <- snap2 <- snap3 where the <- operator can be pronounced "backs", and where the direction of the arrow shows that it is snap1 that depends on base (and not base that depends on snap1). > > we do "qemu-img convert" on snap2 (readonly), generating a snap2' with > the same content as snap2. That part is fine, but why not use QMP to let qemu generate a single file with the same contents, instead of delegating to qemu-img? > > Then we do the rebase while the VM is suspended to make sure the image > files are reopened. > > Please confirm if I'm missing something here. That part is where you are liable to break things. Qemu does NOT have a graceful way to reopen the backing chain, so rebasing snap3 to point to snap2' behind qemu's back is asking for problems. Since qemu may be caching things it has already learned about snap2, you have invalidated that cached data by making snap3 point to snap2', but have no way to force qemu to reread the backing chain to start reading from snap2'. But if you would use qemu block-commit to merge "base <- snap1 <- snap2" into "base'", then the block-commit command will gracefully take care of rewriting the backing image of snap3 to now point to base. You achieve the same result, but without the need for an external qemu-img call, without the need to pause the guest, and the only thing you have to be careful of is dealing with the difference in file names. Or, if you don't want to merge into "base'", you can use block-stream to merge the other direction, so that "base <- snap1 <- snap2" is converted into "snap2'" - but that depends on patches that were only barely added in qemu 2.8 (intermediate block-commit has existed a lot longer than intermediate block-stream). But the point remains that you are still using qemu to do the work, and therefore with no external qemu-img process interfering with the chain, you don't need any guest downtime or any risk of breaking qemu operation by invalidating data it may have cached. > > We are not using block-commit since we want to have more control (keep > the base snapshot unmodified, use compression, etc). If block-commit and block-stream don't have enough power to do what you want, then we should patch them to expose that power, rather than worrying about how to use qemu-img to modify the backing chain behind qemu's back. -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 604 bytes --] ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Qemu-devel] Can qemu reopen image files? 2016-12-19 15:48 ` Eric Blake @ 2016-12-19 16:24 ` Christopher Pereira 2016-12-28 18:51 ` Christopher Pereira 2016-12-29 13:39 ` Eric Blake 0 siblings, 2 replies; 11+ messages in thread From: Christopher Pereira @ 2016-12-19 16:24 UTC (permalink / raw) To: Eric Blake, Stefan Hajnoczi, Fam Zheng; +Cc: qemu-devel Hi Eric, Thanks for your great answer. On 19-Dec-16 12:48, Eric Blake wrote: > >> Then we do the rebase while the VM is suspended to make sure the image >> files are reopened. > That part is where you are liable to break things. Qemu does NOT have a > graceful way to reopen the backing chain, so rebasing snap3 to point to > snap2' behind qemu's back is asking for problems. Since qemu may be > caching things it has already learned about snap2, you have invalidated > that cached data by making snap3 point to snap2', but have no way to > force qemu to reread the backing chain to start reading from snap2'. We are actually doing a save, rebase and restore to reopen the backing chain. We only touch files (rebase) while the VM is down. Can you please confirm this is 100% safe? > Or, if you don't want to merge into "base'", you can use block-stream to > merge the other direction, so that "base <- snap1 <- snap2" is converted > into "snap2'" - but that depends on patches that were only barely added > in qemu 2.8 (intermediate block-commit has existed a lot longer than > intermediate block-stream). But the point remains that you are still > using qemu to do the work, and therefore with no external qemu-img > process interfering with the chain, you don't need any guest downtime or > any risk of breaking qemu operation by invalidating data it may have cached. Right. Since images are backed up remotely, we don't want to merge into base nor touch the backing chain at all (only the active snapshot should be modified). This is to keep things simple and avoid to re-syncs of images (remote backups). Besides, we don't want to merge the whole backing chain, but an intermediate point, so it seems that the clean way is to use the "intermediate block-stream" feature. We didn't try it, because when we researched we got the impression that the patches were not stable yet or not included in the qemu versions shipped with CentOS, so we went with 'qemu-img convert' because we needed something known, simple and stable (we are dealing with critical information for gov. orgs.). > If block-commit and block-stream don't have enough power to do what you > want, then we should patch them to expose that power, rather than > worrying about how to use qemu-img to modify the backing chain behind > qemu's back. "intermediate block-stream" seems to be the right solution for our use case. Does it also allow QCOW2 compression? Compression is interesting, especially when files are sync'ed via network. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Qemu-devel] Can qemu reopen image files? 2016-12-19 16:24 ` Christopher Pereira @ 2016-12-28 18:51 ` Christopher Pereira 2016-12-29 13:42 ` Eric Blake 2016-12-29 13:39 ` Eric Blake 1 sibling, 1 reply; 11+ messages in thread From: Christopher Pereira @ 2016-12-28 18:51 UTC (permalink / raw) To: Eric Blake, Stefan Hajnoczi, Fam Zheng; +Cc: qemu-devel Hi Eric, There is something I don't understand. We are doing: "virsh save", "qemu-img convert", "qemu-img rebase" and "virsh restore". We only touch the backing chain by doing the rebase while the VM is down. Is there any chance this procedure can destroy data? If so, is there any difference between shutting down and just saving/restoring the VM? Maybe save/restore keeps a cache? Best regards, Christopher. On 19-Dec-16 13:24, Christopher Pereira wrote: > Hi Eric, > > Thanks for your great answer. > > On 19-Dec-16 12:48, Eric Blake wrote: > >> >>> Then we do the rebase while the VM is suspended to make sure the image >>> files are reopened. >> That part is where you are liable to break things. Qemu does NOT have a >> graceful way to reopen the backing chain, so rebasing snap3 to point to >> snap2' behind qemu's back is asking for problems. Since qemu may be >> caching things it has already learned about snap2, you have invalidated >> that cached data by making snap3 point to snap2', but have no way to >> force qemu to reread the backing chain to start reading from snap2'. > We are actually doing a save, rebase and restore to reopen the backing > chain. > We only touch files (rebase) while the VM is down. > Can you please confirm this is 100% safe? > >> Or, if you don't want to merge into "base'", you can use block-stream to >> merge the other direction, so that "base <- snap1 <- snap2" is converted >> into "snap2'" - but that depends on patches that were only barely added >> in qemu 2.8 (intermediate block-commit has existed a lot longer than >> intermediate block-stream). But the point remains that you are still >> using qemu to do the work, and therefore with no external qemu-img >> process interfering with the chain, you don't need any guest downtime or >> any risk of breaking qemu operation by invalidating data it may have >> cached. > Right. Since images are backed up remotely, we don't want to merge > into base nor touch the backing chain at all (only the active snapshot > should be modified). This is to keep things simple and avoid to > re-syncs of images (remote backups). > > Besides, we don't want to merge the whole backing chain, but an > intermediate point, so it seems that the clean way is to use the > "intermediate block-stream" feature. > > We didn't try it, because when we researched we got the impression > that the patches were not stable yet or not included in the qemu > versions shipped with CentOS, so we went with 'qemu-img convert' > because we needed something known, simple and stable (we are dealing > with critical information for gov. orgs.). > >> If block-commit and block-stream don't have enough power to do what you >> want, then we should patch them to expose that power, rather than >> worrying about how to use qemu-img to modify the backing chain behind >> qemu's back. > "intermediate block-stream" seems to be the right solution for our use > case. > Does it also allow QCOW2 compression? > Compression is interesting, especially when files are sync'ed via > network. > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Qemu-devel] Can qemu reopen image files? 2016-12-28 18:51 ` Christopher Pereira @ 2016-12-29 13:42 ` Eric Blake 2016-12-29 13:52 ` Eric Blake 0 siblings, 1 reply; 11+ messages in thread From: Eric Blake @ 2016-12-29 13:42 UTC (permalink / raw) To: Christopher Pereira, Stefan Hajnoczi, Fam Zheng; +Cc: qemu-devel [-- Attachment #1: Type: text/plain, Size: 1328 bytes --] On 12/28/2016 12:51 PM, Christopher Pereira wrote: > Hi Eric, > > There is something I don't understand. > > We are doing: "virsh save", "qemu-img convert", "qemu-img rebase" and > "virsh restore". > We only touch the backing chain by doing the rebase while the VM is down. > Is there any chance this procedure can destroy data? Since there are never two active writers to the file (you have made sure that qemu-img and qemu are not both holding the same file open at once), there is no chance for corruption on that front. There's still a possibility of corruption if you botch the instructions, and don't rebuild the chain in such a way that the guest would see the same data, but that's harder to say without seeing an actual transcript. > If so, is there any difference between shutting down and just > saving/restoring the VM? Shutting down vs. saving/restoring shouldn't matter other than the length of time that the guest is offline (including rebooting time with a shut down, which is not present for an incoming migration restoration). > Maybe save/restore keeps a cache? No. Both shutting down and saving end the qemu process, and therefore flush everything to the disk. -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 604 bytes --] ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Qemu-devel] Can qemu reopen image files? 2016-12-29 13:42 ` Eric Blake @ 2016-12-29 13:52 ` Eric Blake 2016-12-29 13:55 ` Eric Blake 0 siblings, 1 reply; 11+ messages in thread From: Eric Blake @ 2016-12-29 13:52 UTC (permalink / raw) To: Christopher Pereira, Stefan Hajnoczi, Fam Zheng; +Cc: qemu-devel [-- Attachment #1: Type: text/plain, Size: 1710 bytes --] On 12/29/2016 07:42 AM, Eric Blake wrote: > On 12/28/2016 12:51 PM, Christopher Pereira wrote: >> Hi Eric, >> >> There is something I don't understand. >> >> We are doing: "virsh save", "qemu-img convert", "qemu-img rebase" and >> "virsh restore". >> We only touch the backing chain by doing the rebase while the VM is down. >> Is there any chance this procedure can destroy data? > > Since there are never two active writers to the file (you have made sure > that qemu-img and qemu are not both holding the same file open at once), > there is no chance for corruption on that front. There's still a > possibility of corruption if you botch the instructions, and don't > rebuild the chain in such a way that the guest would see the same data, > but that's harder to say without seeing an actual transcript. Oh, and since you mention virsh, that means you also have libvirt involved - anything you do to the backing chain with qemu-img is now behind libvirt's back, so you must also make sure that you are using 'virsh edit' if necessary to update the domain XML to match the changes you made to the backing chain. Again, it's hard to say whether that is a factor in your case without an actual transcript (eventually, libvirt plans to track full backing chains even for offline domains; at the moment, however, it only tracks full backing chains for running domains and only the top-most image of an offline domain, which means qemu-img changes to intermediate parts of a backing chain don't affect offline domain XML now, but might in some future libvirt version). -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 604 bytes --] ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Qemu-devel] Can qemu reopen image files? 2016-12-29 13:52 ` Eric Blake @ 2016-12-29 13:55 ` Eric Blake 0 siblings, 0 replies; 11+ messages in thread From: Eric Blake @ 2016-12-29 13:55 UTC (permalink / raw) To: Christopher Pereira, Stefan Hajnoczi, Fam Zheng; +Cc: qemu-devel [-- Attachment #1: Type: text/plain, Size: 2469 bytes --] On 12/29/2016 07:52 AM, Eric Blake wrote: > On 12/29/2016 07:42 AM, Eric Blake wrote: >> On 12/28/2016 12:51 PM, Christopher Pereira wrote: >>> Hi Eric, >>> >>> There is something I don't understand. >>> >>> We are doing: "virsh save", "qemu-img convert", "qemu-img rebase" and >>> "virsh restore". >>> We only touch the backing chain by doing the rebase while the VM is down. >>> Is there any chance this procedure can destroy data? >> >> Since there are never two active writers to the file (you have made sure >> that qemu-img and qemu are not both holding the same file open at once), >> there is no chance for corruption on that front. There's still a >> possibility of corruption if you botch the instructions, and don't >> rebuild the chain in such a way that the guest would see the same data, >> but that's harder to say without seeing an actual transcript. > > Oh, and since you mention virsh, that means you also have libvirt > involved - anything you do to the backing chain with qemu-img is now > behind libvirt's back, so you must also make sure that you are using > 'virsh edit' if necessary to update the domain XML to match the changes > you made to the backing chain. Again, it's hard to say whether that is > a factor in your case without an actual transcript (eventually, libvirt > plans to track full backing chains even for offline domains; at the > moment, however, it only tracks full backing chains for running domains > and only the top-most image of an offline domain, which means qemu-img > changes to intermediate parts of a backing chain don't affect offline > domain XML now, but might in some future libvirt version). And an addendum - ideally, libvirt will eventually expose enough commands for doing image chain manipulation, so that you do it ALL through virsh commands (whether the domain is running or offline), rather than ever having to touch qemu-img yourself. It may be that libvirt will use qemu-img under the hood to do the same things you were doing, but one of the goals of libvirt is to abstract away the need for you to use more than one tool - it's just that in the particular case of image backing chains, it is complex enough that libvirt has not yet reached its ideal of being able to completely manage the backing chain without you having to do external work. -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 604 bytes --] ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Qemu-devel] Can qemu reopen image files? 2016-12-19 16:24 ` Christopher Pereira 2016-12-28 18:51 ` Christopher Pereira @ 2016-12-29 13:39 ` Eric Blake 1 sibling, 0 replies; 11+ messages in thread From: Eric Blake @ 2016-12-29 13:39 UTC (permalink / raw) To: Christopher Pereira, Stefan Hajnoczi, Fam Zheng; +Cc: qemu-devel [-- Attachment #1: Type: text/plain, Size: 5092 bytes --] On 12/19/2016 10:24 AM, Christopher Pereira wrote: > Hi Eric, > > Thanks for your great answer. > > On 19-Dec-16 12:48, Eric Blake wrote: > >> >>> Then we do the rebase while the VM is suspended to make sure the image >>> files are reopened. >> That part is where you are liable to break things. Qemu does NOT have a >> graceful way to reopen the backing chain, so rebasing snap3 to point to >> snap2' behind qemu's back is asking for problems. Since qemu may be >> caching things it has already learned about snap2, you have invalidated >> that cached data by making snap3 point to snap2', but have no way to >> force qemu to reread the backing chain to start reading from snap2'. > We are actually doing a save, rebase and restore to reopen the backing > chain. In which case, after the save, qemu is not running, and therefore anything you do with qemu-img is not operating behind qemu's back, and then the restore should work okay if the guest-visible data is identical even though the backing chain has changed. It is unsafe to edit a backing chain WHILE qemu is also using the same chain, but it sounds like you are taking care (at the expense of guest downtime) to have no qemu process active during the time of your edits, which can indeed be safe. > We only touch files (rebase) while the VM is down. > Can you please confirm this is 100% safe? I can't confirm 100% safety without seeing actual transcripts of what you did, of course, but the idea is sound. But your original problem statement said the VM is suspended (which implies there is still a qemu process with the chain open, where resuming the VM involves another command to that qemu process to resume the cpus - not safe), not saved (which implies that you have created a migration stream and ended the qemu process, and resuming the VM involves starting a new qemu process on an incoming migration stream - can be safe). > >> Or, if you don't want to merge into "base'", you can use block-stream to >> merge the other direction, so that "base <- snap1 <- snap2" is converted >> into "snap2'" - but that depends on patches that were only barely added >> in qemu 2.8 (intermediate block-commit has existed a lot longer than >> intermediate block-stream). But the point remains that you are still >> using qemu to do the work, and therefore with no external qemu-img >> process interfering with the chain, you don't need any guest downtime or >> any risk of breaking qemu operation by invalidating data it may have >> cached. > Right. Since images are backed up remotely, we don't want to merge into > base nor touch the backing chain at all (only the active snapshot should > be modified). This is to keep things simple and avoid to re-syncs of > images (remote backups). > > Besides, we don't want to merge the whole backing chain, but an > intermediate point, so it seems that the clean way is to use the > "intermediate block-stream" feature. > > We didn't try it, because when we researched we got the impression that > the patches were not stable yet or not included in the qemu versions > shipped with CentOS, so we went with 'qemu-img convert' because we > needed something known, simple and stable (we are dealing with critical > information for gov. orgs.). CentOS tracks downstream Red Hat RHEL releases, which indeed tend to use older (but proven stable) qemu releases + patch backports as appropriate. But CentOS is self-supported, so you have no control over which patches are backported, if you don't upgrade to a full Red Hat support contract. On this list, we tend to focus on upstream development, questions about particular features (what has been backported or not) are better directed to downstream vendors that maintain those backports. So whether a particular CentOS build includes intermediate block-stream capabilities is more a question to CentOS lists or Red Hat support contracts, not this list. > >> If block-commit and block-stream don't have enough power to do what you >> want, then we should patch them to expose that power, rather than >> worrying about how to use qemu-img to modify the backing chain behind >> qemu's back. > "intermediate block-stream" seems to be the right solution for our use > case. > Does it also allow QCOW2 compression? Hmm. In the current upstream qapi/block-core.json file, the DriveBackup and 'BlockdevBackup' structs include an optional 'compress':'bool' parameter, but 'block-stream' does not yet do so. It's probably a simple enough patch to add, by basing it on what was done for blockdev-backup, but no one has written it yet. It's certainly worth adding though, as it (should be) fairly obvious that a streaming operation is going to be a write-once action where compressing the resulting sectors being written probably has a benefit. > Compression is interesting, especially when files are sync'ed via network. > > -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 604 bytes --] ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2016-12-29 13:55 UTC | newest] Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2016-12-18 23:52 [Qemu-devel] Can qemu reopen image files? Christopher Pereira 2016-12-19 1:07 ` Fam Zheng 2016-12-19 13:55 ` Stefan Hajnoczi 2016-12-19 15:03 ` Christopher Pereira 2016-12-19 15:48 ` Eric Blake 2016-12-19 16:24 ` Christopher Pereira 2016-12-28 18:51 ` Christopher Pereira 2016-12-29 13:42 ` Eric Blake 2016-12-29 13:52 ` Eric Blake 2016-12-29 13:55 ` Eric Blake 2016-12-29 13:39 ` Eric Blake
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.