All of lore.kernel.org
 help / color / mirror / Atom feed
* "X86_PV_VCPU_MSRS record truncated" during domain restore (was: Re: [qubes-users] DispVM Doesn't work)
       [not found]   ` <345d27df-0424-c7bd-a099-d933b0a45c18@gmail.com>
@ 2016-07-20 23:20     ` Marek Marczykowski-Górecki
       [not found]     ` <20160720232009.GA24847@mail-itl>
  1 sibling, 0 replies; 6+ messages in thread
From: Marek Marczykowski-Górecki @ 2016-07-20 23:20 UTC (permalink / raw)
  To: Massimo Colombi; +Cc: xen-devel

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Wed, Jul 20, 2016 at 02:33:20PM +0200, Massimo Colombi wrote:
> I retried (it's not the first time) to regenerate new savefile, but DispVM
> doesn't work.
> I attach the results.

For me it looks like a bug in savefile handling. Moving to xen-devel. Any
idea?
Background info: it's about restoring a domain through libvirt->libxl
(virsh restore equivalent). Xen 4.6.1. Full error:

> 2016-07-20 14:23:01 CEST xc: error: X86_PV_VCPU_MSRS record truncated: length 8, min 9: Internal error
> 2016-07-20 14:23:01 CEST xc: error: Restore failed (0 = Success): Internal error
> 2016-07-20 14:23:01 CEST libxl: error: libxl_stream_read.c:749:libxl__xc_domain_restore_done: restoring domain: Successo
> 2016-07-20 14:23:01 CEST libxl: error: libxl_create.c:1145:domcreate_rebuild_done: cannot (re-)build domain: -3


If relevant, here is fragment of /proc/cpuinfo (just one core):
processor   : 0
vendor_id   : AuthenticAMD
cpu family  : 22
model           : 48
model name  : AMD A8-6410 APU with AMD Radeon R5 Graphics
stepping        : 1
microcode   : 0x7030105
cpu MHz         : 1996.290
cache size  : 2048 KB
physical id     : 0
siblings        : 4
core id         : 0
cpu cores   : 4
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu de tsc msr pae mce cx8 apic mca cmov pat clflush
mmx fxsr sse sse2 ht
syscall nx mmxext fxsr_opt lm constant_tsc rep_good nopl nonstop_tsc
extd_apicid eagerfpu pni
pclmulqdq ssse3 cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c
rdrand hypervisor lahf_lm
cmp_legacy extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch
perfctr_nb bpext perfctr_l2
arat cpb hw_pstate vmmcall bmi1 xsaveopt
bugs            : fxsave_leak
bogomips        : 3992.58
TLB size        : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management: ts ttp tm 100mhzsteps hwpstate cpb [12] [13]


> Best regards,
>  Massimo
> 
> On 07/20/2016 01:37 PM, Marek Marczykowski-Górecki wrote:
> > Try `qvm-create-default-dvm --default-template` to regenerate new savefile.
> > It looks like your savefile is somehow broken.
> 

> [user@dom0 logs]$ qvm-create-default-dvm --default-template
> --> Creating volatile image: /var/lib/qubes/appvms/fedora-23-dvm/volatile.img...
> --> Loading the VM (type = AppVM)...
> --> Starting Qubes DB...
> --> Setting Qubes DB info for the VM...
> --> Updating firewall rules...
> --> Starting the VM...
> --> Starting Qubes GUId...
> Connecting to VM's GUI agent: ..................connected
> Waiting for DVM fedora-23-dvm ...
> /qubes-used-mem
> Disco scollegato correttamente
> 
> DVM boot complete, memory used=303380. Saving image...
> 
> 
> 
> Domain fedora-23-dvm saved to /var/lib/qubes/appvms/fedora-23-dvm/dvm-savefile
> 
> DVM savefile created successfully.
> [user@dom0 logs]$ echo xterm | /usr/lib/qubes/qfile-daemon-dvm qubes.VMShell dom0 DEFAULT red
> time=1469017378.68, qfile-daemon-dvm init
> time=1469017378.92, creating DispVM
> time=1469017380.48, collection loaded
> time=1469017380.49, VM created
> time=1469017380.59, VM starting
> time=1469017380.6, creating config file
> time=1469017380.86, calling restore
> Traceback (most recent call last):
>   File "/usr/lib/qubes/qfile-daemon-dvm", line 200, in <module>
>     main()
>   File "/usr/lib/qubes/qfile-daemon-dvm", line 188, in main
>     dispvm = qfile.get_dvm()
>   File "/usr/lib/qubes/qfile-daemon-dvm", line 150, in get_dvm
>     return self.do_get_dvm()
>   File "/usr/lib/qubes/qfile-daemon-dvm", line 103, in do_get_dvm
>     dispvm.start()
>   File "/usr/lib64/python2.7/site-packages/qubes/modules/01QubesDisposableVm.py", line 193, in start
>     domain_config, libvirt.VIR_DOMAIN_SAVE_PAUSED)
>   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 4405, in restoreFlags
>     if ret == -1: raise libvirtError ('virDomainRestoreFlags() failed', conn=self)
> libvirt.libvirtError: internal error: libxenlight failed to restore domain 'disp1'
> 


- -- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJXkActAAoJENuP0xzK19csHZ4IAJF35BwBKbKzNFo0yOcTJxng
2NqVIUmHg3hgVZuNdkoNm09L7f5yIUai+0AqsFvM8BbPmwC9C6EBcu7FICMub/lT
NUbfyf4rW5YRuEVG+OB7+Zge4mz3++kb1cLqobn2vA5Z9ayCDW32Yq5yYFff9zd7
/qMtjuj35oG25fKEs1PIZbDtMkdnq2ef1rg7KDj695SpDSt0g3AadtTjnJVh7f2V
v7K6aUKR6O2xcHWADWhqxLNaKEBA8cd2yHeGYmbQwD9cICqQNVFwH/jIHWsAqGun
d/IKvRetZnLajFm42X7862UIgQWE1qTpqDXV6OCWT/KNh/cL4SLwn+0RW8RbfSQ=
=gdks
-----END PGP SIGNATURE-----

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: "X86_PV_VCPU_MSRS record truncated" during domain restore
       [not found]     ` <20160720232009.GA24847@mail-itl>
@ 2016-07-21  0:10       ` Andrew Cooper
  2016-07-21  8:39         ` Massimo Colombi
  0 siblings, 1 reply; 6+ messages in thread
From: Andrew Cooper @ 2016-07-21  0:10 UTC (permalink / raw)
  To: Marek Marczykowski-Górecki, Massimo Colombi; +Cc: xen-devel


[-- Attachment #1.1.1: Type: text/plain, Size: 2461 bytes --]

On 21/07/2016 00:20, Marek Marczykowski-Górecki wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> On Wed, Jul 20, 2016 at 02:33:20PM +0200, Massimo Colombi wrote:
>> I retried (it's not the first time) to regenerate new savefile, but DispVM
>> doesn't work.
>> I attach the results.
> For me it looks like a bug in savefile handling. Moving to xen-devel. Any
> idea?
> Background info: it's about restoring a domain through libvirt->libxl
> (virsh restore equivalent). Xen 4.6.1. Full error:
>
>> 2016-07-20 14:23:01 CEST xc: error: X86_PV_VCPU_MSRS record truncated: length 8, min 9: Internal error
>> 2016-07-20 14:23:01 CEST xc: error: Restore failed (0 = Success): Internal error

[EDIT] While writing the explanation below, I have spotted the bug.

The restore side found a record in the stream of type X86_PV_VCPU_MSRS
and 8 bytes long.

The X86_PV_VCPU_MSRS record is an 8 byte header, followed by 1 or more
bytes which are treated as an opaque blob, and handed back to Xen.  As
such, 8 bytes is insufficient to contain a nonzero sized payload.

Is it possible to do an `xl save` equivalent on the domain, and run
tools/python/scripts/verify-stream-v2 against the resulting file?  That
should identify whether it is a malformed X86_PV_VCPU_MSRS record but
otherwise intact stream, or whether the stream becomes corrupted elsewhere?

If not, another line if inquiry would be to instrument
tools/libxc/xc_sr_save_x86_pv.c write_one_vcpu_msrs() to identify what
XEN_DOMCTL_get_vcpu_msrs is returning on the source side (although this
function does deliberately check for a zero-sized payload and omit the
record).

The hypervisor side implementation is in xen/arch/x86/domctl.c at the
XEN_DOMCTL_get_vcpu_msrs case label.

Re-reading the write_one_vcpu_msrs(), there is an error.  We first query
Xen for the maximum number of MSRs it might potentially give to us,
which will return 4 on this hardware.  We then actually ask for the MSR
content, but Xen only writes non-zero MSRs into payload.

A domain which isn't using Debug Extensions will return 4 for the first
query (maximum number of MSRs), then 0 for the second query (actual MSR
content), and write a X86_PV_VCPU_MSRS record with 0 payload into the
stream, which the receiving side is objecting to.

I will make a patch series tomorrow to address this issue.  I think
there are similar issues for other records as well.

~Andrew


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 834 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: "X86_PV_VCPU_MSRS record truncated" during domain restore
  2016-07-21  0:10       ` "X86_PV_VCPU_MSRS record truncated" during domain restore Andrew Cooper
@ 2016-07-21  8:39         ` Massimo Colombi
  2016-07-21  8:53           ` Andrew Cooper
  0 siblings, 1 reply; 6+ messages in thread
From: Massimo Colombi @ 2016-07-21  8:39 UTC (permalink / raw)
  To: Andrew Cooper, Marek Marczykowski-Górecki; +Cc: xen-devel

This is the output of verify-stream-v2:

[user@dom0 scripts]$ sudo xl save fedora-23-dvm /fedora-23-dvm-savefile
Saving to /fedora-23-dvm-savefile new xl format (info 0x3/0x0/1991)
xc: info: Saving domain 4, type x86 PV
xc: Frames: 912384/912384  100%
xc: End of stream: 0/0    0%

[user@dom0 scripts]$ ./verify-stream-v2 -f xl -i /fedora-23-dvm-savefile
Stream Error:
Traceback (most recent call last):
   File "./verify-stream-v2", line 82, in read_stream
     VerifyLibxl(info, stream_read).verify()
   File "/usr/lib64/python2.7/site-packages/xen/migration/libxl.py", 
line 82, in verify
     while self.verify_record() != REC_TYPE_end:
   File "/usr/lib64/python2.7/site-packages/xen/migration/libxl.py", 
line 136, in verify_record
     record_verifiers[rtype](self, content[:length])
   File "/usr/lib64/python2.7/site-packages/xen/migration/libxl.py", 
line 155, in verify_record_libxc_context
     VerifyLibxc(self.info, self.read).verify()
   File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py", 
line 132, in verify
     while self.verify_record() != REC_TYPE_end:
   File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py", 
line 227, in verify_record
     record_verifiers[rtype](self, content[:length])
   File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py", 
line 429, in <lambda>
     VerifyLibxc.verify_record_x86_pv_vcpu_generic(s, x, "msrs"),
   File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py", 
line 323, in verify_record_x86_pv_vcpu_generic
     " bytes long" % (name, minsz))
RecordError: X86_PV_VCPU_msrs record length must be at least 8 bytes long


On 07/21/2016 02:10 AM, Andrew Cooper wrote:
> Is it possible to do an `xl save` equivalent on the domain, and run
> tools/python/scripts/verify-stream-v2 against the resulting file?  That
> should identify whether it is a malformed X86_PV_VCPU_MSRS record but
> otherwise intact stream, or whether the stream becomes corrupted elsewhere?

Thanks for your explanation of the bug.

Best regards,
  Massimo

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: "X86_PV_VCPU_MSRS record truncated" during domain restore
  2016-07-21  8:39         ` Massimo Colombi
@ 2016-07-21  8:53           ` Andrew Cooper
  2016-07-21  9:21             ` Massimo Colombi
  2016-07-22 15:07             ` Massimo Colombi
  0 siblings, 2 replies; 6+ messages in thread
From: Andrew Cooper @ 2016-07-21  8:53 UTC (permalink / raw)
  To: Massimo Colombi, Marek Marczykowski-Górecki; +Cc: xen-devel

On 21/07/2016 09:39, Massimo Colombi wrote:
> This is the output of verify-stream-v2:
>
> [user@dom0 scripts]$ sudo xl save fedora-23-dvm /fedora-23-dvm-savefile
> Saving to /fedora-23-dvm-savefile new xl format (info 0x3/0x0/1991)
> xc: info: Saving domain 4, type x86 PV
> xc: Frames: 912384/912384  100%
> xc: End of stream: 0/0    0%
>
> [user@dom0 scripts]$ ./verify-stream-v2 -f xl -i /fedora-23-dvm-savefile
> Stream Error:
> Traceback (most recent call last):
>   File "./verify-stream-v2", line 82, in read_stream
>     VerifyLibxl(info, stream_read).verify()
>   File "/usr/lib64/python2.7/site-packages/xen/migration/libxl.py",
> line 82, in verify
>     while self.verify_record() != REC_TYPE_end:
>   File "/usr/lib64/python2.7/site-packages/xen/migration/libxl.py",
> line 136, in verify_record
>     record_verifiers[rtype](self, content[:length])
>   File "/usr/lib64/python2.7/site-packages/xen/migration/libxl.py",
> line 155, in verify_record_libxc_context
>     VerifyLibxc(self.info, self.read).verify()
>   File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py",
> line 132, in verify
>     while self.verify_record() != REC_TYPE_end:
>   File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py",
> line 227, in verify_record
>     record_verifiers[rtype](self, content[:length])
>   File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py",
> line 429, in <lambda>
>     VerifyLibxc.verify_record_x86_pv_vcpu_generic(s, x, "msrs"),
>   File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py",
> line 323, in verify_record_x86_pv_vcpu_generic
>     " bytes long" % (name, minsz))
> RecordError: X86_PV_VCPU_msrs record length must be at least 8 bytes long
>
>
> On 07/21/2016 02:10 AM, Andrew Cooper wrote:
>> Is it possible to do an `xl save` equivalent on the domain, and run
>> tools/python/scripts/verify-stream-v2 against the resulting file?  That
>> should identify whether it is a malformed X86_PV_VCPU_MSRS record but
>> otherwise intact stream, or whether the stream becomes corrupted
>> elsewhere?
>
> Thanks for your explanation of the bug.

I should also improve the reported error message.

Do you mind rerunning with an extra -v to dump a list of the records
found in the stream?

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: "X86_PV_VCPU_MSRS record truncated" during domain restore
  2016-07-21  8:53           ` Andrew Cooper
@ 2016-07-21  9:21             ` Massimo Colombi
  2016-07-22 15:07             ` Massimo Colombi
  1 sibling, 0 replies; 6+ messages in thread
From: Massimo Colombi @ 2016-07-21  9:21 UTC (permalink / raw)
  To: Andrew Cooper, Marek Marczykowski-Górecki; +Cc: xen-devel

[-- Attachment #1: Type: text/plain, Size: 171 bytes --]

OK. I attach verbose output.


On 07/21/2016 10:53 AM, Andrew Cooper wrote:
> Do you mind rerunning with an extra -v to dump a list of the records
> found in the stream?


[-- Attachment #2: stream.txt --]
[-- Type: text/plain, Size: 2047 bytes --]

[user@dom0 scripts]$ ./verify-stream-v2  -f xl -v -i /tmp-savefile 
Processed xl header
Libxl Header: little endian
Libxl Record: Libxc context, length 0
Libxc Image Header: little endian
Domain Header: x86 PV from Xen 4.6
Libxc Record: x86 PV info, length 8
  64bit guest, 4 levels of pagetables
Libxc Record: x86 PV P2M frames, length 14264
  Start pfn 0x0, End 0xdebff
Squashed 891 Page Data records together
Libxc Record: TSC info, length 24
  Mode 2, 0 kHz, 0 ns, incarnation 1
Libxc Record: Shared info, length 4096
Libxc Record: x86 PV vcpu basic, length 5176
  vcpu0 basic context, 5168 bytes
Libxc Record: x86 PV vcpu extended, length 64
  vcpu0 extended context, 56 bytes
Libxc Record: x86 PV vcpu xsave, length 856
  vcpu0 xsave context, 848 bytes
Libxc Record: x86 PV vcpu msrs, length 8
Stream Error:
Traceback (most recent call last):
  File "./verify-stream-v2", line 82, in read_stream
    VerifyLibxl(info, stream_read).verify()
  File "/usr/lib64/python2.7/site-packages/xen/migration/libxl.py", line 82, in verify
    while self.verify_record() != REC_TYPE_end:
  File "/usr/lib64/python2.7/site-packages/xen/migration/libxl.py", line 136, in verify_record
    record_verifiers[rtype](self, content[:length])
  File "/usr/lib64/python2.7/site-packages/xen/migration/libxl.py", line 155, in verify_record_libxc_context
    VerifyLibxc(self.info, self.read).verify()
  File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py", line 132, in verify
    while self.verify_record() != REC_TYPE_end:
  File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py", line 227, in verify_record
    record_verifiers[rtype](self, content[:length])
  File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py", line 429, in <lambda>
    VerifyLibxc.verify_record_x86_pv_vcpu_generic(s, x, "msrs"),
  File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py", line 323, in verify_record_x86_pv_vcpu_generic
    " bytes long" % (name, minsz))
RecordError: X86_PV_VCPU_msrs record length must be at least 8 bytes long


[-- Attachment #3: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: "X86_PV_VCPU_MSRS record truncated" during domain restore
  2016-07-21  8:53           ` Andrew Cooper
  2016-07-21  9:21             ` Massimo Colombi
@ 2016-07-22 15:07             ` Massimo Colombi
  1 sibling, 0 replies; 6+ messages in thread
From: Massimo Colombi @ 2016-07-22 15:07 UTC (permalink / raw)
  To: Andrew Cooper, Marek Marczykowski-Górecki; +Cc: xen-devel

The patches (that are attached to "[PATCH RFC 0/4] Fix issues with 
zero-length records in migration v2") work!

I patched locally qubes-builder to import your patches and recreate rpm 
files. These patches also work on Xen 4.6.1.

Best regards,

   Massimo

On 07/21/2016 10:53 AM, Andrew Cooper wrote:
> On 21/07/2016 09:39, Massimo Colombi wrote:
>> This is the output of verify-stream-v2:
>>
>> [user@dom0 scripts]$ sudo xl save fedora-23-dvm /fedora-23-dvm-savefile
>> Saving to /fedora-23-dvm-savefile new xl format (info 0x3/0x0/1991)
>> xc: info: Saving domain 4, type x86 PV
>> xc: Frames: 912384/912384  100%
>> xc: End of stream: 0/0    0%
>>
>> [user@dom0 scripts]$ ./verify-stream-v2 -f xl -i /fedora-23-dvm-savefile
>> Stream Error:
>> Traceback (most recent call last):
>>    File "./verify-stream-v2", line 82, in read_stream
>>      VerifyLibxl(info, stream_read).verify()
>>    File "/usr/lib64/python2.7/site-packages/xen/migration/libxl.py",
>> line 82, in verify
>>      while self.verify_record() != REC_TYPE_end:
>>    File "/usr/lib64/python2.7/site-packages/xen/migration/libxl.py",
>> line 136, in verify_record
>>      record_verifiers[rtype](self, content[:length])
>>    File "/usr/lib64/python2.7/site-packages/xen/migration/libxl.py",
>> line 155, in verify_record_libxc_context
>>      VerifyLibxc(self.info, self.read).verify()
>>    File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py",
>> line 132, in verify
>>      while self.verify_record() != REC_TYPE_end:
>>    File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py",
>> line 227, in verify_record
>>      record_verifiers[rtype](self, content[:length])
>>    File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py",
>> line 429, in <lambda>
>>      VerifyLibxc.verify_record_x86_pv_vcpu_generic(s, x, "msrs"),
>>    File "/usr/lib64/python2.7/site-packages/xen/migration/libxc.py",
>> line 323, in verify_record_x86_pv_vcpu_generic
>>      " bytes long" % (name, minsz))
>> RecordError: X86_PV_VCPU_msrs record length must be at least 8 bytes long
>>
>>
>> On 07/21/2016 02:10 AM, Andrew Cooper wrote:
>>> Is it possible to do an `xl save` equivalent on the domain, and run
>>> tools/python/scripts/verify-stream-v2 against the resulting file?  That
>>> should identify whether it is a malformed X86_PV_VCPU_MSRS record but
>>> otherwise intact stream, or whether the stream becomes corrupted
>>> elsewhere?
>> Thanks for your explanation of the bug.
> I should also improve the reported error message.
>
> Do you mind rerunning with an extra -v to dump a list of the records
> found in the stream?
>
> ~Andrew


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2016-07-22 15:07 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
     [not found] <nmncv2$m3$1@ger.gmane.org>
     [not found] ` <20160720113752.GT1161@mail-itl>
     [not found]   ` <345d27df-0424-c7bd-a099-d933b0a45c18@gmail.com>
2016-07-20 23:20     ` "X86_PV_VCPU_MSRS record truncated" during domain restore (was: Re: [qubes-users] DispVM Doesn't work) Marek Marczykowski-Górecki
     [not found]     ` <20160720232009.GA24847@mail-itl>
2016-07-21  0:10       ` "X86_PV_VCPU_MSRS record truncated" during domain restore Andrew Cooper
2016-07-21  8:39         ` Massimo Colombi
2016-07-21  8:53           ` Andrew Cooper
2016-07-21  9:21             ` Massimo Colombi
2016-07-22 15:07             ` Massimo Colombi

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.