* [bug] btrfs check clear-space-cache v1 corrupted file system
@ 2019-03-02 0:20 Chris Murphy
2019-03-02 1:05 ` Qu Wenruo
0 siblings, 1 reply; 10+ messages in thread
From: Chris Murphy @ 2019-03-02 0:20 UTC (permalink / raw)
To: Btrfs BTRFS
Problem happens with btrfsprogs 4.19.1
https://bugzilla.kernel.org/show_bug.cgi?id=202717
That bug report includes the trace in user space; but I've also
processed the resulting coredump file with gdb and attached that to
the bug report.
Before using --clear-space-cache v1, btrfs scrub and btrfs check came
up clean no errors. Following the crash, I compiled brfsprogs 4.20.2
and ran 'btrfs check' which finds corruption.
I've taken no further action, and the btrfs check output gives me no
useful plain language information what the next steps are. So I'm just
gonna leave it alone since it's a backup volume anyway.
--
Chris Murphy
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [bug] btrfs check clear-space-cache v1 corrupted file system
2019-03-02 0:20 [bug] btrfs check clear-space-cache v1 corrupted file system Chris Murphy
@ 2019-03-02 1:05 ` Qu Wenruo
2019-03-02 2:29 ` Chris Murphy
0 siblings, 1 reply; 10+ messages in thread
From: Qu Wenruo @ 2019-03-02 1:05 UTC (permalink / raw)
To: Chris Murphy, Btrfs BTRFS
[-- Attachment #1.1: Type: text/plain, Size: 973 bytes --]
On 2019/3/2 上午8:20, Chris Murphy wrote:
> Problem happens with btrfsprogs 4.19.1
>
> https://bugzilla.kernel.org/show_bug.cgi?id=202717
>
> That bug report includes the trace in user space; but I've also
> processed the resulting coredump file with gdb and attached that to
> the bug report.
>
> Before using --clear-space-cache v1, btrfs scrub and btrfs check came
> up clean no errors. Following the crash, I compiled brfsprogs 4.20.2
> and ran 'btrfs check' which finds corruption.
>
> I've taken no further action, and the btrfs check output gives me no
> useful plain language information what the next steps are. So I'm just
> gonna leave it alone since it's a backup volume anyway.
Is the backup volume binary dump of the original fs?
I'm interesting why clear space cache v1 doesn't work, to find out that,
we need tree dump of the original fs.
And is the original check done by latest btrfs-progs or v4.19.1?
Thanks,
Qu
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [bug] btrfs check clear-space-cache v1 corrupted file system
2019-03-02 1:05 ` Qu Wenruo
@ 2019-03-02 2:29 ` Chris Murphy
2019-03-02 2:54 ` Chris Murphy
2019-03-02 4:14 ` Qu Wenruo
0 siblings, 2 replies; 10+ messages in thread
From: Chris Murphy @ 2019-03-02 2:29 UTC (permalink / raw)
To: Qu Wenruo; +Cc: Chris Murphy, Btrfs BTRFS
On Fri, Mar 1, 2019 at 6:05 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2019/3/2 上午8:20, Chris Murphy wrote:
> > Problem happens with btrfsprogs 4.19.1
> >
> > https://bugzilla.kernel.org/show_bug.cgi?id=202717
> >
> > That bug report includes the trace in user space; but I've also
> > processed the resulting coredump file with gdb and attached that to
> > the bug report.
> >
> > Before using --clear-space-cache v1, btrfs scrub and btrfs check came
> > up clean no errors. Following the crash, I compiled brfsprogs 4.20.2
> > and ran 'btrfs check' which finds corruption.
> >
> > I've taken no further action, and the btrfs check output gives me no
> > useful plain language information what the next steps are. So I'm just
> > gonna leave it alone since it's a backup volume anyway.
>
> Is the backup volume binary dump of the original fs?
No. Only btrfs send/receive.
>
> I'm interesting why clear space cache v1 doesn't work, to find out that,
> we need tree dump of the original fs.
btrfs insp dump-t -t 2 ? Or the whole thing?
>
> And is the original check done by latest btrfs-progs or v4.19.1?
The original check is 4.19.1
The clear cache is 4.19.1
The subsequent check showing corruption is 4.20.2 but I've found that
it's the same output for 4.19.1 as well.
--
Chris Murphy
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [bug] btrfs check clear-space-cache v1 corrupted file system
2019-03-02 2:29 ` Chris Murphy
@ 2019-03-02 2:54 ` Chris Murphy
[not found] ` <CAJCQCtQaQ0XFUGFdWfgwmTSPTUShrZgXmsh55c-reS9iS+WpFg@mail.gmail.com>
2019-03-02 4:14 ` Qu Wenruo
1 sibling, 1 reply; 10+ messages in thread
From: Chris Murphy @ 2019-03-02 2:54 UTC (permalink / raw)
To: Chris Murphy; +Cc: Qu Wenruo, Btrfs BTRFS
OK this is screwy. The super has a totally different generation for
the root tree than any of the backup roots.
$ sudo btrfs rescue super -v /dev/mapper/sdd
All Devices:
Device: id = 1, name = /dev/mapper/sdc
Device: id = 2, name = /dev/mapper/sdd
Before Recovering:
[All good supers]:
device name = /dev/mapper/sdc
superblock bytenr = 65536
device name = /dev/mapper/sdc
superblock bytenr = 67108864
device name = /dev/mapper/sdc
superblock bytenr = 274877906944
device name = /dev/mapper/sdd
superblock bytenr = 65536
device name = /dev/mapper/sdd
superblock bytenr = 67108864
device name = /dev/mapper/sdd
superblock bytenr = 274877906944
[All bad supers]:
All supers are valid, no need to recover
$ sudo btrfs insp dump-s -f /dev/mapper/sdd
superblock: bytenr=65536, device=/dev/mapper/sdd
---------------------------------------------------------
csum_type 0 (crc32c)
csum_size 4
csum 0x312a8b5e [match]
bytenr 65536
flags 0x1
( WRITTEN )
magic _BHRfS_M [match]
fsid f5adc913-bbea-4340-8b5f-3411e2cda642
label second
generation 4514
root 29392896
sys_array_size 258
chunk_root_generation 3823
root_level 1
chunk_root 822549151744
chunk_root_level 1
log_root 0
log_root_transid 0
log_root_level 0
total_bytes 1500308553728
bytes_used 735327125504
sectorsize 4096
nodesize 16384
leafsize (deprecated) 16384
stripesize 4096
root_dir 6
num_devices 2
compat_flags 0x0
compat_ro_flags 0x0
incompat_flags 0x161
( MIXED_BACKREF |
BIG_METADATA |
EXTENDED_IREF |
SKINNY_METADATA )
cache_generation 4076
uuid_tree_generation 4076
dev_item.uuid 2fbaa023-7cf8-4f48-b4ed-c6e18c99d2bd
dev_item.fsid f5adc913-bbea-4340-8b5f-3411e2cda642 [match]
dev_item.type 0
dev_item.total_bytes 750154276864
dev_item.bytes_used 737702576128
dev_item.io_align 4096
dev_item.io_width 4096
dev_item.sector_size 4096
dev_item.devid 2
dev_item.dev_group 0
dev_item.seek_speed 0
dev_item.bandwidth 0
dev_item.generation 0
sys_chunk_array[2048]:
item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 20971520)
length 8388608 owner 2 stripe_len 65536 type SYSTEM|RAID1
io_align 65536 io_width 65536 sector_size 4096
num_stripes 2 sub_stripes 0
stripe 0 devid 2 offset 1048576
dev_uuid 2fbaa023-7cf8-4f48-b4ed-c6e18c99d2bd
stripe 1 devid 1 offset 20971520
dev_uuid d071b0ca-1d27-48cb-826e-63c5e9972339
item 1 key (FIRST_CHUNK_TREE CHUNK_ITEM 822549151744)
length 33554432 owner 2 stripe_len 65536 type SYSTEM|RAID1
io_align 65536 io_width 65536 sector_size 4096
num_stripes 2 sub_stripes 1
stripe 0 devid 2 offset 737670070272
dev_uuid 2fbaa023-7cf8-4f48-b4ed-c6e18c99d2bd
stripe 1 devid 1 offset 737689993216
dev_uuid d071b0ca-1d27-48cb-826e-63c5e9972339
backup_roots[4]:
backup 0:
backup_tree_root: 931053568 gen: 4075 level: 1
backup_chunk_root: 822549151744 gen: 3823 level: 1
backup_extent_root: 931086336 gen: 4075 level: 2
backup_fs_root: 537116672 gen: 3806 level: 0
backup_dev_root: 721174528 gen: 3823 level: 1
backup_csum_root: 932249600 gen: 4075 level: 2
backup_total_bytes: 1500308553728
backup_bytes_used: 735441764352
backup_num_devices: 2
backup 1:
backup_tree_root: 930086912 gen: 4076 level: 1
backup_chunk_root: 822549151744 gen: 3823 level: 1
backup_extent_root: 930447360 gen: 4076 level: 2
backup_fs_root: 537116672 gen: 3806 level: 0
backup_dev_root: 930856960 gen: 4076 level: 1
backup_csum_root: 932118528 gen: 4076 level: 2
backup_total_bytes: 1500308553728
backup_bytes_used: 735441764352
backup_num_devices: 2
backup 2:
backup_tree_root: 930070528 gen: 4073 level: 1
backup_chunk_root: 822549151744 gen: 3823 level: 1
backup_extent_root: 930365440 gen: 4073 level: 2
backup_fs_root: 537116672 gen: 3806 level: 0
backup_dev_root: 721174528 gen: 3823 level: 1
backup_csum_root: 931086336 gen: 4073 level: 2
backup_total_bytes: 1500308553728
backup_bytes_used: 735441764352
backup_num_devices: 2
backup 3:
backup_tree_root: 930086912 gen: 4074 level: 1
backup_chunk_root: 822549151744 gen: 3823 level: 1
backup_extent_root: 930447360 gen: 4074 level: 2
backup_fs_root: 537116672 gen: 3806 level: 0
backup_dev_root: 721174528 gen: 3823 level: 1
backup_csum_root: 932052992 gen: 4074 level: 2
backup_total_bytes: 1500308553728
backup_bytes_used: 735441764352
backup_num_devices: 2
$
Super says root tree generation 4514; but the most recent backup says
4076. What?
# btrfs insp dump-t -b 930086912 /dev/mapper/sdd
btrfs-progs v4.19.1
node 930086912 level 1 items 19 free 474 generation 4076 owner ROOT_TREE
fs uuid f5adc913-bbea-4340-8b5f-3411e2cda642
chunk uuid 4af0d265-00fa-4a2b-9ea9-59c06a372284
key (EXTENT_TREE ROOT_ITEM 0) block 931102720 gen 4076
key (279 INODE_ITEM 0) block 535216128 gen 3802
key (307 INODE_ITEM 0) block 715111579648 gen 3511
key (368 EXTENT_DATA 0) block 715103928320 gen 3510
key (430 INODE_ITEM 0) block 715111612416 gen 3511
key (491 EXTENT_DATA 0) block 715111677952 gen 3511
key (553 INODE_ITEM 0) block 715111841792 gen 3511
key (652 EXTENT_DATA 0) block 714560716800 gen 3523
key (711 INODE_ITEM 0) block 934019072 gen 4076
key (773 INODE_ITEM 0) block 934887424 gen 4076
key (829 INODE_ITEM 0) block 671989760 gen 3808
key (884 INODE_ITEM 0) block 714619404288 gen 3525
key (939 ROOT_ITEM 1889) block 715035377664 gen 3506
key (981 ROOT_ITEM 3330) block 933658624 gen 4067
key (1034 INODE_ITEM 0) block 172621824 gen 3546
key (1095 EXTENT_DATA 0) block 934232064 gen 4076
key (FREE_SPACE UNTYPED 11840520192) block 934248448 gen 4076
key (FREE_SPACE UNTYPED 275981008896) block 934658048 gen 4076
key (FREE_SPACE UNTYPED 568038785024) block 933724160 gen 4067
# btrfs insp dump-t -b 29392896 /dev/mapper/sdd
btrfs-progs v4.19.1
node 29392896 level 1 items 10 free 483 generation 4514 owner ROOT_TREE
fs uuid f5adc913-bbea-4340-8b5f-3411e2cda642
chunk uuid 4af0d265-00fa-4a2b-9ea9-59c06a372284
key (EXTENT_TREE ROOT_ITEM 0) block 712994078720 gen 4514
key (773 INODE_ITEM 0) block 29442048 gen 4470
key (829 INODE_ITEM 0) block 671989760 gen 3808
key (884 INODE_ITEM 0) block 714619404288 gen 3525
key (939 ROOT_ITEM 1889) block 715035377664 gen 3506
key (981 ROOT_ITEM 3330) block 933658624 gen 4067
key (1034 INODE_ITEM 0) block 172621824 gen 3546
key (1095 EXTENT_DATA 0) block 29409280 gen 4514
key (FREE_SPACE UNTYPED 559448850432) block 29376512 gen 4483
key (FREE_SPACE UNTYPED 568038785024) block 933724160 gen 4067
#
I guess I don't understand why 'btrfs check' would have successfully
written six updated copies of the super, when the operation hadn't yet
succeeded. I guess this isn't an atomic operation, and perhaps it
can't be.
---
Chris Murphy
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [bug] btrfs check clear-space-cache v1 corrupted file system
[not found] ` <CAJCQCtQaQ0XFUGFdWfgwmTSPTUShrZgXmsh55c-reS9iS+WpFg@mail.gmail.com>
@ 2019-03-02 3:22 ` Chris Murphy
0 siblings, 0 replies; 10+ messages in thread
From: Chris Murphy @ 2019-03-02 3:22 UTC (permalink / raw)
To: Chris Murphy, Btrfs BTRFS, Qu Wenruo
Two more strange things:
# btrfs check -r 930086912 /dev/mapper/sdd
Opening filesystem to check...
parent transid verify failed on 930086912 wanted 4514 found 4076
parent transid verify failed on 930086912 wanted 4514 found 4076
parent transid verify failed on 930086912 wanted 4514 found 4076
parent transid verify failed on 930086912 wanted 4514 found 4076
Ignoring transid failure
Checking filesystem on /dev/mapper/sdd
UUID: f5adc913-bbea-4340-8b5f-3411e2cda642
[1/7] checking root items
parent transid verify failed on 713084665856 wanted 3425 found 4333
parent transid verify failed on 713084665856 wanted 3425 found 4333
parent transid verify failed on 713084665856 wanted 3425 found 4333
parent transid verify failed on 713084665856 wanted 3425 found 4333
Ignoring transid failure
parent transid verify failed on 713050619904 wanted 3461 found 4404
parent transid verify failed on 713050619904 wanted 3461 found 4404
parent transid verify failed on 713050619904 wanted 3461 found 4404
parent transid verify failed on 713050619904 wanted 3461 found 4404
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=672006144 item=178 parent
level=1 child level=1
ERROR: failed to repair root items: Input/output error
#
dmesg most recent messages are four hours ago, nothing at the time of
this input/output error.
Mar 01 16:19:24 fnuc.local kernel: BTRFS info (device dm-2): disk
space caching is enabled
Mar 01 16:19:24 fnuc.local kernel: BTRFS info (device dm-2): has skinny extents
And next, why are backup 1 and backup 3 the same except for the gen
for tree, extent, and csum roots? The addresses are all the same. How
is that COW?
backup 1:
backup_tree_root: 930086912 gen: 4076 level: 1
backup_chunk_root: 822549151744 gen: 3823 level: 1
backup 3:
backup_tree_root: 930086912 gen: 4074 level: 1
backup_chunk_root: 822549151744 gen: 3823 level: 1
When I go inspect that address, it's of course generation 4076. So
this backup 3 slot is actually invalid, it's already been overwritten.
Anyway, if you need more than just the extent tree, I need a hint on
how to sanitize file names. I tried looking through the archives, I
thought you posted a sed trick to do that with debug-tree but I can't
find it. And I know you don't like btrfs-image -s or -ss.
---
Chris Murphy
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [bug] btrfs check clear-space-cache v1 corrupted file system
2019-03-02 2:29 ` Chris Murphy
2019-03-02 2:54 ` Chris Murphy
@ 2019-03-02 4:14 ` Qu Wenruo
[not found] ` <CAJCQCtRyStqPLOXHVWkcha3GkA7wt2u00qSH7DfUyL_ift5ppQ@mail.gmail.com>
1 sibling, 1 reply; 10+ messages in thread
From: Qu Wenruo @ 2019-03-02 4:14 UTC (permalink / raw)
To: Chris Murphy; +Cc: Btrfs BTRFS
[-- Attachment #1.1: Type: text/plain, Size: 1531 bytes --]
On 2019/3/2 上午10:29, Chris Murphy wrote:
> On Fri, Mar 1, 2019 at 6:05 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>
>>
>>
>> On 2019/3/2 上午8:20, Chris Murphy wrote:
>>> Problem happens with btrfsprogs 4.19.1
>>>
>>> https://bugzilla.kernel.org/show_bug.cgi?id=202717
>>>
>>> That bug report includes the trace in user space; but I've also
>>> processed the resulting coredump file with gdb and attached that to
>>> the bug report.
>>>
>>> Before using --clear-space-cache v1, btrfs scrub and btrfs check came
>>> up clean no errors. Following the crash, I compiled brfsprogs 4.20.2
>>> and ran 'btrfs check' which finds corruption.
>>>
>>> I've taken no further action, and the btrfs check output gives me no
>>> useful plain language information what the next steps are. So I'm just
>>> gonna leave it alone since it's a backup volume anyway.
>>
>> Is the backup volume binary dump of the original fs?
>
> No. Only btrfs send/receive.
>
>
>>
>> I'm interesting why clear space cache v1 doesn't work, to find out that,
>> we need tree dump of the original fs.
>
> btrfs insp dump-t -t 2 ? Or the whole thing?
extent tree and root tree.
As those are the only trees modified in clear extent tree.
Thanks,
Qu
>
>>
>> And is the original check done by latest btrfs-progs or v4.19.1?
>
> The original check is 4.19.1
> The clear cache is 4.19.1
> The subsequent check showing corruption is 4.20.2 but I've found that
> it's the same output for 4.19.1 as well.
>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [bug] btrfs check clear-space-cache v1 corrupted file system
[not found] ` <CAJCQCtRyStqPLOXHVWkcha3GkA7wt2u00qSH7DfUyL_ift5ppQ@mail.gmail.com>
@ 2019-03-10 23:20 ` Chris Murphy
2019-08-12 4:24 ` Chris Murphy
0 siblings, 1 reply; 10+ messages in thread
From: Chris Murphy @ 2019-03-10 23:20 UTC (permalink / raw)
To: Btrfs BTRFS; +Cc: Qu Wenruo
On Sat, Mar 2, 2019 at 11:18 AM Chris Murphy <lists@colorremedies.com> wrote:
>
> Sending URL for dump-tree output offlist. Conversation should still be on-list.
Any more information required from me at this point?
--
Chris Murphy
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [bug] btrfs check clear-space-cache v1 corrupted file system
2019-03-10 23:20 ` Chris Murphy
@ 2019-08-12 4:24 ` Chris Murphy
2019-08-12 4:54 ` Qu Wenruo
0 siblings, 1 reply; 10+ messages in thread
From: Chris Murphy @ 2019-08-12 4:24 UTC (permalink / raw)
To: Btrfs BTRFS; +Cc: Qu Wenruo
On Sun, Mar 10, 2019 at 5:20 PM Chris Murphy <lists@colorremedies.com> wrote:
>
> On Sat, Mar 2, 2019 at 11:18 AM Chris Murphy <lists@colorremedies.com> wrote:
> >
> > Sending URL for dump-tree output offlist. Conversation should still be on-list.
>
>
> Any more information required from me at this point?
This file system has been on a shelf since the problem happened. Is
the lack of COW repairs with btrfs check solved? Can this file system
be fixed? Maybe --init-extent-tree is worth a shot?
--
Chris Murphy
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [bug] btrfs check clear-space-cache v1 corrupted file system
2019-08-12 4:24 ` Chris Murphy
@ 2019-08-12 4:54 ` Qu Wenruo
2019-08-12 5:04 ` Chris Murphy
0 siblings, 1 reply; 10+ messages in thread
From: Qu Wenruo @ 2019-08-12 4:54 UTC (permalink / raw)
To: Chris Murphy, Btrfs BTRFS
[-- Attachment #1.1: Type: text/plain, Size: 964 bytes --]
On 2019/8/12 下午12:24, Chris Murphy wrote:
> On Sun, Mar 10, 2019 at 5:20 PM Chris Murphy <lists@colorremedies.com> wrote:
>>
>> On Sat, Mar 2, 2019 at 11:18 AM Chris Murphy <lists@colorremedies.com> wrote:
>>>
>>> Sending URL for dump-tree output offlist. Conversation should still be on-list.
>>
>>
>> Any more information required from me at this point?
>
> This file system has been on a shelf since the problem happened. Is
> the lack of COW repairs with btrfs check solved? Can this file system
> be fixed? Maybe --init-extent-tree is worth a shot?
>
>
Latest btrfs check has solved the problem of bad CoW. In fact it's
solved in btrfs-progs v5.1 release.
So, if you run btrfs check --clear-space-cache v1 again, it shouldn't
cause transid mismatch problem any more.
Although IIRC that fs already got transid error, thus enhanced
btrfs-check won't help much to solve the existing transid mismatch problem.
Thanks,
Qu
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [bug] btrfs check clear-space-cache v1 corrupted file system
2019-08-12 4:54 ` Qu Wenruo
@ 2019-08-12 5:04 ` Chris Murphy
0 siblings, 0 replies; 10+ messages in thread
From: Chris Murphy @ 2019-08-12 5:04 UTC (permalink / raw)
To: Qu Wenruo, Btrfs BTRFS
On Sun, Aug 11, 2019 at 10:54 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2019/8/12 下午12:24, Chris Murphy wrote:
> > On Sun, Mar 10, 2019 at 5:20 PM Chris Murphy <lists@colorremedies.com> wrote:
> >>
> >> On Sat, Mar 2, 2019 at 11:18 AM Chris Murphy <lists@colorremedies.com> wrote:
> >>>
> >>> Sending URL for dump-tree output offlist. Conversation should still be on-list.
> >>
> >>
> >> Any more information required from me at this point?
> >
> > This file system has been on a shelf since the problem happened. Is
> > the lack of COW repairs with btrfs check solved? Can this file system
> > be fixed? Maybe --init-extent-tree is worth a shot?
> >
> >
> Latest btrfs check has solved the problem of bad CoW. In fact it's
> solved in btrfs-progs v5.1 release.
>
> So, if you run btrfs check --clear-space-cache v1 again, it shouldn't
> cause transid mismatch problem any more.
That's good news.
>
> Although IIRC that fs already got transid error, thus enhanced
> btrfs-check won't help much to solve the existing transid mismatch problem.
OK and the other bug report I just posted is this same file system
failing to ro scrub. It's not obvious that the failure is due to
transid mismatch.
If the file system can't be fixed, I'll wipe it.
--
Chris Murphy
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2019-08-12 5:04 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-03-02 0:20 [bug] btrfs check clear-space-cache v1 corrupted file system Chris Murphy
2019-03-02 1:05 ` Qu Wenruo
2019-03-02 2:29 ` Chris Murphy
2019-03-02 2:54 ` Chris Murphy
[not found] ` <CAJCQCtQaQ0XFUGFdWfgwmTSPTUShrZgXmsh55c-reS9iS+WpFg@mail.gmail.com>
2019-03-02 3:22 ` Chris Murphy
2019-03-02 4:14 ` Qu Wenruo
[not found] ` <CAJCQCtRyStqPLOXHVWkcha3GkA7wt2u00qSH7DfUyL_ift5ppQ@mail.gmail.com>
2019-03-10 23:20 ` Chris Murphy
2019-08-12 4:24 ` Chris Murphy
2019-08-12 4:54 ` Qu Wenruo
2019-08-12 5:04 ` Chris Murphy
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).