* Continous failing osd-scrub-repair.sh @ 2017-05-31 20:58 Willem Jan Withagen 2017-05-31 21:38 ` David Zafman 0 siblings, 1 reply; 7+ messages in thread From: Willem Jan Withagen @ 2017-05-31 20:58 UTC (permalink / raw) To: Ceph Development Since a few days I have a continuous failing test: osd-scrub-repair.sh There is a diff failing, where the data looks almost correct. :( Looks there is something wrong with omap_digest. Could this be due to bluestore being enable, whilst FreeBSD does not have that? Anyways, all suggestions are welcome --WjW /home/jenkins/workspace/ceph-master/src/test/osd/osd-scrub-repair.sh:1569: TEST_corrupt_scrub_replicated: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 322c322 < "selected_object_info": "2:f4981d31:::ROBJ4:head(dirty|omap|data_digest|omap_digest s 7 uv 12 dd 2ddbf8f5 od d7178dfe alloc_hint [0 0 0])", --- > "selected_object_info": "2:f4981d31:::ROBJ4:head(dirty|omap|data_digest s 7 uv 12 dd 2ddbf8f5 alloc_hint [0 0 0])", 333,335c333 < "errors": [ < "omap_digest_mismatch_oi" < ], --- > "errors": [], 341,343c339 < "union_shard_errors": [ < "omap_digest_mismatch_oi" < ] --- > "union_shard_errors": [] 356c352 < "selected_object_info": "2:f4bfd4d1:::ROBJ5:head(dirty|omap|data_digest|omap_digest s 7 uv 15 dd 2ddbf8f5 od 1a862a41 alloc_hint [0 0 0])", --- > "selected_object_info": "2:f4bfd4d1:::ROBJ5:head(dirty|omap|data_digest s 7 uv 15 dd 2ddbf8f5 alloc_hint [0 0 0])", 367,369c363 < "errors": [ < "omap_digest_mismatch_oi" < ], --- > "errors": [], 375,377c369 < "union_shard_errors": [ < "omap_digest_mismatch_oi" < ] --- > "union_shard_errors": [] 390c382 < "selected_object_info": "2:a53c12e8:::ROBJ6:head(dirty|omap|data_digest|omap_digest s 7 uv 18 dd 2ddbf8f5 od 689ee887 alloc_hint [0 0 0])", --- > "selected_object_info": "2:a53c12e8:::ROBJ6:head(dirty|omap|data_digest s 7 uv 18 dd 2ddbf8f5 alloc_hint [0 0 0])", 401,403c393 < "errors": [ < "omap_digest_mismatch_oi" < ], --- > "errors": [], 409,411c399 < "union_shard_errors": [ < "omap_digest_mismatch_oi" < ] --- > "union_shard_errors": [] 424c412 < "selected_object_info": "2:8b55fa4b:::ROBJ7:head(dirty|omap|data_digest|omap_digest s 7 uv 21 dd 2ddbf8f5 od efced57a alloc_hint [0 0 0])", --- > "selected_object_info": "2:8b55fa4b:::ROBJ7:head(dirty|omap|data_digest s 7 uv 21 dd 2ddbf8f5 alloc_hint [0 0 0])", 435,437c423 < "errors": [ < "omap_digest_mismatch_oi" < ], --- > "errors": [], 443,445c429 < "union_shard_errors": [ < "omap_digest_mismatch_oi" < ] --- > "union_shard_errors": [] 459c443 < "selected_object_info": "2:86586531:::ROBJ8:head(dirty|omap|data_digest|omap_digest s 7 uv 26 dd 2ddbf8f5 od d6be81dc alloc_hint [0 0 0])", --- > "selected_object_info": "2:86586531:::ROBJ8:head(dirty|omap|data_digest s 7 uv 26 dd 2ddbf8f5 alloc_hint [0 0 0])", 532c516 < "version": 56 --- > "version": 50 534c518 < "selected_object_info": "2:ffdb2004:::ROBJ9:head(dirty|omap|data_digest|omap_digest s 3 uv 56 dd 1f26fb26 od 2eecc539 alloc_hint [0 0 0])", --- > "selected_object_info": "2:ffdb2004:::ROBJ9:head(dirty|omap|data_digest s 3 uv 50 dd 1f26fb26 alloc_hint [0 0 0])", 571c555 < "object_info": "2:ffdb2004:::ROBJ9:head(dirty|omap|data_digest|omap_digest s 3 uv 56 dd 1f26fb26 od 2eecc539 alloc_hint [0 0 0])", --- > "object_info": "2:ffdb2004:::ROBJ9:head(dirty|omap|data_digest s 3 uv 50 dd 1f26fb26 alloc_hint [0 0 0])", /home/jenkins/workspace/ceph-master/src/test/osd/osd-scrub-repair.sh:1569: TEST_corrupt_scrub_replicated: test no = yes /home/jenkins/workspace/ceph-master/src/test/osd/osd-scrub-repair.sh:1569: ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Continous failing osd-scrub-repair.sh 2017-05-31 20:58 Continous failing osd-scrub-repair.sh Willem Jan Withagen @ 2017-05-31 21:38 ` David Zafman 2017-05-31 23:14 ` Willem Jan Withagen 0 siblings, 1 reply; 7+ messages in thread From: David Zafman @ 2017-05-31 21:38 UTC (permalink / raw) To: Willem Jan Withagen, Ceph Development All the failures relate to not have the omap digest available in the object info. That use to be present before so that the error omap_digest_mismatch_oi would be detected during this test. David On 5/31/17 1:58 PM, Willem Jan Withagen wrote: > Since a few days I have a continuous failing test: osd-scrub-repair.sh > > There is a diff failing, where the data looks almost correct. :( > Looks there is something wrong with omap_digest. > > Could this be due to bluestore being enable, whilst FreeBSD does not > have that? > > Anyways, all suggestions are welcome > > --WjW > > /home/jenkins/workspace/ceph-master/src/test/osd/osd-scrub-repair.sh:1569: > TEST_corrupt_scrub_replicated: diff td/osd-scrub-repair/checkcsjson > td/osd-scrub-repair/csjson > 322c322 > < "selected_object_info": > "2:f4981d31:::ROBJ4:head(dirty|omap|data_digest|omap_digest s 7 uv 12 dd > 2ddbf8f5 od d7178dfe alloc_hint [0 0 0])", > --- >> "selected_object_info": > "2:f4981d31:::ROBJ4:head(dirty|omap|data_digest s 7 uv 12 dd 2ddbf8f5 > alloc_hint [0 0 0])", > 333,335c333 > < "errors": [ > < "omap_digest_mismatch_oi" > < ], > --- >> "errors": [], > 341,343c339 > < "union_shard_errors": [ > < "omap_digest_mismatch_oi" > < ] > --- >> "union_shard_errors": [] > 356c352 > < "selected_object_info": > "2:f4bfd4d1:::ROBJ5:head(dirty|omap|data_digest|omap_digest s 7 uv 15 dd > 2ddbf8f5 od 1a862a41 alloc_hint [0 0 0])", > --- >> "selected_object_info": > "2:f4bfd4d1:::ROBJ5:head(dirty|omap|data_digest s 7 uv 15 dd 2ddbf8f5 > alloc_hint [0 0 0])", > 367,369c363 > < "errors": [ > < "omap_digest_mismatch_oi" > < ], > --- >> "errors": [], > 375,377c369 > < "union_shard_errors": [ > < "omap_digest_mismatch_oi" > < ] > --- >> "union_shard_errors": [] > 390c382 > < "selected_object_info": > "2:a53c12e8:::ROBJ6:head(dirty|omap|data_digest|omap_digest s 7 uv 18 dd > 2ddbf8f5 od 689ee887 alloc_hint [0 0 0])", > --- >> "selected_object_info": > "2:a53c12e8:::ROBJ6:head(dirty|omap|data_digest s 7 uv 18 dd 2ddbf8f5 > alloc_hint [0 0 0])", > 401,403c393 > < "errors": [ > < "omap_digest_mismatch_oi" > < ], > --- >> "errors": [], > 409,411c399 > < "union_shard_errors": [ > < "omap_digest_mismatch_oi" > < ] > --- >> "union_shard_errors": [] > 424c412 > < "selected_object_info": > "2:8b55fa4b:::ROBJ7:head(dirty|omap|data_digest|omap_digest s 7 uv 21 dd > 2ddbf8f5 od efced57a alloc_hint [0 0 0])", > --- >> "selected_object_info": > "2:8b55fa4b:::ROBJ7:head(dirty|omap|data_digest s 7 uv 21 dd 2ddbf8f5 > alloc_hint [0 0 0])", > 435,437c423 > < "errors": [ > < "omap_digest_mismatch_oi" > < ], > --- >> "errors": [], > 443,445c429 > < "union_shard_errors": [ > < "omap_digest_mismatch_oi" > < ] > --- >> "union_shard_errors": [] > 459c443 > < "selected_object_info": > "2:86586531:::ROBJ8:head(dirty|omap|data_digest|omap_digest s 7 uv 26 dd > 2ddbf8f5 od d6be81dc alloc_hint [0 0 0])", > --- >> "selected_object_info": > "2:86586531:::ROBJ8:head(dirty|omap|data_digest s 7 uv 26 dd 2ddbf8f5 > alloc_hint [0 0 0])", > 532c516 > < "version": 56 > --- >> "version": 50 > 534c518 > < "selected_object_info": > "2:ffdb2004:::ROBJ9:head(dirty|omap|data_digest|omap_digest s 3 uv 56 dd > 1f26fb26 od 2eecc539 alloc_hint [0 0 0])", > --- >> "selected_object_info": > "2:ffdb2004:::ROBJ9:head(dirty|omap|data_digest s 3 uv 50 dd 1f26fb26 > alloc_hint [0 0 0])", > 571c555 > < "object_info": > "2:ffdb2004:::ROBJ9:head(dirty|omap|data_digest|omap_digest s 3 uv 56 dd > 1f26fb26 od 2eecc539 alloc_hint [0 0 0])", > --- >> "object_info": "2:ffdb2004:::ROBJ9:head(dirty|omap|data_digest > s 3 uv 50 dd 1f26fb26 alloc_hint [0 0 0])", > /home/jenkins/workspace/ceph-master/src/test/osd/osd-scrub-repair.sh:1569: > TEST_corrupt_scrub_replicated: test no = yes > /home/jenkins/workspace/ceph-master/src/test/osd/osd-scrub-repair.sh:1569: > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Continous failing osd-scrub-repair.sh 2017-05-31 21:38 ` David Zafman @ 2017-05-31 23:14 ` Willem Jan Withagen 2017-06-01 1:11 ` David Zafman 0 siblings, 1 reply; 7+ messages in thread From: Willem Jan Withagen @ 2017-05-31 23:14 UTC (permalink / raw) To: David Zafman, Ceph Development On 31/05/2017 23:38, David Zafman wrote: > > All the failures relate to not have the omap digest available in the > object info. That use to be present before so > > that the error omap_digest_mismatch_oi would be detected during this > test. > So, do you have any idea how this is caused? What has happened to not have omap digests any longer? --WjW > David > > > On 5/31/17 1:58 PM, Willem Jan Withagen wrote: >> Since a few days I have a continuous failing test: osd-scrub-repair.sh >> >> There is a diff failing, where the data looks almost correct. :( >> Looks there is something wrong with omap_digest. >> >> Could this be due to bluestore being enable, whilst FreeBSD does not >> have that? >> >> Anyways, all suggestions are welcome >> >> --WjW >> >> /home/jenkins/workspace/ceph-master/src/test/osd/osd-scrub-repair.sh:1569: >> >> TEST_corrupt_scrub_replicated: diff td/osd-scrub-repair/checkcsjson >> td/osd-scrub-repair/csjson >> 322c322 >> < "selected_object_info": >> "2:f4981d31:::ROBJ4:head(dirty|omap|data_digest|omap_digest s 7 uv 12 dd >> 2ddbf8f5 od d7178dfe alloc_hint [0 0 0])", >> --- >>> "selected_object_info": >> "2:f4981d31:::ROBJ4:head(dirty|omap|data_digest s 7 uv 12 dd 2ddbf8f5 >> alloc_hint [0 0 0])", >> 333,335c333 >> < "errors": [ >> < "omap_digest_mismatch_oi" >> < ], >> --- >>> "errors": [], >> 341,343c339 >> < "union_shard_errors": [ >> < "omap_digest_mismatch_oi" >> < ] >> --- >>> "union_shard_errors": [] >> 356c352 >> < "selected_object_info": >> "2:f4bfd4d1:::ROBJ5:head(dirty|omap|data_digest|omap_digest s 7 uv 15 dd >> 2ddbf8f5 od 1a862a41 alloc_hint [0 0 0])", >> --- >>> "selected_object_info": >> "2:f4bfd4d1:::ROBJ5:head(dirty|omap|data_digest s 7 uv 15 dd 2ddbf8f5 >> alloc_hint [0 0 0])", >> 367,369c363 >> < "errors": [ >> < "omap_digest_mismatch_oi" >> < ], >> --- >>> "errors": [], >> 375,377c369 >> < "union_shard_errors": [ >> < "omap_digest_mismatch_oi" >> < ] >> --- >>> "union_shard_errors": [] >> 390c382 >> < "selected_object_info": >> "2:a53c12e8:::ROBJ6:head(dirty|omap|data_digest|omap_digest s 7 uv 18 dd >> 2ddbf8f5 od 689ee887 alloc_hint [0 0 0])", >> --- >>> "selected_object_info": >> "2:a53c12e8:::ROBJ6:head(dirty|omap|data_digest s 7 uv 18 dd 2ddbf8f5 >> alloc_hint [0 0 0])", >> 401,403c393 >> < "errors": [ >> < "omap_digest_mismatch_oi" >> < ], >> --- >>> "errors": [], >> 409,411c399 >> < "union_shard_errors": [ >> < "omap_digest_mismatch_oi" >> < ] >> --- >>> "union_shard_errors": [] >> 424c412 >> < "selected_object_info": >> "2:8b55fa4b:::ROBJ7:head(dirty|omap|data_digest|omap_digest s 7 uv 21 dd >> 2ddbf8f5 od efced57a alloc_hint [0 0 0])", >> --- >>> "selected_object_info": >> "2:8b55fa4b:::ROBJ7:head(dirty|omap|data_digest s 7 uv 21 dd 2ddbf8f5 >> alloc_hint [0 0 0])", >> 435,437c423 >> < "errors": [ >> < "omap_digest_mismatch_oi" >> < ], >> --- >>> "errors": [], >> 443,445c429 >> < "union_shard_errors": [ >> < "omap_digest_mismatch_oi" >> < ] >> --- >>> "union_shard_errors": [] >> 459c443 >> < "selected_object_info": >> "2:86586531:::ROBJ8:head(dirty|omap|data_digest|omap_digest s 7 uv 26 dd >> 2ddbf8f5 od d6be81dc alloc_hint [0 0 0])", >> --- >>> "selected_object_info": >> "2:86586531:::ROBJ8:head(dirty|omap|data_digest s 7 uv 26 dd 2ddbf8f5 >> alloc_hint [0 0 0])", >> 532c516 >> < "version": 56 >> --- >>> "version": 50 >> 534c518 >> < "selected_object_info": >> "2:ffdb2004:::ROBJ9:head(dirty|omap|data_digest|omap_digest s 3 uv 56 dd >> 1f26fb26 od 2eecc539 alloc_hint [0 0 0])", >> --- >>> "selected_object_info": >> "2:ffdb2004:::ROBJ9:head(dirty|omap|data_digest s 3 uv 50 dd 1f26fb26 >> alloc_hint [0 0 0])", >> 571c555 >> < "object_info": >> "2:ffdb2004:::ROBJ9:head(dirty|omap|data_digest|omap_digest s 3 uv 56 dd >> 1f26fb26 od 2eecc539 alloc_hint [0 0 0])", >> --- >>> "object_info": "2:ffdb2004:::ROBJ9:head(dirty|omap|data_digest >> s 3 uv 50 dd 1f26fb26 alloc_hint [0 0 0])", >> /home/jenkins/workspace/ceph-master/src/test/osd/osd-scrub-repair.sh:1569: >> >> TEST_corrupt_scrub_replicated: test no = yes >> /home/jenkins/workspace/ceph-master/src/test/osd/osd-scrub-repair.sh:1569: >> >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Continous failing osd-scrub-repair.sh 2017-05-31 23:14 ` Willem Jan Withagen @ 2017-06-01 1:11 ` David Zafman 2017-06-01 8:11 ` Willem Jan Withagen 0 siblings, 1 reply; 7+ messages in thread From: David Zafman @ 2017-06-01 1:11 UTC (permalink / raw) To: Willem Jan Withagen, Ceph Development, Josh Durgin The change 3ca750d4 from Josh broke this. I didn't see before that the test itself was broken. David On 5/31/17 4:14 PM, Willem Jan Withagen wrote: > > > On 31/05/2017 23:38, David Zafman wrote: >> >> All the failures relate to not have the omap digest available in the >> object info. That use to be present before so >> >> that the error omap_digest_mismatch_oi would be detected during this >> test. >> > So, do you have any idea how this is caused? > What has happened to not have omap digests any longer? > > --WjW > >> David >> >> >> On 5/31/17 1:58 PM, Willem Jan Withagen wrote: >>> Since a few days I have a continuous failing test: osd-scrub-repair.sh >>> >>> There is a diff failing, where the data looks almost correct. :( >>> Looks there is something wrong with omap_digest. >>> >>> Could this be due to bluestore being enable, whilst FreeBSD does not >>> have that? >>> >>> Anyways, all suggestions are welcome >>> >>> --WjW >>> >>> /home/jenkins/workspace/ceph-master/src/test/osd/osd-scrub-repair.sh:1569: >>> >>> TEST_corrupt_scrub_replicated: diff td/osd-scrub-repair/checkcsjson >>> td/osd-scrub-repair/csjson >>> 322c322 >>> < "selected_object_info": >>> "2:f4981d31:::ROBJ4:head(dirty|omap|data_digest|omap_digest s 7 uv >>> 12 dd >>> 2ddbf8f5 od d7178dfe alloc_hint [0 0 0])", >>> --- >>>> "selected_object_info": >>> "2:f4981d31:::ROBJ4:head(dirty|omap|data_digest s 7 uv 12 dd 2ddbf8f5 >>> alloc_hint [0 0 0])", >>> 333,335c333 >>> < "errors": [ >>> < "omap_digest_mismatch_oi" >>> < ], >>> --- >>>> "errors": [], >>> 341,343c339 >>> < "union_shard_errors": [ >>> < "omap_digest_mismatch_oi" >>> < ] >>> --- >>>> "union_shard_errors": [] >>> 356c352 >>> < "selected_object_info": >>> "2:f4bfd4d1:::ROBJ5:head(dirty|omap|data_digest|omap_digest s 7 uv >>> 15 dd >>> 2ddbf8f5 od 1a862a41 alloc_hint [0 0 0])", >>> --- >>>> "selected_object_info": >>> "2:f4bfd4d1:::ROBJ5:head(dirty|omap|data_digest s 7 uv 15 dd 2ddbf8f5 >>> alloc_hint [0 0 0])", >>> 367,369c363 >>> < "errors": [ >>> < "omap_digest_mismatch_oi" >>> < ], >>> --- >>>> "errors": [], >>> 375,377c369 >>> < "union_shard_errors": [ >>> < "omap_digest_mismatch_oi" >>> < ] >>> --- >>>> "union_shard_errors": [] >>> 390c382 >>> < "selected_object_info": >>> "2:a53c12e8:::ROBJ6:head(dirty|omap|data_digest|omap_digest s 7 uv >>> 18 dd >>> 2ddbf8f5 od 689ee887 alloc_hint [0 0 0])", >>> --- >>>> "selected_object_info": >>> "2:a53c12e8:::ROBJ6:head(dirty|omap|data_digest s 7 uv 18 dd 2ddbf8f5 >>> alloc_hint [0 0 0])", >>> 401,403c393 >>> < "errors": [ >>> < "omap_digest_mismatch_oi" >>> < ], >>> --- >>>> "errors": [], >>> 409,411c399 >>> < "union_shard_errors": [ >>> < "omap_digest_mismatch_oi" >>> < ] >>> --- >>>> "union_shard_errors": [] >>> 424c412 >>> < "selected_object_info": >>> "2:8b55fa4b:::ROBJ7:head(dirty|omap|data_digest|omap_digest s 7 uv >>> 21 dd >>> 2ddbf8f5 od efced57a alloc_hint [0 0 0])", >>> --- >>>> "selected_object_info": >>> "2:8b55fa4b:::ROBJ7:head(dirty|omap|data_digest s 7 uv 21 dd 2ddbf8f5 >>> alloc_hint [0 0 0])", >>> 435,437c423 >>> < "errors": [ >>> < "omap_digest_mismatch_oi" >>> < ], >>> --- >>>> "errors": [], >>> 443,445c429 >>> < "union_shard_errors": [ >>> < "omap_digest_mismatch_oi" >>> < ] >>> --- >>>> "union_shard_errors": [] >>> 459c443 >>> < "selected_object_info": >>> "2:86586531:::ROBJ8:head(dirty|omap|data_digest|omap_digest s 7 uv >>> 26 dd >>> 2ddbf8f5 od d6be81dc alloc_hint [0 0 0])", >>> --- >>>> "selected_object_info": >>> "2:86586531:::ROBJ8:head(dirty|omap|data_digest s 7 uv 26 dd 2ddbf8f5 >>> alloc_hint [0 0 0])", >>> 532c516 >>> < "version": 56 >>> --- >>>> "version": 50 >>> 534c518 >>> < "selected_object_info": >>> "2:ffdb2004:::ROBJ9:head(dirty|omap|data_digest|omap_digest s 3 uv >>> 56 dd >>> 1f26fb26 od 2eecc539 alloc_hint [0 0 0])", >>> --- >>>> "selected_object_info": >>> "2:ffdb2004:::ROBJ9:head(dirty|omap|data_digest s 3 uv 50 dd 1f26fb26 >>> alloc_hint [0 0 0])", >>> 571c555 >>> < "object_info": >>> "2:ffdb2004:::ROBJ9:head(dirty|omap|data_digest|omap_digest s 3 uv >>> 56 dd >>> 1f26fb26 od 2eecc539 alloc_hint [0 0 0])", >>> --- >>>> "object_info": >>>> "2:ffdb2004:::ROBJ9:head(dirty|omap|data_digest >>> s 3 uv 50 dd 1f26fb26 alloc_hint [0 0 0])", >>> /home/jenkins/workspace/ceph-master/src/test/osd/osd-scrub-repair.sh:1569: >>> >>> TEST_corrupt_scrub_replicated: test no = yes >>> /home/jenkins/workspace/ceph-master/src/test/osd/osd-scrub-repair.sh:1569: >>> >>> -- >>> To unsubscribe from this list: send the line "unsubscribe >>> ceph-devel" in >>> the body of a message to majordomo@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >> > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Continous failing osd-scrub-repair.sh 2017-06-01 1:11 ` David Zafman @ 2017-06-01 8:11 ` Willem Jan Withagen 2017-06-01 8:16 ` David Zafman 0 siblings, 1 reply; 7+ messages in thread From: Willem Jan Withagen @ 2017-06-01 8:11 UTC (permalink / raw) To: David Zafman, Ceph Development, Josh Durgin On 01/06/2017 03:11, David Zafman wrote: > > The change 3ca750d4 from Josh broke this. I didn't see before that > the test itself was broken. > Well then again, this is on FreeBSD, where your millage can (and will) vary. For one, we do not do BlueStore atm. So what is my best option: - See if it is possible if we can fix this? - Disable the test on basis of FreeBSD - Get a different diff-file to compare with? --WjW > David > > > On 5/31/17 4:14 PM, Willem Jan Withagen wrote: >> >> >> On 31/05/2017 23:38, David Zafman wrote: >>> >>> All the failures relate to not have the omap digest available in the >>> object info. That use to be present before so >>> >>> that the error omap_digest_mismatch_oi would be detected during this >>> test. >>> >> So, do you have any idea how this is caused? >> What has happened to not have omap digests any longer? >> >> --WjW >> >>> David >>> >>> >>> On 5/31/17 1:58 PM, Willem Jan Withagen wrote: >>>> Since a few days I have a continuous failing test: osd-scrub-repair.sh >>>> >>>> There is a diff failing, where the data looks almost correct. :( >>>> Looks there is something wrong with omap_digest. >>>> >>>> Could this be due to bluestore being enable, whilst FreeBSD does not >>>> have that? >>>> >>>> Anyways, all suggestions are welcome >>>> >>>> --WjW >>>> >>>> /home/jenkins/workspace/ceph-master/src/test/osd/osd-scrub-repair.sh:1569: >>>> >>>> TEST_corrupt_scrub_replicated: diff td/osd-scrub-repair/checkcsjson >>>> td/osd-scrub-repair/csjson >>>> 322c322 >>>> < "selected_object_info": >>>> "2:f4981d31:::ROBJ4:head(dirty|omap|data_digest|omap_digest s 7 uv >>>> 12 dd >>>> 2ddbf8f5 od d7178dfe alloc_hint [0 0 0])", >>>> --- >>>>> "selected_object_info": >>>> "2:f4981d31:::ROBJ4:head(dirty|omap|data_digest s 7 uv 12 dd 2ddbf8f5 >>>> alloc_hint [0 0 0])", >>>> 333,335c333 >>>> < "errors": [ >>>> < "omap_digest_mismatch_oi" >>>> < ], >>>> --- >>>>> "errors": [], >>>> 341,343c339 >>>> < "union_shard_errors": [ >>>> < "omap_digest_mismatch_oi" >>>> < ] >>>> --- >>>>> "union_shard_errors": [] >>>> 356c352 >>>> < "selected_object_info": >>>> "2:f4bfd4d1:::ROBJ5:head(dirty|omap|data_digest|omap_digest s 7 uv >>>> 15 dd >>>> 2ddbf8f5 od 1a862a41 alloc_hint [0 0 0])", >>>> --- >>>>> "selected_object_info": >>>> "2:f4bfd4d1:::ROBJ5:head(dirty|omap|data_digest s 7 uv 15 dd 2ddbf8f5 >>>> alloc_hint [0 0 0])", >>>> 367,369c363 >>>> < "errors": [ >>>> < "omap_digest_mismatch_oi" >>>> < ], >>>> --- >>>>> "errors": [], >>>> 375,377c369 >>>> < "union_shard_errors": [ >>>> < "omap_digest_mismatch_oi" >>>> < ] >>>> --- >>>>> "union_shard_errors": [] >>>> 390c382 >>>> < "selected_object_info": >>>> "2:a53c12e8:::ROBJ6:head(dirty|omap|data_digest|omap_digest s 7 uv >>>> 18 dd >>>> 2ddbf8f5 od 689ee887 alloc_hint [0 0 0])", >>>> --- >>>>> "selected_object_info": >>>> "2:a53c12e8:::ROBJ6:head(dirty|omap|data_digest s 7 uv 18 dd 2ddbf8f5 >>>> alloc_hint [0 0 0])", >>>> 401,403c393 >>>> < "errors": [ >>>> < "omap_digest_mismatch_oi" >>>> < ], >>>> --- >>>>> "errors": [], >>>> 409,411c399 >>>> < "union_shard_errors": [ >>>> < "omap_digest_mismatch_oi" >>>> < ] >>>> --- >>>>> "union_shard_errors": [] >>>> 424c412 >>>> < "selected_object_info": >>>> "2:8b55fa4b:::ROBJ7:head(dirty|omap|data_digest|omap_digest s 7 uv >>>> 21 dd >>>> 2ddbf8f5 od efced57a alloc_hint [0 0 0])", >>>> --- >>>>> "selected_object_info": >>>> "2:8b55fa4b:::ROBJ7:head(dirty|omap|data_digest s 7 uv 21 dd 2ddbf8f5 >>>> alloc_hint [0 0 0])", >>>> 435,437c423 >>>> < "errors": [ >>>> < "omap_digest_mismatch_oi" >>>> < ], >>>> --- >>>>> "errors": [], >>>> 443,445c429 >>>> < "union_shard_errors": [ >>>> < "omap_digest_mismatch_oi" >>>> < ] >>>> --- >>>>> "union_shard_errors": [] >>>> 459c443 >>>> < "selected_object_info": >>>> "2:86586531:::ROBJ8:head(dirty|omap|data_digest|omap_digest s 7 uv >>>> 26 dd >>>> 2ddbf8f5 od d6be81dc alloc_hint [0 0 0])", >>>> --- >>>>> "selected_object_info": >>>> "2:86586531:::ROBJ8:head(dirty|omap|data_digest s 7 uv 26 dd 2ddbf8f5 >>>> alloc_hint [0 0 0])", >>>> 532c516 >>>> < "version": 56 >>>> --- >>>>> "version": 50 >>>> 534c518 >>>> < "selected_object_info": >>>> "2:ffdb2004:::ROBJ9:head(dirty|omap|data_digest|omap_digest s 3 uv >>>> 56 dd >>>> 1f26fb26 od 2eecc539 alloc_hint [0 0 0])", >>>> --- >>>>> "selected_object_info": >>>> "2:ffdb2004:::ROBJ9:head(dirty|omap|data_digest s 3 uv 50 dd 1f26fb26 >>>> alloc_hint [0 0 0])", >>>> 571c555 >>>> < "object_info": >>>> "2:ffdb2004:::ROBJ9:head(dirty|omap|data_digest|omap_digest s 3 uv >>>> 56 dd >>>> 1f26fb26 od 2eecc539 alloc_hint [0 0 0])", >>>> --- >>>>> "object_info": >>>>> "2:ffdb2004:::ROBJ9:head(dirty|omap|data_digest >>>> s 3 uv 50 dd 1f26fb26 alloc_hint [0 0 0])", >>>> /home/jenkins/workspace/ceph-master/src/test/osd/osd-scrub-repair.sh:1569: >>>> >>>> TEST_corrupt_scrub_replicated: test no = yes >>>> /home/jenkins/workspace/ceph-master/src/test/osd/osd-scrub-repair.sh:1569: >>>> >>>> -- >>>> To unsubscribe from this list: send the line "unsubscribe >>>> ceph-devel" in >>>> the body of a message to majordomo@vger.kernel.org >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> >> >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Continous failing osd-scrub-repair.sh 2017-06-01 8:11 ` Willem Jan Withagen @ 2017-06-01 8:16 ` David Zafman 2017-06-01 8:28 ` Willem Jan Withagen 0 siblings, 1 reply; 7+ messages in thread From: David Zafman @ 2017-06-01 8:16 UTC (permalink / raw) To: Willem Jan Withagen, Ceph Development, Josh Durgin Willem, Yes, I was wrong this wasn't caused by 3ca750d4. That change change only affected the test that did EC on bluestore with EC overwrites. If you have 44736aac74493b5532c9e3574a37a3642766f212 revert it and see if that fixes your problem. David On 6/1/17 1:11 AM, Willem Jan Withagen wrote: > > > On 01/06/2017 03:11, David Zafman wrote: >> >> The change 3ca750d4 from Josh broke this. I didn't see before that >> the test itself was broken. >> > Well then again, this is on FreeBSD, where your millage can (and will) > vary. > For one, we do not do BlueStore atm. > > So what is my best option: > - See if it is possible if we can fix this? > - Disable the test on basis of FreeBSD > - Get a different diff-file to compare with? > > --WjW > >> David >> >> >> On 5/31/17 4:14 PM, Willem Jan Withagen wrote: >>> >>> >>> On 31/05/2017 23:38, David Zafman wrote: >>>> >>>> All the failures relate to not have the omap digest available in >>>> the object info. That use to be present before so >>>> >>>> that the error omap_digest_mismatch_oi would be detected during >>>> this test. >>>> >>> So, do you have any idea how this is caused? >>> What has happened to not have omap digests any longer? >>> >>> --WjW >>> >>>> David >>>> >>>> >>>> On 5/31/17 1:58 PM, Willem Jan Withagen wrote: >>>>> Since a few days I have a continuous failing test: >>>>> osd-scrub-repair.sh >>>>> >>>>> There is a diff failing, where the data looks almost correct. :( >>>>> Looks there is something wrong with omap_digest. >>>>> >>>>> Could this be due to bluestore being enable, whilst FreeBSD does not >>>>> have that? >>>>> >>>>> Anyways, all suggestions are welcome >>>>> >>>>> --WjW >>>>> >>>>> /home/jenkins/workspace/ceph-master/src/test/osd/osd-scrub-repair.sh:1569: >>>>> >>>>> TEST_corrupt_scrub_replicated: diff td/osd-scrub-repair/checkcsjson >>>>> td/osd-scrub-repair/csjson >>>>> 322c322 >>>>> < "selected_object_info": >>>>> "2:f4981d31:::ROBJ4:head(dirty|omap|data_digest|omap_digest s 7 uv >>>>> 12 dd >>>>> 2ddbf8f5 od d7178dfe alloc_hint [0 0 0])", >>>>> --- >>>>>> "selected_object_info": >>>>> "2:f4981d31:::ROBJ4:head(dirty|omap|data_digest s 7 uv 12 dd 2ddbf8f5 >>>>> alloc_hint [0 0 0])", >>>>> 333,335c333 >>>>> < "errors": [ >>>>> < "omap_digest_mismatch_oi" >>>>> < ], >>>>> --- >>>>>> "errors": [], >>>>> 341,343c339 >>>>> < "union_shard_errors": [ >>>>> < "omap_digest_mismatch_oi" >>>>> < ] >>>>> --- >>>>>> "union_shard_errors": [] >>>>> 356c352 >>>>> < "selected_object_info": >>>>> "2:f4bfd4d1:::ROBJ5:head(dirty|omap|data_digest|omap_digest s 7 uv >>>>> 15 dd >>>>> 2ddbf8f5 od 1a862a41 alloc_hint [0 0 0])", >>>>> --- >>>>>> "selected_object_info": >>>>> "2:f4bfd4d1:::ROBJ5:head(dirty|omap|data_digest s 7 uv 15 dd 2ddbf8f5 >>>>> alloc_hint [0 0 0])", >>>>> 367,369c363 >>>>> < "errors": [ >>>>> < "omap_digest_mismatch_oi" >>>>> < ], >>>>> --- >>>>>> "errors": [], >>>>> 375,377c369 >>>>> < "union_shard_errors": [ >>>>> < "omap_digest_mismatch_oi" >>>>> < ] >>>>> --- >>>>>> "union_shard_errors": [] >>>>> 390c382 >>>>> < "selected_object_info": >>>>> "2:a53c12e8:::ROBJ6:head(dirty|omap|data_digest|omap_digest s 7 uv >>>>> 18 dd >>>>> 2ddbf8f5 od 689ee887 alloc_hint [0 0 0])", >>>>> --- >>>>>> "selected_object_info": >>>>> "2:a53c12e8:::ROBJ6:head(dirty|omap|data_digest s 7 uv 18 dd 2ddbf8f5 >>>>> alloc_hint [0 0 0])", >>>>> 401,403c393 >>>>> < "errors": [ >>>>> < "omap_digest_mismatch_oi" >>>>> < ], >>>>> --- >>>>>> "errors": [], >>>>> 409,411c399 >>>>> < "union_shard_errors": [ >>>>> < "omap_digest_mismatch_oi" >>>>> < ] >>>>> --- >>>>>> "union_shard_errors": [] >>>>> 424c412 >>>>> < "selected_object_info": >>>>> "2:8b55fa4b:::ROBJ7:head(dirty|omap|data_digest|omap_digest s 7 uv >>>>> 21 dd >>>>> 2ddbf8f5 od efced57a alloc_hint [0 0 0])", >>>>> --- >>>>>> "selected_object_info": >>>>> "2:8b55fa4b:::ROBJ7:head(dirty|omap|data_digest s 7 uv 21 dd 2ddbf8f5 >>>>> alloc_hint [0 0 0])", >>>>> 435,437c423 >>>>> < "errors": [ >>>>> < "omap_digest_mismatch_oi" >>>>> < ], >>>>> --- >>>>>> "errors": [], >>>>> 443,445c429 >>>>> < "union_shard_errors": [ >>>>> < "omap_digest_mismatch_oi" >>>>> < ] >>>>> --- >>>>>> "union_shard_errors": [] >>>>> 459c443 >>>>> < "selected_object_info": >>>>> "2:86586531:::ROBJ8:head(dirty|omap|data_digest|omap_digest s 7 uv >>>>> 26 dd >>>>> 2ddbf8f5 od d6be81dc alloc_hint [0 0 0])", >>>>> --- >>>>>> "selected_object_info": >>>>> "2:86586531:::ROBJ8:head(dirty|omap|data_digest s 7 uv 26 dd 2ddbf8f5 >>>>> alloc_hint [0 0 0])", >>>>> 532c516 >>>>> < "version": 56 >>>>> --- >>>>>> "version": 50 >>>>> 534c518 >>>>> < "selected_object_info": >>>>> "2:ffdb2004:::ROBJ9:head(dirty|omap|data_digest|omap_digest s 3 uv >>>>> 56 dd >>>>> 1f26fb26 od 2eecc539 alloc_hint [0 0 0])", >>>>> --- >>>>>> "selected_object_info": >>>>> "2:ffdb2004:::ROBJ9:head(dirty|omap|data_digest s 3 uv 50 dd 1f26fb26 >>>>> alloc_hint [0 0 0])", >>>>> 571c555 >>>>> < "object_info": >>>>> "2:ffdb2004:::ROBJ9:head(dirty|omap|data_digest|omap_digest s 3 uv >>>>> 56 dd >>>>> 1f26fb26 od 2eecc539 alloc_hint [0 0 0])", >>>>> --- >>>>>> "object_info": >>>>>> "2:ffdb2004:::ROBJ9:head(dirty|omap|data_digest >>>>> s 3 uv 50 dd 1f26fb26 alloc_hint [0 0 0])", >>>>> /home/jenkins/workspace/ceph-master/src/test/osd/osd-scrub-repair.sh:1569: >>>>> >>>>> TEST_corrupt_scrub_replicated: test no = yes >>>>> /home/jenkins/workspace/ceph-master/src/test/osd/osd-scrub-repair.sh:1569: >>>>> >>>>> -- >>>>> To unsubscribe from this list: send the line "unsubscribe >>>>> ceph-devel" in >>>>> the body of a message to majordomo@vger.kernel.org >>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>> >>> >>> -- >>> To unsubscribe from this list: send the line "unsubscribe >>> ceph-devel" in >>> the body of a message to majordomo@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >> > ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Continous failing osd-scrub-repair.sh 2017-06-01 8:16 ` David Zafman @ 2017-06-01 8:28 ` Willem Jan Withagen 0 siblings, 0 replies; 7+ messages in thread From: Willem Jan Withagen @ 2017-06-01 8:28 UTC (permalink / raw) To: David Zafman, Ceph Development, Josh Durgin On 01/06/2017 10:16, David Zafman wrote: > > Willem, > > Yes, I was wrong this wasn't caused by 3ca750d4. That change > change only affected the test that did EC on bluestore with EC > overwrites. > > If you have 44736aac74493b5532c9e3574a37a3642766f212 revert it and see > if that fixes your problem. > Thanx, I'll put this on my work queue... --WjW > David > > > On 6/1/17 1:11 AM, Willem Jan Withagen wrote: >> >> >> On 01/06/2017 03:11, David Zafman wrote: >>> >>> The change 3ca750d4 from Josh broke this. I didn't see before that >>> the test itself was broken. >>> >> Well then again, this is on FreeBSD, where your millage can (and >> will) vary. >> For one, we do not do BlueStore atm. >> >> So what is my best option: >> - See if it is possible if we can fix this? >> - Disable the test on basis of FreeBSD >> - Get a different diff-file to compare with? >> >> --WjW >> >>> David >>> >>> >>> On 5/31/17 4:14 PM, Willem Jan Withagen wrote: >>>> >>>> >>>> On 31/05/2017 23:38, David Zafman wrote: >>>>> >>>>> All the failures relate to not have the omap digest available in >>>>> the object info. That use to be present before so >>>>> >>>>> that the error omap_digest_mismatch_oi would be detected during >>>>> this test. >>>>> >>>> So, do you have any idea how this is caused? >>>> What has happened to not have omap digests any longer? >>>> >>>> --WjW >>>> >>>>> David >>>>> >>>>> >>>>> On 5/31/17 1:58 PM, Willem Jan Withagen wrote: >>>>>> Since a few days I have a continuous failing test: >>>>>> osd-scrub-repair.sh >>>>>> >>>>>> There is a diff failing, where the data looks almost correct. :( >>>>>> Looks there is something wrong with omap_digest. >>>>>> >>>>>> Could this be due to bluestore being enable, whilst FreeBSD does not >>>>>> have that? >>>>>> >>>>>> Anyways, all suggestions are welcome >>>>>> >>>>>> --WjW >>>>>> >>>>>> /home/jenkins/workspace/ceph-master/src/test/osd/osd-scrub-repair.sh:1569: >>>>>> >>>>>> TEST_corrupt_scrub_replicated: diff td/osd-scrub-repair/checkcsjson >>>>>> td/osd-scrub-repair/csjson >>>>>> 322c322 >>>>>> < "selected_object_info": >>>>>> "2:f4981d31:::ROBJ4:head(dirty|omap|data_digest|omap_digest s 7 >>>>>> uv 12 dd >>>>>> 2ddbf8f5 od d7178dfe alloc_hint [0 0 0])", >>>>>> --- >>>>>>> "selected_object_info": >>>>>> "2:f4981d31:::ROBJ4:head(dirty|omap|data_digest s 7 uv 12 dd >>>>>> 2ddbf8f5 >>>>>> alloc_hint [0 0 0])", >>>>>> 333,335c333 >>>>>> < "errors": [ >>>>>> < "omap_digest_mismatch_oi" >>>>>> < ], >>>>>> --- >>>>>>> "errors": [], >>>>>> 341,343c339 >>>>>> < "union_shard_errors": [ >>>>>> < "omap_digest_mismatch_oi" >>>>>> < ] >>>>>> --- >>>>>>> "union_shard_errors": [] >>>>>> 356c352 >>>>>> < "selected_object_info": >>>>>> "2:f4bfd4d1:::ROBJ5:head(dirty|omap|data_digest|omap_digest s 7 >>>>>> uv 15 dd >>>>>> 2ddbf8f5 od 1a862a41 alloc_hint [0 0 0])", >>>>>> --- >>>>>>> "selected_object_info": >>>>>> "2:f4bfd4d1:::ROBJ5:head(dirty|omap|data_digest s 7 uv 15 dd >>>>>> 2ddbf8f5 >>>>>> alloc_hint [0 0 0])", >>>>>> 367,369c363 >>>>>> < "errors": [ >>>>>> < "omap_digest_mismatch_oi" >>>>>> < ], >>>>>> --- >>>>>>> "errors": [], >>>>>> 375,377c369 >>>>>> < "union_shard_errors": [ >>>>>> < "omap_digest_mismatch_oi" >>>>>> < ] >>>>>> --- >>>>>>> "union_shard_errors": [] >>>>>> 390c382 >>>>>> < "selected_object_info": >>>>>> "2:a53c12e8:::ROBJ6:head(dirty|omap|data_digest|omap_digest s 7 >>>>>> uv 18 dd >>>>>> 2ddbf8f5 od 689ee887 alloc_hint [0 0 0])", >>>>>> --- >>>>>>> "selected_object_info": >>>>>> "2:a53c12e8:::ROBJ6:head(dirty|omap|data_digest s 7 uv 18 dd >>>>>> 2ddbf8f5 >>>>>> alloc_hint [0 0 0])", >>>>>> 401,403c393 >>>>>> < "errors": [ >>>>>> < "omap_digest_mismatch_oi" >>>>>> < ], >>>>>> --- >>>>>>> "errors": [], >>>>>> 409,411c399 >>>>>> < "union_shard_errors": [ >>>>>> < "omap_digest_mismatch_oi" >>>>>> < ] >>>>>> --- >>>>>>> "union_shard_errors": [] >>>>>> 424c412 >>>>>> < "selected_object_info": >>>>>> "2:8b55fa4b:::ROBJ7:head(dirty|omap|data_digest|omap_digest s 7 >>>>>> uv 21 dd >>>>>> 2ddbf8f5 od efced57a alloc_hint [0 0 0])", >>>>>> --- >>>>>>> "selected_object_info": >>>>>> "2:8b55fa4b:::ROBJ7:head(dirty|omap|data_digest s 7 uv 21 dd >>>>>> 2ddbf8f5 >>>>>> alloc_hint [0 0 0])", >>>>>> 435,437c423 >>>>>> < "errors": [ >>>>>> < "omap_digest_mismatch_oi" >>>>>> < ], >>>>>> --- >>>>>>> "errors": [], >>>>>> 443,445c429 >>>>>> < "union_shard_errors": [ >>>>>> < "omap_digest_mismatch_oi" >>>>>> < ] >>>>>> --- >>>>>>> "union_shard_errors": [] >>>>>> 459c443 >>>>>> < "selected_object_info": >>>>>> "2:86586531:::ROBJ8:head(dirty|omap|data_digest|omap_digest s 7 >>>>>> uv 26 dd >>>>>> 2ddbf8f5 od d6be81dc alloc_hint [0 0 0])", >>>>>> --- >>>>>>> "selected_object_info": >>>>>> "2:86586531:::ROBJ8:head(dirty|omap|data_digest s 7 uv 26 dd >>>>>> 2ddbf8f5 >>>>>> alloc_hint [0 0 0])", >>>>>> 532c516 >>>>>> < "version": 56 >>>>>> --- >>>>>>> "version": 50 >>>>>> 534c518 >>>>>> < "selected_object_info": >>>>>> "2:ffdb2004:::ROBJ9:head(dirty|omap|data_digest|omap_digest s 3 >>>>>> uv 56 dd >>>>>> 1f26fb26 od 2eecc539 alloc_hint [0 0 0])", >>>>>> --- >>>>>>> "selected_object_info": >>>>>> "2:ffdb2004:::ROBJ9:head(dirty|omap|data_digest s 3 uv 50 dd >>>>>> 1f26fb26 >>>>>> alloc_hint [0 0 0])", >>>>>> 571c555 >>>>>> < "object_info": >>>>>> "2:ffdb2004:::ROBJ9:head(dirty|omap|data_digest|omap_digest s 3 >>>>>> uv 56 dd >>>>>> 1f26fb26 od 2eecc539 alloc_hint [0 0 0])", >>>>>> --- >>>>>>> "object_info": >>>>>>> "2:ffdb2004:::ROBJ9:head(dirty|omap|data_digest >>>>>> s 3 uv 50 dd 1f26fb26 alloc_hint [0 0 0])", >>>>>> /home/jenkins/workspace/ceph-master/src/test/osd/osd-scrub-repair.sh:1569: >>>>>> >>>>>> TEST_corrupt_scrub_replicated: test no = yes >>>>>> /home/jenkins/workspace/ceph-master/src/test/osd/osd-scrub-repair.sh:1569: >>>>>> >>>>>> -- >>>>>> To unsubscribe from this list: send the line "unsubscribe >>>>>> ceph-devel" in >>>>>> the body of a message to majordomo@vger.kernel.org >>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>> >>>> >>>> -- >>>> To unsubscribe from this list: send the line "unsubscribe >>>> ceph-devel" in >>>> the body of a message to majordomo@vger.kernel.org >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> >> > ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2017-06-01 8:28 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2017-05-31 20:58 Continous failing osd-scrub-repair.sh Willem Jan Withagen 2017-05-31 21:38 ` David Zafman 2017-05-31 23:14 ` Willem Jan Withagen 2017-06-01 1:11 ` David Zafman 2017-06-01 8:11 ` Willem Jan Withagen 2017-06-01 8:16 ` David Zafman 2017-06-01 8:28 ` Willem Jan Withagen
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.