* RBD trim / unmap support?
@ 2012-11-01 23:33 Stefan Priebe
2012-11-01 23:36 ` Josh Durgin
0 siblings, 1 reply; 13+ messages in thread
From: Stefan Priebe @ 2012-11-01 23:33 UTC (permalink / raw)
To: ceph-devel
Hello list,
does rbd support trim / unmap? Or is it planned to support it?
Greets,
Stefan
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: RBD trim / unmap support?
2012-11-01 23:33 RBD trim / unmap support? Stefan Priebe
@ 2012-11-01 23:36 ` Josh Durgin
2012-11-02 8:20 ` Stefan Priebe - Profihost AG
0 siblings, 1 reply; 13+ messages in thread
From: Josh Durgin @ 2012-11-01 23:36 UTC (permalink / raw)
To: Stefan Priebe; +Cc: ceph-devel
On 11/01/2012 04:33 PM, Stefan Priebe wrote:
> Hello list,
>
> does rbd support trim / unmap? Or is it planned to support it?
>
> Greets,
> Stefan
librbd (and thus qemu) support it. The rbd kernel module does not yet.
See http://ceph.com/docs/master/rbd/qemu-rbd/#enabling-discard-trim
Josh
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: RBD trim / unmap support?
2012-11-01 23:36 ` Josh Durgin
@ 2012-11-02 8:20 ` Stefan Priebe - Profihost AG
2012-11-02 8:21 ` Dietmar Maurer
2012-11-02 9:08 ` Stefan Priebe - Profihost AG
0 siblings, 2 replies; 13+ messages in thread
From: Stefan Priebe - Profihost AG @ 2012-11-02 8:20 UTC (permalink / raw)
To: Josh Durgin; +Cc: ceph-devel
Am 02.11.2012 00:36, schrieb Josh Durgin:
> On 11/01/2012 04:33 PM, Stefan Priebe wrote:
>> Hello list,
>>
>> does rbd support trim / unmap? Or is it planned to support it?
>>
>> Greets,
>> Stefan
>
> librbd (and thus qemu) support it. The rbd kernel module does not yet.
> See http://ceph.com/docs/master/rbd/qemu-rbd/#enabling-discard-trim
Thanks! Is there any recommanded value for discard_granularity? With
fstrim and iscsi i use 128kb.
Stefan
^ permalink raw reply [flat|nested] 13+ messages in thread
* RE: RBD trim / unmap support?
2012-11-02 8:20 ` Stefan Priebe - Profihost AG
@ 2012-11-02 8:21 ` Dietmar Maurer
2012-11-02 9:08 ` Stefan Priebe - Profihost AG
1 sibling, 0 replies; 13+ messages in thread
From: Dietmar Maurer @ 2012-11-02 8:21 UTC (permalink / raw)
To: Stefan Priebe - Profihost AG, Josh Durgin; +Cc: ceph-devel
hw/ide/qdev.c: error_report("discard_granularity must be 512 for ide");
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-
> owner@vger.kernel.org] On Behalf Of Stefan Priebe - Profihost AG
> Sent: Freitag, 02. November 2012 09:20
> To: Josh Durgin
> Cc: ceph-devel@vger.kernel.org
> Subject: Re: RBD trim / unmap support?
>
> Am 02.11.2012 00:36, schrieb Josh Durgin:
> > On 11/01/2012 04:33 PM, Stefan Priebe wrote:
> >> Hello list,
> >>
> >> does rbd support trim / unmap? Or is it planned to support it?
> >>
> >> Greets,
> >> Stefan
> >
> > librbd (and thus qemu) support it. The rbd kernel module does not yet.
> > See http://ceph.com/docs/master/rbd/qemu-rbd/#enabling-discard-trim
>
> Thanks! Is there any recommanded value for discard_granularity? With
> fstrim and iscsi i use 128kb.
>
> Stefan
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the
> body of a message to majordomo@vger.kernel.org More majordomo info at
> http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: RBD trim / unmap support?
2012-11-02 8:20 ` Stefan Priebe - Profihost AG
2012-11-02 8:21 ` Dietmar Maurer
@ 2012-11-02 9:08 ` Stefan Priebe - Profihost AG
2012-11-02 14:18 ` [Qemu-devel] " Stefan Priebe - Profihost AG
2012-11-02 16:16 ` Josh Durgin
1 sibling, 2 replies; 13+ messages in thread
From: Stefan Priebe - Profihost AG @ 2012-11-02 9:08 UTC (permalink / raw)
To: Josh Durgin; +Cc: ceph-devel
Hi Jost,
i tried scsi-hd with discard_granularity=512 the UNMAP / Trim itself
works fine the space is freed at ceph but at the VM i'm getting:
[ 75.076895] sd 2:0:0:4: [sdc]
[ 75.078353] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[ 75.079377] sd 2:0:0:4: [sdc]
[ 75.080410] Sense Key : Aborted Command [current]
[ 75.081446] sd 2:0:0:4: [sdc]
[ 75.082459] Add. Sense: I/O process terminated
[ 75.083466] sd 2:0:0:4: [sdc] CDB:
[ 75.084473] Write same(16): 93 08 00 00 00 00 00 00 00 00 00 7f ff ff
00 00
[ 75.085535] end_request: I/O error, dev sdc, sector 0
[ 75.086567] sd 2:0:0:4: [sdc]
[ 75.087569] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[ 75.088605] sd 2:0:0:4: [sdc]
[ 75.089610] Sense Key : Aborted Command [current]
[ 75.090630] sd 2:0:0:4: [sdc]
[ 75.091631] Add. Sense: I/O process terminated
[ 75.092621] sd 2:0:0:4: [sdc] CDB:
[ 75.093579] Write same(16): 93 08 00 00 00 00 00 7f ff ff 00 7f ff ff
00 00
[ 75.094581] end_request: I/O error, dev sdc, sector 8388607
[ 75.095558] sd 2:0:0:4: [sdc]
[ 75.096561] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[ 75.097553] sd 2:0:0:4: [sdc]
[ 75.098520] Sense Key : Aborted Command [current]
[ 75.099484] sd 2:0:0:4: [sdc]
[ 75.100471] Add. Sense: I/O process terminated
[ 75.101439] sd 2:0:0:4: [sdc] CDB:
[ 75.102432] Write same(16): 93 08 00 00 00 00 00 ff ff fe 00 7f ff ff
00 00
[ 75.103464] end_request: I/O error, dev sdc, sector 16777214
[ 75.104503] sd 2:0:0:4: [sdc]
[ 75.105507] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[ 75.106514] sd 2:0:0:4: [sdc]
[ 75.107525] Sense Key : Aborted Command [current]
[ 75.108545] sd 2:0:0:4: [sdc]
[ 75.109525] Add. Sense: I/O process terminated
[ 75.110487] sd 2:0:0:4: [sdc] CDB:
[ 75.111418] Write same(16): 93 08 00 00 00 00 01 7f ff fd 00 7f ff ff
00 00
[ 75.112419] end_request: I/O error, dev sdc, sector 25165821
[ 75.238627] sd 2:0:0:4: [sdc]
[ 75.239946] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[ 75.241197] sd 2:0:0:4: [sdc]
[ 75.242097] Sense Key : Aborted Command [current]
[ 75.242627] sd 2:0:0:4: [sdc]
[ 75.242627] Add. Sense: I/O process terminated
[ 75.242627] sd 2:0:0:4: [sdc] CDB:
[ 75.242627] Write same(16): 93 08 00 00 00 00 01 ff ff fc 00 7f ff ff
00 00
[ 75.242627] end_request: I/O error, dev sdc, sector 33554428
[ 75.447311] sd 2:0:0:4: [sdc]
[ 75.448505] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[ 75.449392] sd 2:0:0:4: [sdc]
[ 75.450262] Sense Key : Aborted Command [current]
[ 75.451134] sd 2:0:0:4: [sdc]
[ 75.451312] Add. Sense: I/O process terminated
[ 75.451312] sd 2:0:0:4: [sdc] CDB:
[ 75.451312] Write same(16): 93 08 00 00 00 00 02 7f ff fb 00 7f ff ff
00 00
[ 75.451312] end_request: I/O error, dev sdc, sector 41943035
[ 75.476364] sd 2:0:0:4: [sdc]
[ 75.477252] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[ 75.478135] sd 2:0:0:4: [sdc]
[ 75.479004] Sense Key : Aborted Command [current]
[ 75.479891] sd 2:0:0:4: [sdc]
[ 75.480358] Add. Sense: I/O process terminated
[ 75.480358] sd 2:0:0:4: [sdc] CDB:
[ 75.480358] Write same(16): 93 08 00 00 00 00 02 ff ff fa 00 7f ff ff
00 00
[ 75.480358] end_request: I/O error, dev sdc, sector 50331642
[ 75.496374] sd 2:0:0:4: [sdc]
[ 75.497202] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[ 75.498077] sd 2:0:0:4: [sdc]
[ 75.498948] Sense Key : Aborted Command [current]
[ 75.499818] sd 2:0:0:4: [sdc]
[ 75.500374] Add. Sense: I/O process terminated
[ 75.500374] sd 2:0:0:4: [sdc] CDB:
[ 75.500374] Write same(16): 93 08 00 00 00 00 03 7f ff f9 00 7f ff ff
00 00
[ 75.500374] end_request: I/O error, dev sdc, sector 58720249
Stefan
Am 02.11.2012 09:20, schrieb Stefan Priebe - Profihost AG:
> Am 02.11.2012 00:36, schrieb Josh Durgin:
>> On 11/01/2012 04:33 PM, Stefan Priebe wrote:
>>> Hello list,
>>>
>>> does rbd support trim / unmap? Or is it planned to support it?
>>>
>>> Greets,
>>> Stefan
>>
>> librbd (and thus qemu) support it. The rbd kernel module does not yet.
>> See http://ceph.com/docs/master/rbd/qemu-rbd/#enabling-discard-trim
>
> Thanks! Is there any recommanded value for discard_granularity? With
> fstrim and iscsi i use 128kb.
>
> Stefan
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: RBD trim / unmap support?
2012-11-02 9:08 ` Stefan Priebe - Profihost AG
@ 2012-11-02 14:18 ` Stefan Priebe - Profihost AG
2012-11-02 16:16 ` Josh Durgin
1 sibling, 0 replies; 13+ messages in thread
From: Stefan Priebe - Profihost AG @ 2012-11-02 14:18 UTC (permalink / raw)
To: Josh Durgin; +Cc: ceph-devel, qemu-devel
Hello qemu list,
i cc you as i'm not sure where the problem is.
When i use scsi-hd with discard_granularity the discard works fine on
target size but the client print these - immediatly after sending
discard / trim commands with mkfs.btrfs.
> [ 75.076895] sd 2:0:0:4: [sdc]
> [ 75.078353] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [ 75.079377] sd 2:0:0:4: [sdc]
> [ 75.080410] Sense Key : Aborted Command [current]
> [ 75.081446] sd 2:0:0:4: [sdc]
> [ 75.082459] Add. Sense: I/O process terminated
> [ 75.083466] sd 2:0:0:4: [sdc] CDB:
> [ 75.084473] Write same(16): 93 08 00 00 00 00 00 00 00 00 00 7f ff ff
> 00 00
> [ 75.085535] end_request: I/O error, dev sdc, sector 0
> [ 75.086567] sd 2:0:0:4: [sdc]
> [ 75.087569] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [ 75.088605] sd 2:0:0:4: [sdc]
> [ 75.089610] Sense Key : Aborted Command [current]
> [ 75.090630] sd 2:0:0:4: [sdc]
> [ 75.091631] Add. Sense: I/O process terminated
> [ 75.092621] sd 2:0:0:4: [sdc] CDB:
> [ 75.093579] Write same(16): 93 08 00 00 00 00 00 7f ff ff 00 7f ff ff
> 00 00
> [ 75.094581] end_request: I/O error, dev sdc, sector 8388607
> [ 75.095558] sd 2:0:0:4: [sdc]
> [ 75.096561] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [ 75.097553] sd 2:0:0:4: [sdc]
> [ 75.098520] Sense Key : Aborted Command [current]
> [ 75.099484] sd 2:0:0:4: [sdc]
> [ 75.100471] Add. Sense: I/O process terminated
> [ 75.101439] sd 2:0:0:4: [sdc] CDB:
> [ 75.102432] Write same(16): 93 08 00 00 00 00 00 ff ff fe 00 7f ff ff
> 00 00
> [ 75.103464] end_request: I/O error, dev sdc, sector 16777214
> [ 75.104503] sd 2:0:0:4: [sdc]
> [ 75.105507] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [ 75.106514] sd 2:0:0:4: [sdc]
> [ 75.107525] Sense Key : Aborted Command [current]
> [ 75.108545] sd 2:0:0:4: [sdc]
> [ 75.109525] Add. Sense: I/O process terminated
> [ 75.110487] sd 2:0:0:4: [sdc] CDB:
> [ 75.111418] Write same(16): 93 08 00 00 00 00 01 7f ff fd 00 7f ff ff
> 00 00
> [ 75.112419] end_request: I/O error, dev sdc, sector 25165821
> [ 75.238627] sd 2:0:0:4: [sdc]
> [ 75.239946] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [ 75.241197] sd 2:0:0:4: [sdc]
> [ 75.242097] Sense Key : Aborted Command [current]
> [ 75.242627] sd 2:0:0:4: [sdc]
> [ 75.242627] Add. Sense: I/O process terminated
> [ 75.242627] sd 2:0:0:4: [sdc] CDB:
> [ 75.242627] Write same(16): 93 08 00 00 00 00 01 ff ff fc 00 7f ff ff
> 00 00
> [ 75.242627] end_request: I/O error, dev sdc, sector 33554428
> [ 75.447311] sd 2:0:0:4: [sdc]
> [ 75.448505] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [ 75.449392] sd 2:0:0:4: [sdc]
> [ 75.450262] Sense Key : Aborted Command [current]
> [ 75.451134] sd 2:0:0:4: [sdc]
> [ 75.451312] Add. Sense: I/O process terminated
> [ 75.451312] sd 2:0:0:4: [sdc] CDB:
> [ 75.451312] Write same(16): 93 08 00 00 00 00 02 7f ff fb 00 7f ff ff
> 00 00
> [ 75.451312] end_request: I/O error, dev sdc, sector 41943035
> [ 75.476364] sd 2:0:0:4: [sdc]
> [ 75.477252] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [ 75.478135] sd 2:0:0:4: [sdc]
> [ 75.479004] Sense Key : Aborted Command [current]
> [ 75.479891] sd 2:0:0:4: [sdc]
> [ 75.480358] Add. Sense: I/O process terminated
> [ 75.480358] sd 2:0:0:4: [sdc] CDB:
> [ 75.480358] Write same(16): 93 08 00 00 00 00 02 ff ff fa 00 7f ff ff
> 00 00
> [ 75.480358] end_request: I/O error, dev sdc, sector 50331642
> [ 75.496374] sd 2:0:0:4: [sdc]
> [ 75.497202] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [ 75.498077] sd 2:0:0:4: [sdc]
> [ 75.498948] Sense Key : Aborted Command [current]
> [ 75.499818] sd 2:0:0:4: [sdc]
> [ 75.500374] Add. Sense: I/O process terminated
> [ 75.500374] sd 2:0:0:4: [sdc] CDB:
> [ 75.500374] Write same(16): 93 08 00 00 00 00 03 7f ff f9 00 7f ff ff
> 00 00
> [ 75.500374] end_request: I/O error, dev sdc, sector 58720249
>
> Stefan
>
> Am 02.11.2012 09:20, schrieb Stefan Priebe - Profihost AG:
>> Am 02.11.2012 00:36, schrieb Josh Durgin:
>>> On 11/01/2012 04:33 PM, Stefan Priebe wrote:
>>>> Hello list,
>>>>
>>>> does rbd support trim / unmap? Or is it planned to support it?
>>>>
>>>> Greets,
>>>> Stefan
>>>
>>> librbd (and thus qemu) support it. The rbd kernel module does not yet.
>>> See http://ceph.com/docs/master/rbd/qemu-rbd/#enabling-discard-trim
>>
>> Thanks! Is there any recommanded value for discard_granularity? With
>> fstrim and iscsi i use 128kb.
>>
>> Stefan
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [Qemu-devel] RBD trim / unmap support?
@ 2012-11-02 14:18 ` Stefan Priebe - Profihost AG
0 siblings, 0 replies; 13+ messages in thread
From: Stefan Priebe - Profihost AG @ 2012-11-02 14:18 UTC (permalink / raw)
To: Josh Durgin; +Cc: ceph-devel, qemu-devel
Hello qemu list,
i cc you as i'm not sure where the problem is.
When i use scsi-hd with discard_granularity the discard works fine on
target size but the client print these - immediatly after sending
discard / trim commands with mkfs.btrfs.
> [ 75.076895] sd 2:0:0:4: [sdc]
> [ 75.078353] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [ 75.079377] sd 2:0:0:4: [sdc]
> [ 75.080410] Sense Key : Aborted Command [current]
> [ 75.081446] sd 2:0:0:4: [sdc]
> [ 75.082459] Add. Sense: I/O process terminated
> [ 75.083466] sd 2:0:0:4: [sdc] CDB:
> [ 75.084473] Write same(16): 93 08 00 00 00 00 00 00 00 00 00 7f ff ff
> 00 00
> [ 75.085535] end_request: I/O error, dev sdc, sector 0
> [ 75.086567] sd 2:0:0:4: [sdc]
> [ 75.087569] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [ 75.088605] sd 2:0:0:4: [sdc]
> [ 75.089610] Sense Key : Aborted Command [current]
> [ 75.090630] sd 2:0:0:4: [sdc]
> [ 75.091631] Add. Sense: I/O process terminated
> [ 75.092621] sd 2:0:0:4: [sdc] CDB:
> [ 75.093579] Write same(16): 93 08 00 00 00 00 00 7f ff ff 00 7f ff ff
> 00 00
> [ 75.094581] end_request: I/O error, dev sdc, sector 8388607
> [ 75.095558] sd 2:0:0:4: [sdc]
> [ 75.096561] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [ 75.097553] sd 2:0:0:4: [sdc]
> [ 75.098520] Sense Key : Aborted Command [current]
> [ 75.099484] sd 2:0:0:4: [sdc]
> [ 75.100471] Add. Sense: I/O process terminated
> [ 75.101439] sd 2:0:0:4: [sdc] CDB:
> [ 75.102432] Write same(16): 93 08 00 00 00 00 00 ff ff fe 00 7f ff ff
> 00 00
> [ 75.103464] end_request: I/O error, dev sdc, sector 16777214
> [ 75.104503] sd 2:0:0:4: [sdc]
> [ 75.105507] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [ 75.106514] sd 2:0:0:4: [sdc]
> [ 75.107525] Sense Key : Aborted Command [current]
> [ 75.108545] sd 2:0:0:4: [sdc]
> [ 75.109525] Add. Sense: I/O process terminated
> [ 75.110487] sd 2:0:0:4: [sdc] CDB:
> [ 75.111418] Write same(16): 93 08 00 00 00 00 01 7f ff fd 00 7f ff ff
> 00 00
> [ 75.112419] end_request: I/O error, dev sdc, sector 25165821
> [ 75.238627] sd 2:0:0:4: [sdc]
> [ 75.239946] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [ 75.241197] sd 2:0:0:4: [sdc]
> [ 75.242097] Sense Key : Aborted Command [current]
> [ 75.242627] sd 2:0:0:4: [sdc]
> [ 75.242627] Add. Sense: I/O process terminated
> [ 75.242627] sd 2:0:0:4: [sdc] CDB:
> [ 75.242627] Write same(16): 93 08 00 00 00 00 01 ff ff fc 00 7f ff ff
> 00 00
> [ 75.242627] end_request: I/O error, dev sdc, sector 33554428
> [ 75.447311] sd 2:0:0:4: [sdc]
> [ 75.448505] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [ 75.449392] sd 2:0:0:4: [sdc]
> [ 75.450262] Sense Key : Aborted Command [current]
> [ 75.451134] sd 2:0:0:4: [sdc]
> [ 75.451312] Add. Sense: I/O process terminated
> [ 75.451312] sd 2:0:0:4: [sdc] CDB:
> [ 75.451312] Write same(16): 93 08 00 00 00 00 02 7f ff fb 00 7f ff ff
> 00 00
> [ 75.451312] end_request: I/O error, dev sdc, sector 41943035
> [ 75.476364] sd 2:0:0:4: [sdc]
> [ 75.477252] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [ 75.478135] sd 2:0:0:4: [sdc]
> [ 75.479004] Sense Key : Aborted Command [current]
> [ 75.479891] sd 2:0:0:4: [sdc]
> [ 75.480358] Add. Sense: I/O process terminated
> [ 75.480358] sd 2:0:0:4: [sdc] CDB:
> [ 75.480358] Write same(16): 93 08 00 00 00 00 02 ff ff fa 00 7f ff ff
> 00 00
> [ 75.480358] end_request: I/O error, dev sdc, sector 50331642
> [ 75.496374] sd 2:0:0:4: [sdc]
> [ 75.497202] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [ 75.498077] sd 2:0:0:4: [sdc]
> [ 75.498948] Sense Key : Aborted Command [current]
> [ 75.499818] sd 2:0:0:4: [sdc]
> [ 75.500374] Add. Sense: I/O process terminated
> [ 75.500374] sd 2:0:0:4: [sdc] CDB:
> [ 75.500374] Write same(16): 93 08 00 00 00 00 03 7f ff f9 00 7f ff ff
> 00 00
> [ 75.500374] end_request: I/O error, dev sdc, sector 58720249
>
> Stefan
>
> Am 02.11.2012 09:20, schrieb Stefan Priebe - Profihost AG:
>> Am 02.11.2012 00:36, schrieb Josh Durgin:
>>> On 11/01/2012 04:33 PM, Stefan Priebe wrote:
>>>> Hello list,
>>>>
>>>> does rbd support trim / unmap? Or is it planned to support it?
>>>>
>>>> Greets,
>>>> Stefan
>>>
>>> librbd (and thus qemu) support it. The rbd kernel module does not yet.
>>> See http://ceph.com/docs/master/rbd/qemu-rbd/#enabling-discard-trim
>>
>> Thanks! Is there any recommanded value for discard_granularity? With
>> fstrim and iscsi i use 128kb.
>>
>> Stefan
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: RBD trim / unmap support?
2012-11-02 9:08 ` Stefan Priebe - Profihost AG
2012-11-02 14:18 ` [Qemu-devel] " Stefan Priebe - Profihost AG
@ 2012-11-02 16:16 ` Josh Durgin
2012-11-02 20:13 ` Stefan Priebe
2012-11-06 22:45 ` Stefan Priebe
1 sibling, 2 replies; 13+ messages in thread
From: Josh Durgin @ 2012-11-02 16:16 UTC (permalink / raw)
To: Stefan Priebe - Profihost AG; +Cc: ceph-devel
On 11/02/2012 02:08 AM, Stefan Priebe - Profihost AG wrote:
> Hi Jost,
>
> i tried scsi-hd with discard_granularity=512 the UNMAP / Trim itself
> works fine the space is freed at ceph but at the VM i'm getting:
> [ 75.076895] sd 2:0:0:4: [sdc]
> [ 75.078353] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [ 75.079377] sd 2:0:0:4: [sdc]
> [ 75.080410] Sense Key : Aborted Command [current]
> [ 75.081446] sd 2:0:0:4: [sdc]
> [ 75.082459] Add. Sense: I/O process terminated
> [ 75.083466] sd 2:0:0:4: [sdc] CDB:
> [ 75.084473] Write same(16): 93 08 00 00 00 00 00 00 00 00 00 7f ff ff
> 00 00
> [ 75.085535] end_request: I/O error, dev sdc, sector 0
> [ 75.086567] sd 2:0:0:4: [sdc]
> [ 75.087569] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [ 75.088605] sd 2:0:0:4: [sdc]
> [ 75.089610] Sense Key : Aborted Command [current]
> [ 75.090630] sd 2:0:0:4: [sdc]
> [ 75.091631] Add. Sense: I/O process terminated
> [ 75.092621] sd 2:0:0:4: [sdc] CDB:
> [ 75.093579] Write same(16): 93 08 00 00 00 00 00 7f ff ff 00 7f ff ff
> 00 00
> [ 75.094581] end_request: I/O error, dev sdc, sector 8388607
> [ 75.095558] sd 2:0:0:4: [sdc]
> [ 75.096561] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [ 75.097553] sd 2:0:0:4: [sdc]
> [ 75.098520] Sense Key : Aborted Command [current]
> [ 75.099484] sd 2:0:0:4: [sdc]
> [ 75.100471] Add. Sense: I/O process terminated
> [ 75.101439] sd 2:0:0:4: [sdc] CDB:
> [ 75.102432] Write same(16): 93 08 00 00 00 00 00 ff ff fe 00 7f ff ff
> 00 00
> [ 75.103464] end_request: I/O error, dev sdc, sector 16777214
> [ 75.104503] sd 2:0:0:4: [sdc]
> [ 75.105507] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [ 75.106514] sd 2:0:0:4: [sdc]
> [ 75.107525] Sense Key : Aborted Command [current]
> [ 75.108545] sd 2:0:0:4: [sdc]
> [ 75.109525] Add. Sense: I/O process terminated
> [ 75.110487] sd 2:0:0:4: [sdc] CDB:
> [ 75.111418] Write same(16): 93 08 00 00 00 00 01 7f ff fd 00 7f ff ff
> 00 00
> [ 75.112419] end_request: I/O error, dev sdc, sector 25165821
> [ 75.238627] sd 2:0:0:4: [sdc]
> [ 75.239946] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [ 75.241197] sd 2:0:0:4: [sdc]
> [ 75.242097] Sense Key : Aborted Command [current]
> [ 75.242627] sd 2:0:0:4: [sdc]
> [ 75.242627] Add. Sense: I/O process terminated
> [ 75.242627] sd 2:0:0:4: [sdc] CDB:
> [ 75.242627] Write same(16): 93 08 00 00 00 00 01 ff ff fc 00 7f ff ff
> 00 00
> [ 75.242627] end_request: I/O error, dev sdc, sector 33554428
> [ 75.447311] sd 2:0:0:4: [sdc]
> [ 75.448505] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [ 75.449392] sd 2:0:0:4: [sdc]
> [ 75.450262] Sense Key : Aborted Command [current]
> [ 75.451134] sd 2:0:0:4: [sdc]
> [ 75.451312] Add. Sense: I/O process terminated
> [ 75.451312] sd 2:0:0:4: [sdc] CDB:
> [ 75.451312] Write same(16): 93 08 00 00 00 00 02 7f ff fb 00 7f ff ff
> 00 00
> [ 75.451312] end_request: I/O error, dev sdc, sector 41943035
> [ 75.476364] sd 2:0:0:4: [sdc]
> [ 75.477252] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [ 75.478135] sd 2:0:0:4: [sdc]
> [ 75.479004] Sense Key : Aborted Command [current]
> [ 75.479891] sd 2:0:0:4: [sdc]
> [ 75.480358] Add. Sense: I/O process terminated
> [ 75.480358] sd 2:0:0:4: [sdc] CDB:
> [ 75.480358] Write same(16): 93 08 00 00 00 00 02 ff ff fa 00 7f ff ff
> 00 00
> [ 75.480358] end_request: I/O error, dev sdc, sector 50331642
> [ 75.496374] sd 2:0:0:4: [sdc]
> [ 75.497202] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [ 75.498077] sd 2:0:0:4: [sdc]
> [ 75.498948] Sense Key : Aborted Command [current]
> [ 75.499818] sd 2:0:0:4: [sdc]
> [ 75.500374] Add. Sense: I/O process terminated
> [ 75.500374] sd 2:0:0:4: [sdc] CDB:
> [ 75.500374] Write same(16): 93 08 00 00 00 00 03 7f ff f9 00 7f ff ff
> 00 00
> [ 75.500374] end_request: I/O error, dev sdc, sector 58720249
Could you add 'debug ms = 1', 'debug rbd = 20', and 'log file =
/tmp/librbd.log' to the [client] section of your ceph.conf?
That log should tell us where the error is happening.
Josh
> Stefan
>
> Am 02.11.2012 09:20, schrieb Stefan Priebe - Profihost AG:
>> Am 02.11.2012 00:36, schrieb Josh Durgin:
>>> On 11/01/2012 04:33 PM, Stefan Priebe wrote:
>>>> Hello list,
>>>>
>>>> does rbd support trim / unmap? Or is it planned to support it?
>>>>
>>>> Greets,
>>>> Stefan
>>>
>>> librbd (and thus qemu) support it. The rbd kernel module does not yet.
>>> See http://ceph.com/docs/master/rbd/qemu-rbd/#enabling-discard-trim
>>
>> Thanks! Is there any recommanded value for discard_granularity? With
>> fstrim and iscsi i use 128kb.
>>
>> Stefan
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: RBD trim / unmap support?
2012-11-02 16:16 ` Josh Durgin
@ 2012-11-02 20:13 ` Stefan Priebe
2012-11-03 18:47 ` Stefan Priebe
2012-11-06 22:45 ` Stefan Priebe
1 sibling, 1 reply; 13+ messages in thread
From: Stefan Priebe @ 2012-11-02 20:13 UTC (permalink / raw)
To: Josh Durgin; +Cc: ceph-devel
>> [ 75.480358] end_request: I/O error, dev sdc, sector 50331642
>> [ 75.496374] sd 2:0:0:4: [sdc]
>> [ 75.497202] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
>> [ 75.498077] sd 2:0:0:4: [sdc]
>> [ 75.498948] Sense Key : Aborted Command [current]
>> [ 75.499818] sd 2:0:0:4: [sdc]
>> [ 75.500374] Add. Sense: I/O process terminated
>> [ 75.500374] sd 2:0:0:4: [sdc] CDB:
>> [ 75.500374] Write same(16): 93 08 00 00 00 00 03 7f ff f9 00 7f ff ff
>> 00 00
>> [ 75.500374] end_request: I/O error, dev sdc, sector 58720249
>
> Could you add 'debug ms = 1', 'debug rbd = 20', and 'log file =
> /tmp/librbd.log' to the [client] section of your ceph.conf?
>
> That log should tell us where the error is happening.
Thanks - here it comes:
https://www.dropbox.com/s/blegk8u14n30ntq/librbd.log.gz
Hope that helps and we can fix it before 0.54.
Greets
Stefan
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: RBD trim / unmap support?
2012-11-02 20:13 ` Stefan Priebe
@ 2012-11-03 18:47 ` Stefan Priebe
2012-11-06 8:09 ` Stefan Priebe - Profihost AG
0 siblings, 1 reply; 13+ messages in thread
From: Stefan Priebe @ 2012-11-03 18:47 UTC (permalink / raw)
To: Josh Durgin; +Cc: ceph-devel
Hi Josh,
> Thanks - here it comes:
> https://www.dropbox.com/s/blegk8u14n30ntq/librbd.log.gz
It this one OK? Do you get an idea what this could be?
Stefan
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: RBD trim / unmap support?
2012-11-03 18:47 ` Stefan Priebe
@ 2012-11-06 8:09 ` Stefan Priebe - Profihost AG
0 siblings, 0 replies; 13+ messages in thread
From: Stefan Priebe - Profihost AG @ 2012-11-06 8:09 UTC (permalink / raw)
To: Josh Durgin; +Cc: ceph-devel
Am 03.11.2012 19:47, schrieb Stefan Priebe:
> Hi Josh,
>
>> Thanks - here it comes:
>> https://www.dropbox.com/s/blegk8u14n30ntq/librbd.log.gz
>
> It this one OK? Do you get an idea what this could be?
I checked this again today with latest git and wip-rbd-read branch but i
still get the same I/O errors.
Stefan
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: RBD trim / unmap support?
2012-11-02 16:16 ` Josh Durgin
2012-11-02 20:13 ` Stefan Priebe
@ 2012-11-06 22:45 ` Stefan Priebe
2012-11-11 12:08 ` Stefan Priebe
1 sibling, 1 reply; 13+ messages in thread
From: Stefan Priebe @ 2012-11-06 22:45 UTC (permalink / raw)
To: Josh Durgin; +Cc: ceph-devel
Am 02.11.2012 17:16, schrieb Josh Durgin:
> On 11/02/2012 02:08 AM, Stefan Priebe - Profihost AG wrote:
>> Hi Jost,
>>
>> i tried scsi-hd with discard_granularity=512 the UNMAP / Trim itself
>> works fine the space is freed at ceph but at the VM i'm getting:
>> [ 75.076895] sd 2:0:0:4: [sdc]
>> [ 75.078353] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
>> [ 75.079377] sd 2:0:0:4: [sdc]
>> [ 75.080410] Sense Key : Aborted Command [current]
>> [ 75.081446] sd 2:0:0:4: [sdc]
>> [ 75.082459] Add. Sense: I/O process terminated
>> [ 75.083466] sd 2:0:0:4: [sdc] CDB:
>> [ 75.084473] Write same(16): 93 08 00 00 00 00 00 00 00 00 00 7f ff ff
>> 00 00
>> [ 75.085535] end_request: I/O error, dev sdc, sector 0
>> [ 75.086567] sd 2:0:0:4: [sdc]
>> [ 75.087569] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
>> [ 75.088605] sd 2:0:0:4: [sdc]
>> [ 75.089610] Sense Key : Aborted Command [current]
>> [ 75.090630] sd 2:0:0:4: [sdc]
>> [ 75.091631] Add. Sense: I/O process terminated
>> [ 75.092621] sd 2:0:0:4: [sdc] CDB:
>> [ 75.093579] Write same(16): 93 08 00 00 00 00 00 7f ff ff 00 7f ff ff
>> 00 00
>> [ 75.094581] end_request: I/O error, dev sdc, sector 8388607
>> [ 75.095558] sd 2:0:0:4: [sdc]
>> [ 75.096561] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
>> [ 75.097553] sd 2:0:0:4: [sdc]
>> [ 75.098520] Sense Key : Aborted Command [current]
>> [ 75.099484] sd 2:0:0:4: [sdc]
>> [ 75.100471] Add. Sense: I/O process terminated
>> [ 75.101439] sd 2:0:0:4: [sdc] CDB:
>> [ 75.102432] Write same(16): 93 08 00 00 00 00 00 ff ff fe 00 7f ff ff
>> 00 00
>> [ 75.103464] end_request: I/O error, dev sdc, sector 16777214
>> [ 75.104503] sd 2:0:0:4: [sdc]
>> [ 75.105507] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
>> [ 75.106514] sd 2:0:0:4: [sdc]
>> [ 75.107525] Sense Key : Aborted Command [current]
>> [ 75.108545] sd 2:0:0:4: [sdc]
>> [ 75.109525] Add. Sense: I/O process terminated
>> [ 75.110487] sd 2:0:0:4: [sdc] CDB:
>> [ 75.111418] Write same(16): 93 08 00 00 00 00 01 7f ff fd 00 7f ff ff
>> 00 00
>> [ 75.112419] end_request: I/O error, dev sdc, sector 25165821
>> [ 75.238627] sd 2:0:0:4: [sdc]
>> [ 75.239946] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
>> [ 75.241197] sd 2:0:0:4: [sdc]
>> [ 75.242097] Sense Key : Aborted Command [current]
>> [ 75.242627] sd 2:0:0:4: [sdc]
>> [ 75.242627] Add. Sense: I/O process terminated
>> [ 75.242627] sd 2:0:0:4: [sdc] CDB:
>> [ 75.242627] Write same(16): 93 08 00 00 00 00 01 ff ff fc 00 7f ff ff
>> 00 00
>> [ 75.242627] end_request: I/O error, dev sdc, sector 33554428
>> [ 75.447311] sd 2:0:0:4: [sdc]
>> [ 75.448505] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
>> [ 75.449392] sd 2:0:0:4: [sdc]
>> [ 75.450262] Sense Key : Aborted Command [current]
>> [ 75.451134] sd 2:0:0:4: [sdc]
>> [ 75.451312] Add. Sense: I/O process terminated
>> [ 75.451312] sd 2:0:0:4: [sdc] CDB:
>> [ 75.451312] Write same(16): 93 08 00 00 00 00 02 7f ff fb 00 7f ff ff
>> 00 00
>> [ 75.451312] end_request: I/O error, dev sdc, sector 41943035
>> [ 75.476364] sd 2:0:0:4: [sdc]
>> [ 75.477252] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
>> [ 75.478135] sd 2:0:0:4: [sdc]
>> [ 75.479004] Sense Key : Aborted Command [current]
>> [ 75.479891] sd 2:0:0:4: [sdc]
>> [ 75.480358] Add. Sense: I/O process terminated
>> [ 75.480358] sd 2:0:0:4: [sdc] CDB:
>> [ 75.480358] Write same(16): 93 08 00 00 00 00 02 ff ff fa 00 7f ff ff
>> 00 00
>> [ 75.480358] end_request: I/O error, dev sdc, sector 50331642
>> [ 75.496374] sd 2:0:0:4: [sdc]
>> [ 75.497202] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
>> [ 75.498077] sd 2:0:0:4: [sdc]
>> [ 75.498948] Sense Key : Aborted Command [current]
>> [ 75.499818] sd 2:0:0:4: [sdc]
>> [ 75.500374] Add. Sense: I/O process terminated
>> [ 75.500374] sd 2:0:0:4: [sdc] CDB:
>> [ 75.500374] Write same(16): 93 08 00 00 00 00 03 7f ff f9 00 7f ff ff
>> 00 00
>> [ 75.500374] end_request: I/O error, dev sdc, sector 58720249
>
> Could you add 'debug ms = 1', 'debug rbd = 20', and 'log file =
> /tmp/librbd.log' to the [client] section of your ceph.conf?
>
> That log should tell us where the error is happening.
Paolo from Redhat / qemu means:
"
The discard request is failing. Please check why with a breakpoint on
rbd_aio_discard_wrapper or scsi_handle_rw_error for example.
"
Greets,
Stefan
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: RBD trim / unmap support?
2012-11-06 22:45 ` Stefan Priebe
@ 2012-11-11 12:08 ` Stefan Priebe
0 siblings, 0 replies; 13+ messages in thread
From: Stefan Priebe @ 2012-11-11 12:08 UTC (permalink / raw)
To: Josh Durgin; +Cc: ceph-devel
Still nobody who can help with the broken unmap support of rbd?
Greets,
Stefan
Am 06.11.2012 23:45, schrieb Stefan Priebe:
> Am 02.11.2012 17:16, schrieb Josh Durgin:
>> On 11/02/2012 02:08 AM, Stefan Priebe - Profihost AG wrote:
>>> Hi Jost,
>>>
>>> i tried scsi-hd with discard_granularity=512 the UNMAP / Trim itself
>>> works fine the space is freed at ceph but at the VM i'm getting:
>>> [ 75.076895] sd 2:0:0:4: [sdc]
>>> [ 75.078353] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
>>> [ 75.079377] sd 2:0:0:4: [sdc]
>>> [ 75.080410] Sense Key : Aborted Command [current]
>>> [ 75.081446] sd 2:0:0:4: [sdc]
>>> [ 75.082459] Add. Sense: I/O process terminated
>>> [ 75.083466] sd 2:0:0:4: [sdc] CDB:
>>> [ 75.084473] Write same(16): 93 08 00 00 00 00 00 00 00 00 00 7f ff ff
>>> 00 00
>>> [ 75.085535] end_request: I/O error, dev sdc, sector 0
>>> [ 75.086567] sd 2:0:0:4: [sdc]
>>> [ 75.087569] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
>>> [ 75.088605] sd 2:0:0:4: [sdc]
>>> [ 75.089610] Sense Key : Aborted Command [current]
>>> [ 75.090630] sd 2:0:0:4: [sdc]
>>> [ 75.091631] Add. Sense: I/O process terminated
>>> [ 75.092621] sd 2:0:0:4: [sdc] CDB:
>>> [ 75.093579] Write same(16): 93 08 00 00 00 00 00 7f ff ff 00 7f ff ff
>>> 00 00
>>> [ 75.094581] end_request: I/O error, dev sdc, sector 8388607
>>> [ 75.095558] sd 2:0:0:4: [sdc]
>>> [ 75.096561] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
>>> [ 75.097553] sd 2:0:0:4: [sdc]
>>> [ 75.098520] Sense Key : Aborted Command [current]
>>> [ 75.099484] sd 2:0:0:4: [sdc]
>>> [ 75.100471] Add. Sense: I/O process terminated
>>> [ 75.101439] sd 2:0:0:4: [sdc] CDB:
>>> [ 75.102432] Write same(16): 93 08 00 00 00 00 00 ff ff fe 00 7f ff ff
>>> 00 00
>>> [ 75.103464] end_request: I/O error, dev sdc, sector 16777214
>>> [ 75.104503] sd 2:0:0:4: [sdc]
>>> [ 75.105507] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
>>> [ 75.106514] sd 2:0:0:4: [sdc]
>>> [ 75.107525] Sense Key : Aborted Command [current]
>>> [ 75.108545] sd 2:0:0:4: [sdc]
>>> [ 75.109525] Add. Sense: I/O process terminated
>>> [ 75.110487] sd 2:0:0:4: [sdc] CDB:
>>> [ 75.111418] Write same(16): 93 08 00 00 00 00 01 7f ff fd 00 7f ff ff
>>> 00 00
>>> [ 75.112419] end_request: I/O error, dev sdc, sector 25165821
>>> [ 75.238627] sd 2:0:0:4: [sdc]
>>> [ 75.239946] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
>>> [ 75.241197] sd 2:0:0:4: [sdc]
>>> [ 75.242097] Sense Key : Aborted Command [current]
>>> [ 75.242627] sd 2:0:0:4: [sdc]
>>> [ 75.242627] Add. Sense: I/O process terminated
>>> [ 75.242627] sd 2:0:0:4: [sdc] CDB:
>>> [ 75.242627] Write same(16): 93 08 00 00 00 00 01 ff ff fc 00 7f ff ff
>>> 00 00
>>> [ 75.242627] end_request: I/O error, dev sdc, sector 33554428
>>> [ 75.447311] sd 2:0:0:4: [sdc]
>>> [ 75.448505] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
>>> [ 75.449392] sd 2:0:0:4: [sdc]
>>> [ 75.450262] Sense Key : Aborted Command [current]
>>> [ 75.451134] sd 2:0:0:4: [sdc]
>>> [ 75.451312] Add. Sense: I/O process terminated
>>> [ 75.451312] sd 2:0:0:4: [sdc] CDB:
>>> [ 75.451312] Write same(16): 93 08 00 00 00 00 02 7f ff fb 00 7f ff ff
>>> 00 00
>>> [ 75.451312] end_request: I/O error, dev sdc, sector 41943035
>>> [ 75.476364] sd 2:0:0:4: [sdc]
>>> [ 75.477252] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
>>> [ 75.478135] sd 2:0:0:4: [sdc]
>>> [ 75.479004] Sense Key : Aborted Command [current]
>>> [ 75.479891] sd 2:0:0:4: [sdc]
>>> [ 75.480358] Add. Sense: I/O process terminated
>>> [ 75.480358] sd 2:0:0:4: [sdc] CDB:
>>> [ 75.480358] Write same(16): 93 08 00 00 00 00 02 ff ff fa 00 7f ff ff
>>> 00 00
>>> [ 75.480358] end_request: I/O error, dev sdc, sector 50331642
>>> [ 75.496374] sd 2:0:0:4: [sdc]
>>> [ 75.497202] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
>>> [ 75.498077] sd 2:0:0:4: [sdc]
>>> [ 75.498948] Sense Key : Aborted Command [current]
>>> [ 75.499818] sd 2:0:0:4: [sdc]
>>> [ 75.500374] Add. Sense: I/O process terminated
>>> [ 75.500374] sd 2:0:0:4: [sdc] CDB:
>>> [ 75.500374] Write same(16): 93 08 00 00 00 00 03 7f ff f9 00 7f ff ff
>>> 00 00
>>> [ 75.500374] end_request: I/O error, dev sdc, sector 58720249
>>
>> Could you add 'debug ms = 1', 'debug rbd = 20', and 'log file =
>> /tmp/librbd.log' to the [client] section of your ceph.conf?
>>
>> That log should tell us where the error is happening.
>
> Paolo from Redhat / qemu means:
>
> "
> The discard request is failing. Please check why with a breakpoint on
> rbd_aio_discard_wrapper or scsi_handle_rw_error for example.
> "
>
> Greets,
> Stefan
^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2012-11-11 12:08 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-11-01 23:33 RBD trim / unmap support? Stefan Priebe
2012-11-01 23:36 ` Josh Durgin
2012-11-02 8:20 ` Stefan Priebe - Profihost AG
2012-11-02 8:21 ` Dietmar Maurer
2012-11-02 9:08 ` Stefan Priebe - Profihost AG
2012-11-02 14:18 ` Stefan Priebe - Profihost AG
2012-11-02 14:18 ` [Qemu-devel] " Stefan Priebe - Profihost AG
2012-11-02 16:16 ` Josh Durgin
2012-11-02 20:13 ` Stefan Priebe
2012-11-03 18:47 ` Stefan Priebe
2012-11-06 8:09 ` Stefan Priebe - Profihost AG
2012-11-06 22:45 ` Stefan Priebe
2012-11-11 12:08 ` Stefan Priebe
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.