* [dm-crypt] trouble with dm-crypt and NFS: server hangs
@ 2014-03-20 14:48 Stephen Cousins
2014-03-20 19:16 ` Milan Broz
0 siblings, 1 reply; 7+ messages in thread
From: Stephen Cousins @ 2014-03-20 14:48 UTC (permalink / raw)
To: dm-crypt
[-- Attachment #1: Type: text/plain, Size: 1393 bytes --]
I am testing dm-crypt with a system that serves data over NFS. I'm finding
that when serving the data over NFS4 it will go for a little while and then
it will just stop copying and networking on the NFS server will stall out
so that other connections to other parts of the storage (a non-dm-crypt
volume) will become very laggy. Soon after canceling the copy the other
connections resume back to normal.
I have tried this on two different servers, one with a single disk with ZFS
and another with a ZFS raidz2 pool and they both do the same thing. CPU
doesn't seem to be an issue. Doing local copies is not a problem. That is,
copying to/from these volumes made up of dm-crypted disks from other disks
on the server have no problems.
One system (the one with the RAID pool) is CentOS 6.4 and the other one is
CentOS 6.5.
I'm curious if anyone else has run into this before and if so, what can be
done to make it work? I'm going to test to see if taking ZFS out of the
equation helps. I haven't run into this with ZFS with disks without
dm-crypt.
Thanks,
Steve
--
________________________________________________________________
Steve Cousins Supercomputer Engineer/Administrator
Advanced Computing Group University of Maine System
244 Neville Hall (UMS Data Center) (207) 561-3574
Orono ME 04469 steve.cousins at maine.edu
[-- Attachment #2: Type: text/html, Size: 3056 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [dm-crypt] trouble with dm-crypt and NFS: server hangs
@ 2014-03-20 19:16 ` Milan Broz
0 siblings, 0 replies; 7+ messages in thread
From: Milan Broz @ 2014-03-20 19:16 UTC (permalink / raw)
To: Stephen Cousins, dm-crypt; +Cc: device-mapper development
On 03/20/2014 03:48 PM, Stephen Cousins wrote:
> I am testing dm-crypt with a system that serves data over NFS. I'm
> finding that when serving the data over NFS4 it will go for a little
> while and then it will just stop copying and networking on the NFS
> server will stall out so that other connections to other parts of the
> storage (a non-dm-crypt volume) will become very laggy. Soon after
> canceling the copy the other connections resume back to normal.
>
> I have tried this on two different servers, one with a single disk
> with ZFS and another with a ZFS raidz2 pool and they both do the same
> thing. CPU doesn't seem to be an issue. Doing local copies is not a
> problem. That is, copying to/from these volumes made up of dm-crypted
> disks from other disks on the server have no problems.
>
> One system (the one with the RAID pool) is CentOS 6.4 and the other
> one is CentOS 6.5.
>
> I'm curious if anyone else has run into this before and if so, what
> can be done to make it work? I'm going to test to see if taking ZFS
> out of the equation helps. I haven't run into this with ZFS with
> disks without dm-crypt.
Hi,
unfortunately the dmcrypt code in Centos 6.x is not what is upstream
kernel (mainly because of per cpu processing changes upstream).
(It would be interesting if it is reproducible with upstream kernel,
but I guess it is not easy to test...)
I am adding cc to dm-devel list, maybe someone there could have some idea
how to help here.
Milan
>
> Thanks,
>
> Steve
>
>
>
> -- ________________________________________________________________
> Steve Cousins Supercomputer Engineer/Administrator
> Advanced Computing Group University of Maine System 244
> Neville Hall (UMS Data Center) (207) 561-3574
> <tel:%28207%29%20561-3574> Orono ME 04469
> steve.cousins at maine.edu <http://maine.edu>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: trouble with dm-crypt and NFS: server hangs
@ 2014-03-20 19:16 ` Milan Broz
0 siblings, 0 replies; 7+ messages in thread
From: Milan Broz @ 2014-03-20 19:16 UTC (permalink / raw)
To: Stephen Cousins, dm-crypt-4q3lyFh4P1g; +Cc: device-mapper development
On 03/20/2014 03:48 PM, Stephen Cousins wrote:
> I am testing dm-crypt with a system that serves data over NFS. I'm
> finding that when serving the data over NFS4 it will go for a little
> while and then it will just stop copying and networking on the NFS
> server will stall out so that other connections to other parts of the
> storage (a non-dm-crypt volume) will become very laggy. Soon after
> canceling the copy the other connections resume back to normal.
>
> I have tried this on two different servers, one with a single disk
> with ZFS and another with a ZFS raidz2 pool and they both do the same
> thing. CPU doesn't seem to be an issue. Doing local copies is not a
> problem. That is, copying to/from these volumes made up of dm-crypted
> disks from other disks on the server have no problems.
>
> One system (the one with the RAID pool) is CentOS 6.4 and the other
> one is CentOS 6.5.
>
> I'm curious if anyone else has run into this before and if so, what
> can be done to make it work? I'm going to test to see if taking ZFS
> out of the equation helps. I haven't run into this with ZFS with
> disks without dm-crypt.
Hi,
unfortunately the dmcrypt code in Centos 6.x is not what is upstream
kernel (mainly because of per cpu processing changes upstream).
(It would be interesting if it is reproducible with upstream kernel,
but I guess it is not easy to test...)
I am adding cc to dm-devel list, maybe someone there could have some idea
how to help here.
Milan
>
> Thanks,
>
> Steve
>
>
>
> -- ________________________________________________________________
> Steve Cousins Supercomputer Engineer/Administrator
> Advanced Computing Group University of Maine System 244
> Neville Hall (UMS Data Center) (207) 561-3574
> <tel:%28207%29%20561-3574> Orono ME 04469
> steve.cousins at maine.edu <http://maine.edu>
_______________________________________________
dm-crypt mailing list
dm-crypt-4q3lyFh4P1g@public.gmane.org
http://www.saout.de/mailman/listinfo/dm-crypt
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [dm-crypt] trouble with dm-crypt and NFS: server hangs
@ 2014-03-20 22:12 ` Stephen Cousins
0 siblings, 0 replies; 7+ messages in thread
From: Stephen Cousins @ 2014-03-20 22:12 UTC (permalink / raw)
To: Milan Broz; +Cc: dm-crypt, device-mapper development
[-- Attachment #1: Type: text/plain, Size: 3016 bytes --]
I'm getting mixed results but it is looking like ZFS have something to do
with the problem afterall.
I tried it with dm-crypt -> XFS -> NFS and I'm not seeing the same lag
issue. I say mixed results mainly because I've been using rsync to do the
testing and for some reason rsync has trouble with this setup. It works but
really slowly. Like around 3 MB/sec over a Gigabit link. If I use "cp -pR"
then it goes at 60 MB/sec. I use rsync all the time and I've never seen
this before so it took a while for me to realize that rsync was acting up.
Anyone run into this before?
I'll bring this to the ZFSonLinux people and see what they say.
Thanks,
Steve
On Thu, Mar 20, 2014 at 3:16 PM, Milan Broz <gmazyland@gmail.com> wrote:
> On 03/20/2014 03:48 PM, Stephen Cousins wrote:
> > I am testing dm-crypt with a system that serves data over NFS. I'm
> > finding that when serving the data over NFS4 it will go for a little
> > while and then it will just stop copying and networking on the NFS
> > server will stall out so that other connections to other parts of the
> > storage (a non-dm-crypt volume) will become very laggy. Soon after
> > canceling the copy the other connections resume back to normal.
> >
> > I have tried this on two different servers, one with a single disk
> > with ZFS and another with a ZFS raidz2 pool and they both do the same
> > thing. CPU doesn't seem to be an issue. Doing local copies is not a
> > problem. That is, copying to/from these volumes made up of dm-crypted
> > disks from other disks on the server have no problems.
> >
> > One system (the one with the RAID pool) is CentOS 6.4 and the other
> > one is CentOS 6.5.
> >
> > I'm curious if anyone else has run into this before and if so, what
> > can be done to make it work? I'm going to test to see if taking ZFS
> > out of the equation helps. I haven't run into this with ZFS with
> > disks without dm-crypt.
>
> Hi,
>
> unfortunately the dmcrypt code in Centos 6.x is not what is upstream
> kernel (mainly because of per cpu processing changes upstream).
> (It would be interesting if it is reproducible with upstream kernel,
> but I guess it is not easy to test...)
>
> I am adding cc to dm-devel list, maybe someone there could have some idea
> how to help here.
>
> Milan
>
> >
> > Thanks,
> >
> > Steve
> >
> >
> >
> > -- ________________________________________________________________
> > Steve Cousins Supercomputer Engineer/Administrator
> > Advanced Computing Group University of Maine System 244
> > Neville Hall (UMS Data Center) (207) 561-3574
> > <tel:%28207%29%20561-3574> Orono ME 04469
> > steve.cousins at maine.edu <http://maine.edu>
>
--
________________________________________________________________
Steve Cousins Supercomputer Engineer/Administrator
Advanced Computing Group University of Maine System
244 Neville Hall (UMS Data Center) (207) 561-3574
Orono ME 04469 steve.cousins at maine.edu
[-- Attachment #2: Type: text/html, Size: 5062 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: trouble with dm-crypt and NFS: server hangs
@ 2014-03-20 22:12 ` Stephen Cousins
0 siblings, 0 replies; 7+ messages in thread
From: Stephen Cousins @ 2014-03-20 22:12 UTC (permalink / raw)
To: Milan Broz; +Cc: dm-crypt-4q3lyFh4P1g, device-mapper development
[-- Attachment #1.1: Type: text/plain, Size: 3046 bytes --]
I'm getting mixed results but it is looking like ZFS have something to do
with the problem afterall.
I tried it with dm-crypt -> XFS -> NFS and I'm not seeing the same lag
issue. I say mixed results mainly because I've been using rsync to do the
testing and for some reason rsync has trouble with this setup. It works but
really slowly. Like around 3 MB/sec over a Gigabit link. If I use "cp -pR"
then it goes at 60 MB/sec. I use rsync all the time and I've never seen
this before so it took a while for me to realize that rsync was acting up.
Anyone run into this before?
I'll bring this to the ZFSonLinux people and see what they say.
Thanks,
Steve
On Thu, Mar 20, 2014 at 3:16 PM, Milan Broz <gmazyland-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> On 03/20/2014 03:48 PM, Stephen Cousins wrote:
> > I am testing dm-crypt with a system that serves data over NFS. I'm
> > finding that when serving the data over NFS4 it will go for a little
> > while and then it will just stop copying and networking on the NFS
> > server will stall out so that other connections to other parts of the
> > storage (a non-dm-crypt volume) will become very laggy. Soon after
> > canceling the copy the other connections resume back to normal.
> >
> > I have tried this on two different servers, one with a single disk
> > with ZFS and another with a ZFS raidz2 pool and they both do the same
> > thing. CPU doesn't seem to be an issue. Doing local copies is not a
> > problem. That is, copying to/from these volumes made up of dm-crypted
> > disks from other disks on the server have no problems.
> >
> > One system (the one with the RAID pool) is CentOS 6.4 and the other
> > one is CentOS 6.5.
> >
> > I'm curious if anyone else has run into this before and if so, what
> > can be done to make it work? I'm going to test to see if taking ZFS
> > out of the equation helps. I haven't run into this with ZFS with
> > disks without dm-crypt.
>
> Hi,
>
> unfortunately the dmcrypt code in Centos 6.x is not what is upstream
> kernel (mainly because of per cpu processing changes upstream).
> (It would be interesting if it is reproducible with upstream kernel,
> but I guess it is not easy to test...)
>
> I am adding cc to dm-devel list, maybe someone there could have some idea
> how to help here.
>
> Milan
>
> >
> > Thanks,
> >
> > Steve
> >
> >
> >
> > -- ________________________________________________________________
> > Steve Cousins Supercomputer Engineer/Administrator
> > Advanced Computing Group University of Maine System 244
> > Neville Hall (UMS Data Center) (207) 561-3574
> > <tel:%28207%29%20561-3574> Orono ME 04469
> > steve.cousins at maine.edu <http://maine.edu>
>
--
________________________________________________________________
Steve Cousins Supercomputer Engineer/Administrator
Advanced Computing Group University of Maine System
244 Neville Hall (UMS Data Center) (207) 561-3574
Orono ME 04469 steve.cousins at maine.edu
[-- Attachment #1.2: Type: text/html, Size: 5122 bytes --]
[-- Attachment #2: Type: text/plain, Size: 154 bytes --]
_______________________________________________
dm-crypt mailing list
dm-crypt-4q3lyFh4P1g@public.gmane.org
http://www.saout.de/mailman/listinfo/dm-crypt
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [dm-crypt] trouble with dm-crypt and NFS: server hangs
2014-03-20 22:12 ` Stephen Cousins
(?)
@ 2014-03-27 17:56 ` Stephen Cousins
2014-03-27 20:28 ` Arno Wagner
-1 siblings, 1 reply; 7+ messages in thread
From: Stephen Cousins @ 2014-03-27 17:56 UTC (permalink / raw)
To: Milan Broz; +Cc: dm-crypt
[-- Attachment #1: Type: text/plain, Size: 3708 bytes --]
I'm following up on this to let you know that it has been identified as a
ZFS sync issue that has been addressed recently. No need to contact
dm-devel.
Thanks for your help.
Steve
On Thu, Mar 20, 2014 at 6:12 PM, Stephen Cousins <steve.cousins@maine.edu>wrote:
> I'm getting mixed results but it is looking like ZFS have something to do
> with the problem afterall.
>
> I tried it with dm-crypt -> XFS -> NFS and I'm not seeing the same lag
> issue. I say mixed results mainly because I've been using rsync to do the
> testing and for some reason rsync has trouble with this setup. It works but
> really slowly. Like around 3 MB/sec over a Gigabit link. If I use "cp -pR"
> then it goes at 60 MB/sec. I use rsync all the time and I've never seen
> this before so it took a while for me to realize that rsync was acting up.
> Anyone run into this before?
>
> I'll bring this to the ZFSonLinux people and see what they say.
>
> Thanks,
>
> Steve
>
>
> On Thu, Mar 20, 2014 at 3:16 PM, Milan Broz <gmazyland@gmail.com> wrote:
>
>> On 03/20/2014 03:48 PM, Stephen Cousins wrote:
>> > I am testing dm-crypt with a system that serves data over NFS. I'm
>> > finding that when serving the data over NFS4 it will go for a little
>> > while and then it will just stop copying and networking on the NFS
>> > server will stall out so that other connections to other parts of the
>> > storage (a non-dm-crypt volume) will become very laggy. Soon after
>> > canceling the copy the other connections resume back to normal.
>> >
>> > I have tried this on two different servers, one with a single disk
>> > with ZFS and another with a ZFS raidz2 pool and they both do the same
>> > thing. CPU doesn't seem to be an issue. Doing local copies is not a
>> > problem. That is, copying to/from these volumes made up of dm-crypted
>> > disks from other disks on the server have no problems.
>> >
>> > One system (the one with the RAID pool) is CentOS 6.4 and the other
>> > one is CentOS 6.5.
>> >
>> > I'm curious if anyone else has run into this before and if so, what
>> > can be done to make it work? I'm going to test to see if taking ZFS
>> > out of the equation helps. I haven't run into this with ZFS with
>> > disks without dm-crypt.
>>
>> Hi,
>>
>> unfortunately the dmcrypt code in Centos 6.x is not what is upstream
>> kernel (mainly because of per cpu processing changes upstream).
>> (It would be interesting if it is reproducible with upstream kernel,
>> but I guess it is not easy to test...)
>>
>> I am adding cc to dm-devel list, maybe someone there could have some idea
>> how to help here.
>>
>> Milan
>>
>> >
>> > Thanks,
>> >
>> > Steve
>> >
>> >
>> >
>> > -- ________________________________________________________________
>> > Steve Cousins Supercomputer Engineer/Administrator
>> > Advanced Computing Group University of Maine System 244
>> > Neville Hall (UMS Data Center) (207) 561-3574
>> > <tel:%28207%29%20561-3574> Orono ME 04469
>> > steve.cousins at maine.edu <http://maine.edu>
>>
>
>
>
> --
> ________________________________________________________________
> Steve Cousins Supercomputer Engineer/Administrator
> Advanced Computing Group University of Maine System
> 244 Neville Hall (UMS Data Center) (207) 561-3574
> Orono ME 04469 steve.cousins at maine.edu
>
>
--
________________________________________________________________
Steve Cousins Supercomputer Engineer/Administrator
Advanced Computing Group University of Maine System
244 Neville Hall (UMS Data Center) (207) 561-3574
Orono ME 04469 steve.cousins at maine.edu
[-- Attachment #2: Type: text/html, Size: 7089 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [dm-crypt] trouble with dm-crypt and NFS: server hangs
2014-03-27 17:56 ` [dm-crypt] " Stephen Cousins
@ 2014-03-27 20:28 ` Arno Wagner
0 siblings, 0 replies; 7+ messages in thread
From: Arno Wagner @ 2014-03-27 20:28 UTC (permalink / raw)
To: dm-crypt
Thank you!
Arno
On Thu, Mar 27, 2014 at 18:56:06 CET, Stephen Cousins wrote:
> I'm following up on this to let you know that it has been identified as a
> ZFS sync issue that has been addressed recently. No need to contact
> dm-devel.
>
> Thanks for your help.
>
> Steve
>
>
> On Thu, Mar 20, 2014 at 6:12 PM, Stephen Cousins <steve.cousins@maine.edu>wrote:
>
> > I'm getting mixed results but it is looking like ZFS have something to do
> > with the problem afterall.
> >
> > I tried it with dm-crypt -> XFS -> NFS and I'm not seeing the same lag
> > issue. I say mixed results mainly because I've been using rsync to do the
> > testing and for some reason rsync has trouble with this setup. It works but
> > really slowly. Like around 3 MB/sec over a Gigabit link. If I use "cp -pR"
> > then it goes at 60 MB/sec. I use rsync all the time and I've never seen
> > this before so it took a while for me to realize that rsync was acting up.
> > Anyone run into this before?
> >
> > I'll bring this to the ZFSonLinux people and see what they say.
> >
> > Thanks,
> >
> > Steve
> >
> >
> > On Thu, Mar 20, 2014 at 3:16 PM, Milan Broz <gmazyland@gmail.com> wrote:
> >
> >> On 03/20/2014 03:48 PM, Stephen Cousins wrote:
> >> > I am testing dm-crypt with a system that serves data over NFS. I'm
> >> > finding that when serving the data over NFS4 it will go for a little
> >> > while and then it will just stop copying and networking on the NFS
> >> > server will stall out so that other connections to other parts of the
> >> > storage (a non-dm-crypt volume) will become very laggy. Soon after
> >> > canceling the copy the other connections resume back to normal.
> >> >
> >> > I have tried this on two different servers, one with a single disk
> >> > with ZFS and another with a ZFS raidz2 pool and they both do the same
> >> > thing. CPU doesn't seem to be an issue. Doing local copies is not a
> >> > problem. That is, copying to/from these volumes made up of dm-crypted
> >> > disks from other disks on the server have no problems.
> >> >
> >> > One system (the one with the RAID pool) is CentOS 6.4 and the other
> >> > one is CentOS 6.5.
> >> >
> >> > I'm curious if anyone else has run into this before and if so, what
> >> > can be done to make it work? I'm going to test to see if taking ZFS
> >> > out of the equation helps. I haven't run into this with ZFS with
> >> > disks without dm-crypt.
> >>
> >> Hi,
> >>
> >> unfortunately the dmcrypt code in Centos 6.x is not what is upstream
> >> kernel (mainly because of per cpu processing changes upstream).
> >> (It would be interesting if it is reproducible with upstream kernel,
> >> but I guess it is not easy to test...)
> >>
> >> I am adding cc to dm-devel list, maybe someone there could have some idea
> >> how to help here.
> >>
> >> Milan
> >>
> >> >
> >> > Thanks,
> >> >
> >> > Steve
> >> >
> >> >
> >> >
> >> > -- ________________________________________________________________
> >> > Steve Cousins Supercomputer Engineer/Administrator
> >> > Advanced Computing Group University of Maine System 244
> >> > Neville Hall (UMS Data Center) (207) 561-3574
> >> > <tel:%28207%29%20561-3574> Orono ME 04469
> >> > steve.cousins at maine.edu <http://maine.edu>
> >>
> >
> >
> >
> > --
> > ________________________________________________________________
> > Steve Cousins Supercomputer Engineer/Administrator
> > Advanced Computing Group University of Maine System
> > 244 Neville Hall (UMS Data Center) (207) 561-3574
> > Orono ME 04469 steve.cousins at maine.edu
> >
> >
>
>
> --
> ________________________________________________________________
> Steve Cousins Supercomputer Engineer/Administrator
> Advanced Computing Group University of Maine System
> 244 Neville Hall (UMS Data Center) (207) 561-3574
> Orono ME 04469 steve.cousins at maine.edu
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt
--
Arno Wagner, Dr. sc. techn., Dipl. Inform., Email: arno@wagner.name
GnuPG: ID: CB5D9718 FP: 12D6 C03B 1B30 33BB 13CF B774 E35C 5FA1 CB5D 9718
----
A good decision is based on knowledge and not on numbers. - Plato
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2014-03-27 20:28 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-03-20 14:48 [dm-crypt] trouble with dm-crypt and NFS: server hangs Stephen Cousins
2014-03-20 19:16 ` Milan Broz
2014-03-20 19:16 ` Milan Broz
2014-03-20 22:12 ` [dm-crypt] " Stephen Cousins
2014-03-20 22:12 ` Stephen Cousins
2014-03-27 17:56 ` [dm-crypt] " Stephen Cousins
2014-03-27 20:28 ` Arno Wagner
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.