Fwd: XFS Memory allocation deadlock in kmem_alloc

Fwd: XFS Memory allocation deadlock in kmem_alloc

[Andrew Carr]

Hello,

This list has recommended enabling stack traces to determine the root
cause of issues with XFS deadlocks occurring in Centos 7.7
(3.10.0-1062).

Based on what was recommended by Eric Sandeen, we have tried updating
the following files to generate XFS stack traces:

# echo 11 > /proc/sys/fs/xfs/error_level

And

# echo 3 > /proc/sys/fs/xfs/error_level

But no stack traces are printed to dmesg.  I was thinking of
re-compiling the kernel with debug flags enabled.  Do you think this
is necessary?

Thanks so much for your time and keep up the good work!

Sincerely,
--
Andrew Carr | Enterprise Architect
Rogue Wave Software, Inc.
Innovate with Confidence
P 720.295.8044
www.roguewave.com | andrew.carr@roguewave.com


--
With Regards,
Andrew Carr

Re: Fwd: XFS Memory allocation deadlock in kmem_alloc

[Eric Sandeen]

On 11/15/19 1:11 PM, Andrew Carr wrote:
> Hello,
> 
> This list has recommended enabling stack traces to determine the root
> cause of issues with XFS deadlocks occurring in Centos 7.7
> (3.10.0-1062).
> 
> Based on what was recommended by Eric Sandeen, we have tried updating
> the following files to generate XFS stack traces:
> 
> # echo 11 > /proc/sys/fs/xfs/error_level
> 
> And
> 
> # echo 3 > /proc/sys/fs/xfs/error_level
> 
> But no stack traces are printed to dmesg.  I was thinking of
> re-compiling the kernel with debug flags enabled.  Do you think this
> is necessary?
> 
> Thanks so much for your time and keep up the good work!

I've looked over the way xfs_err() gets defined, and I cannot see how
we can call xfs_err with error_level == 11 and not get a stack trace.

Maybe other eyes can spot something...

-Eric

Re: Fwd: XFS Memory allocation deadlock in kmem_alloc

[Dave Chinner]

On Fri, Nov 15, 2019 at 01:52:57PM -0600, Eric Sandeen wrote:
> On 11/15/19 1:11 PM, Andrew Carr wrote:
> > Hello,
> > 
> > This list has recommended enabling stack traces to determine the root
> > cause of issues with XFS deadlocks occurring in Centos 7.7
> > (3.10.0-1062).
> > 
> > Based on what was recommended by Eric Sandeen, we have tried updating
> > the following files to generate XFS stack traces:
> > 
> > # echo 11 > /proc/sys/fs/xfs/error_level
> > 
> > And
> > 
> > # echo 3 > /proc/sys/fs/xfs/error_level
> > 
> > But no stack traces are printed to dmesg.  I was thinking of
> > re-compiling the kernel with debug flags enabled.  Do you think this
> > is necessary?

dmesg -n 7 will remove all filters on the console/dmesg output - if
you've utrned this down in the past you may not be seeing messages
of the error level XFS is using...

Did you check syslog - that should have all the unfiltered messages
in it...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: Fwd: XFS Memory allocation deadlock in kmem_alloc

[Andrew Carr]

Thanks Dave,
Checking now.

On Fri, Nov 15, 2019 at 6:43 PM Dave Chinner <david@fromorbit.com> wrote:
>
> On Fri, Nov 15, 2019 at 01:52:57PM -0600, Eric Sandeen wrote:
> > On 11/15/19 1:11 PM, Andrew Carr wrote:
> > > Hello,
> > >
> > > This list has recommended enabling stack traces to determine the root
> > > cause of issues with XFS deadlocks occurring in Centos 7.7
> > > (3.10.0-1062).
> > >
> > > Based on what was recommended by Eric Sandeen, we have tried updating
> > > the following files to generate XFS stack traces:
> > >
> > > # echo 11 > /proc/sys/fs/xfs/error_level
> > >
> > > And
> > >
> > > # echo 3 > /proc/sys/fs/xfs/error_level
> > >
> > > But no stack traces are printed to dmesg.  I was thinking of
> > > re-compiling the kernel with debug flags enabled.  Do you think this
> > > is necessary?
>
> dmesg -n 7 will remove all filters on the console/dmesg output - if
> you've utrned this down in the past you may not be seeing messages
> of the error level XFS is using...
>
> Did you check syslog - that should have all the unfiltered messages
> in it...
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com



-- 
With Regards,
Andrew Carr

e. andrewlanecarr@gmail.com
w. andrew.carr@openlogic.com
c. 4239489206
a. P.O. Box 1231, Greeneville, TN, 37744

Re: Fwd: XFS Memory allocation deadlock in kmem_alloc

[Andrew Carr]

Dave / Eric / Others,

Syslog: https://pastebin.com/QYQYpPFY

Dmesg: https://pastebin.com/MdBCPmp9

-Andrew Carr


On Sat, Nov 16, 2019 at 11:19 AM Andrew Carr <andrewlanecarr@gmail.com> wrote:
>
> Thanks Dave,
> Checking now.
>
> On Fri, Nov 15, 2019 at 6:43 PM Dave Chinner <david@fromorbit.com> wrote:
> >
> > On Fri, Nov 15, 2019 at 01:52:57PM -0600, Eric Sandeen wrote:
> > > On 11/15/19 1:11 PM, Andrew Carr wrote:
> > > > Hello,
> > > >
> > > > This list has recommended enabling stack traces to determine the root
> > > > cause of issues with XFS deadlocks occurring in Centos 7.7
> > > > (3.10.0-1062).
> > > >
> > > > Based on what was recommended by Eric Sandeen, we have tried updating
> > > > the following files to generate XFS stack traces:
> > > >
> > > > # echo 11 > /proc/sys/fs/xfs/error_level
> > > >
> > > > And
> > > >
> > > > # echo 3 > /proc/sys/fs/xfs/error_level
> > > >
> > > > But no stack traces are printed to dmesg.  I was thinking of
> > > > re-compiling the kernel with debug flags enabled.  Do you think this
> > > > is necessary?
> >
> > dmesg -n 7 will remove all filters on the console/dmesg output - if
> > you've utrned this down in the past you may not be seeing messages
> > of the error level XFS is using...
> >
> > Did you check syslog - that should have all the unfiltered messages
> > in it...
> >
> > Cheers,
> >
> > Dave.
> > --
> > Dave Chinner
> > david@fromorbit.com
>
>
>
> --
> With Regards,
> Andrew Carr
>
> e. andrewlanecarr@gmail.com
> w. andrew.carr@openlogic.com
> c. 4239489206
> a. P.O. Box 1231, Greeneville, TN, 37744



-- 
With Regards,
Andrew Carr

e. andrewlanecarr@gmail.com
w. andrew.carr@openlogic.com
c. 4239489206
a. P.O. Box 1231, Greeneville, TN, 37744

Re: Fwd: XFS Memory allocation deadlock in kmem_alloc

[Dave Chinner]

On Tue, Nov 19, 2019 at 10:49:56AM -0500, Andrew Carr wrote:
> Dave / Eric / Others,
> 
> Syslog: https://pastebin.com/QYQYpPFY
> 
> Dmesg: https://pastebin.com/MdBCPmp9

which shows no stack traces, again.



Anyway, you've twiddled mkfs knobs on these filesystems, and that
is the likely cause of the issue: the filesystem is using 64k
directory blocks - the allocation size is larger than 64kB:

[Sun Nov 17 21:40:05 2019] XFS: nginx(31293) possible memory allocation deadlock size 65728 in kmem_alloc (mode:0x250)

Upstream fixed this some time ago:

$ ▶ gl -n 1 -p cb0a8d23024e
commit cb0a8d23024e7bd234dea4d0fc5c4902a8dda766
Author: Dave Chinner <dchinner@redhat.com>
Date:   Tue Mar 6 17:03:28 2018 -0800

    xfs: fall back to vmalloc when allocation log vector buffers
    
    When using large directory blocks, we regularly see memory
    allocations of >64k being made for the shadow log vector buffer.
    When we are under memory pressure, kmalloc() may not be able to find
    contiguous memory chunks large enough to satisfy these allocations
    easily, and if memory is fragmented we can potentially stall here.
    
    TO avoid this problem, switch the log vector buffer allocation to
    use kmem_alloc_large(). This will allow failed allocations to fall
    back to vmalloc and so remove the dependency on large contiguous
    regions of memory being available. This should prevent slowdowns
    and potential stalls when memory is low and/or fragmented.
    
    Signed-Off-By: Dave Chinner <dchinner@redhat.com>
    Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>
    Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>


Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: Fwd: XFS Memory allocation deadlock in kmem_alloc

[Andrew Carr]

Genius Dave, Thanks so much!

On Tue, Nov 19, 2019 at 3:21 PM Dave Chinner <david@fromorbit.com> wrote:
>
> On Tue, Nov 19, 2019 at 10:49:56AM -0500, Andrew Carr wrote:
> > Dave / Eric / Others,
> >
> > Syslog: https://pastebin.com/QYQYpPFY
> >
> > Dmesg: https://pastebin.com/MdBCPmp9
>
> which shows no stack traces, again.
>
>
>
> Anyway, you've twiddled mkfs knobs on these filesystems, and that
> is the likely cause of the issue: the filesystem is using 64k
> directory blocks - the allocation size is larger than 64kB:
>
> [Sun Nov 17 21:40:05 2019] XFS: nginx(31293) possible memory allocation deadlock size 65728 in kmem_alloc (mode:0x250)
>
> Upstream fixed this some time ago:
>
> $ ▶ gl -n 1 -p cb0a8d23024e
> commit cb0a8d23024e7bd234dea4d0fc5c4902a8dda766
> Author: Dave Chinner <dchinner@redhat.com>
> Date:   Tue Mar 6 17:03:28 2018 -0800
>
>     xfs: fall back to vmalloc when allocation log vector buffers
>
>     When using large directory blocks, we regularly see memory
>     allocations of >64k being made for the shadow log vector buffer.
>     When we are under memory pressure, kmalloc() may not be able to find
>     contiguous memory chunks large enough to satisfy these allocations
>     easily, and if memory is fragmented we can potentially stall here.
>
>     TO avoid this problem, switch the log vector buffer allocation to
>     use kmem_alloc_large(). This will allow failed allocations to fall
>     back to vmalloc and so remove the dependency on large contiguous
>     regions of memory being available. This should prevent slowdowns
>     and potential stalls when memory is low and/or fragmented.
>
>     Signed-Off-By: Dave Chinner <dchinner@redhat.com>
>     Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>
>     Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
>
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com



-- 
With Regards,
Andrew Carr

e. andrewlanecarr@gmail.com
w. andrew.carr@openlogic.com
c. 4239489206
a. P.O. Box 1231, Greeneville, TN, 37744

Re: Fwd: XFS Memory allocation deadlock in kmem_alloc

[Andrew Carr]

Hi Dave  / Others,

It appears upgrading to 4.17+ has indeed fixed the deadlock issue, or
at least no deadlocks are occurring now.

There are segfaults in xfs_db appearing now though.  I am attempting
to get the full syslog, here is an example.... thoughts?

[Thu Nov 21 10:43:20 2019] xfs_db[13076]: segfault at 12ff6001 ip
0000000000407922 sp 00007ffe1a27b2e0 error 4 in xfs_db[400000+8a000]
[Thu Nov 21 10:43:20 2019] Code: 89 cc 55 48 89 d5 53 48 89 f3 48 83
ec 48 0f b6 57 01 44 0f b6 4f 02 64 48 8b 04 25 28 00 00 00 48 89 44
24 38 31 c0 0f b6 07 <44> 0f b6 57 0d 48 8d 74 24 10 c1 e2 10 41 c1 e1
08 c1 e0 18 41 c1

Thanks so much in advance!
Andrew

On Wed, Nov 20, 2019 at 10:43 AM Andrew Carr <andrewlanecarr@gmail.com> wrote:
>
> Genius Dave, Thanks so much!
>
> On Tue, Nov 19, 2019 at 3:21 PM Dave Chinner <david@fromorbit.com> wrote:
> >
> > On Tue, Nov 19, 2019 at 10:49:56AM -0500, Andrew Carr wrote:
> > > Dave / Eric / Others,
> > >
> > > Syslog: https://pastebin.com/QYQYpPFY
> > >
> > > Dmesg: https://pastebin.com/MdBCPmp9
> >
> > which shows no stack traces, again.
> >
> >
> >
> > Anyway, you've twiddled mkfs knobs on these filesystems, and that
> > is the likely cause of the issue: the filesystem is using 64k
> > directory blocks - the allocation size is larger than 64kB:
> >
> > [Sun Nov 17 21:40:05 2019] XFS: nginx(31293) possible memory allocation deadlock size 65728 in kmem_alloc (mode:0x250)
> >
> > Upstream fixed this some time ago:
> >
> > $ ▶ gl -n 1 -p cb0a8d23024e
> > commit cb0a8d23024e7bd234dea4d0fc5c4902a8dda766
> > Author: Dave Chinner <dchinner@redhat.com>
> > Date:   Tue Mar 6 17:03:28 2018 -0800
> >
> >     xfs: fall back to vmalloc when allocation log vector buffers
> >
> >     When using large directory blocks, we regularly see memory
> >     allocations of >64k being made for the shadow log vector buffer.
> >     When we are under memory pressure, kmalloc() may not be able to find
> >     contiguous memory chunks large enough to satisfy these allocations
> >     easily, and if memory is fragmented we can potentially stall here.
> >
> >     TO avoid this problem, switch the log vector buffer allocation to
> >     use kmem_alloc_large(). This will allow failed allocations to fall
> >     back to vmalloc and so remove the dependency on large contiguous
> >     regions of memory being available. This should prevent slowdowns
> >     and potential stalls when memory is low and/or fragmented.
> >
> >     Signed-Off-By: Dave Chinner <dchinner@redhat.com>
> >     Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>
> >     Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> >
> >
> > Cheers,
> >
> > Dave.
> > --
> > Dave Chinner
> > david@fromorbit.com
>
>
>
> --
> With Regards,
> Andrew Carr
>
> e. andrewlanecarr@gmail.com
> w. andrew.carr@openlogic.com
> c. 4239489206
> a. P.O. Box 1231, Greeneville, TN, 37744



-- 
With Regards,
Andrew Carr

e. andrewlanecarr@gmail.com
w. andrew.carr@openlogic.com
c. 4239489206
a. P.O. Box 1231, Greeneville, TN, 37744

Re: Fwd: XFS Memory allocation deadlock in kmem_alloc

[Darrick J. Wong]

On Fri, Nov 22, 2019 at 09:08:26AM -0500, Andrew Carr wrote:
> Hi Dave  / Others,
> 
> It appears upgrading to 4.17+ has indeed fixed the deadlock issue, or
> at least no deadlocks are occurring now.
> 
> There are segfaults in xfs_db appearing now though.  I am attempting
> to get the full syslog, here is an example.... thoughts?
> 
> [Thu Nov 21 10:43:20 2019] xfs_db[13076]: segfault at 12ff6001 ip
> 0000000000407922 sp 00007ffe1a27b2e0 error 4 in xfs_db[400000+8a000]
> [Thu Nov 21 10:43:20 2019] Code: 89 cc 55 48 89 d5 53 48 89 f3 48 83
> ec 48 0f b6 57 01 44 0f b6 4f 02 64 48 8b 04 25 28 00 00 00 48 89 44
> 24 38 31 c0 0f b6 07 <44> 0f b6 57 0d 48 8d 74 24 10 c1 e2 10 41 c1 e1
> 08 c1 e0 18 41 c1

Actual coredumps of the crashed xfs_db would help.

--D

> Thanks so much in advance!
> Andrew
> 
> On Wed, Nov 20, 2019 at 10:43 AM Andrew Carr <andrewlanecarr@gmail.com> wrote:
> >
> > Genius Dave, Thanks so much!
> >
> > On Tue, Nov 19, 2019 at 3:21 PM Dave Chinner <david@fromorbit.com> wrote:
> > >
> > > On Tue, Nov 19, 2019 at 10:49:56AM -0500, Andrew Carr wrote:
> > > > Dave / Eric / Others,
> > > >
> > > > Syslog: https://pastebin.com/QYQYpPFY
> > > >
> > > > Dmesg: https://pastebin.com/MdBCPmp9
> > >
> > > which shows no stack traces, again.
> > >
> > >
> > >
> > > Anyway, you've twiddled mkfs knobs on these filesystems, and that
> > > is the likely cause of the issue: the filesystem is using 64k
> > > directory blocks - the allocation size is larger than 64kB:
> > >
> > > [Sun Nov 17 21:40:05 2019] XFS: nginx(31293) possible memory allocation deadlock size 65728 in kmem_alloc (mode:0x250)
> > >
> > > Upstream fixed this some time ago:
> > >
> > > $ ▶ gl -n 1 -p cb0a8d23024e
> > > commit cb0a8d23024e7bd234dea4d0fc5c4902a8dda766
> > > Author: Dave Chinner <dchinner@redhat.com>
> > > Date:   Tue Mar 6 17:03:28 2018 -0800
> > >
> > >     xfs: fall back to vmalloc when allocation log vector buffers
> > >
> > >     When using large directory blocks, we regularly see memory
> > >     allocations of >64k being made for the shadow log vector buffer.
> > >     When we are under memory pressure, kmalloc() may not be able to find
> > >     contiguous memory chunks large enough to satisfy these allocations
> > >     easily, and if memory is fragmented we can potentially stall here.
> > >
> > >     TO avoid this problem, switch the log vector buffer allocation to
> > >     use kmem_alloc_large(). This will allow failed allocations to fall
> > >     back to vmalloc and so remove the dependency on large contiguous
> > >     regions of memory being available. This should prevent slowdowns
> > >     and potential stalls when memory is low and/or fragmented.
> > >
> > >     Signed-Off-By: Dave Chinner <dchinner@redhat.com>
> > >     Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>
> > >     Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> > >
> > >
> > > Cheers,
> > >
> > > Dave.
> > > --
> > > Dave Chinner
> > > david@fromorbit.com
> >
> >
> >
> > --
> > With Regards,
> > Andrew Carr
> >
> > e. andrewlanecarr@gmail.com
> > w. andrew.carr@openlogic.com
> > c. 4239489206
> > a. P.O. Box 1231, Greeneville, TN, 37744
> 
> 
> 
> -- 
> With Regards,
> Andrew Carr
> 
> e. andrewlanecarr@gmail.com
> w. andrew.carr@openlogic.com
> c. 4239489206
> a. P.O. Box 1231, Greeneville, TN, 37744

Re: Fwd: XFS Memory allocation deadlock in kmem_alloc

[Darrick J. Wong]

On Fri, Nov 22, 2019 at 10:17:33AM -0800, Blake Golliher wrote:
> Where would those core dumps be?  Are they automatically dumped or do we
> have to set a flag, then trigger the condition?

ulimit -c 9999999999, then run whatever it was you were running that
invokes xfs_db.

--D

> On Fri, Nov 22, 2019 at 8:12 AM Darrick J. Wong <darrick.wong@oracle.com>
> wrote:
> 
> >
> >
> > CAUTION: External Email
> >
> >
> >
> >
> > On Fri, Nov 22, 2019 at 09:08:26AM -0500, Andrew Carr wrote:
> > > Hi Dave  / Others,
> > >
> > > It appears upgrading to 4.17+ has indeed fixed the deadlock issue, or
> > > at least no deadlocks are occurring now.
> > >
> > > There are segfaults in xfs_db appearing now though.  I am attempting
> > > to get the full syslog, here is an example.... thoughts?
> > >
> > > [Thu Nov 21 10:43:20 2019] xfs_db[13076]: segfault at 12ff6001 ip
> > > 0000000000407922 sp 00007ffe1a27b2e0 error 4 in xfs_db[400000+8a000]
> > > [Thu Nov 21 10:43:20 2019] Code: 89 cc 55 48 89 d5 53 48 89 f3 48 83
> > > ec 48 0f b6 57 01 44 0f b6 4f 02 64 48 8b 04 25 28 00 00 00 48 89 44
> > > 24 38 31 c0 0f b6 07 <44> 0f b6 57 0d 48 8d 74 24 10 c1 e2 10 41 c1 e1
> > > 08 c1 e0 18 41 c1
> >
> > Actual coredumps of the crashed xfs_db would help.
> >
> > --D
> >
> > > Thanks so much in advance!
> > > Andrew
> > >
> > > On Wed, Nov 20, 2019 at 10:43 AM Andrew Carr <andrewlanecarr@gmail.com>
> > wrote:
> > > >
> > > > Genius Dave, Thanks so much!
> > > >
> > > > On Tue, Nov 19, 2019 at 3:21 PM Dave Chinner <david@fromorbit.com>
> > wrote:
> > > > >
> > > > > On Tue, Nov 19, 2019 at 10:49:56AM -0500, Andrew Carr wrote:
> > > > > > Dave / Eric / Others,
> > > > > >
> > > > > > Syslog:
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__pastebin.com_QYQYpPFY&d=DwIDaQ&c=4dvmKrCYCD_MWOWC_k7VMw&r=NRVQX89iLxYf06dcpbIrijtLC-DKd-z7vxj002MWTmI&m=gtReaQZA21GCSFtWKk0Ycbpr-Ra30apUfn69fetsCyI&s=cFo_9R18qcbqlKAa2jfsMB02h74aHd4m04zbNPYS1-I&e=
> > > > > >
> > > > > > Dmesg:
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__pastebin.com_MdBCPmp9&d=DwIDaQ&c=4dvmKrCYCD_MWOWC_k7VMw&r=NRVQX89iLxYf06dcpbIrijtLC-DKd-z7vxj002MWTmI&m=gtReaQZA21GCSFtWKk0Ycbpr-Ra30apUfn69fetsCyI&s=E9ryV4GnH02exSAsoFGbq1arjRLkyffNjka_kZ4MV60&e=
> > > > >
> > > > > which shows no stack traces, again.
> > > > >
> > > > >
> > > > >
> > > > > Anyway, you've twiddled mkfs knobs on these filesystems, and that
> > > > > is the likely cause of the issue: the filesystem is using 64k
> > > > > directory blocks - the allocation size is larger than 64kB:
> > > > >
> > > > > [Sun Nov 17 21:40:05 2019] XFS: nginx(31293) possible memory
> > allocation deadlock size 65728 in kmem_alloc (mode:0x250)
> > > > >
> > > > > Upstream fixed this some time ago:
> > > > >
> > > > > $ ▶ gl -n 1 -p cb0a8d23024e
> > > > > commit cb0a8d23024e7bd234dea4d0fc5c4902a8dda766
> > > > > Author: Dave Chinner <dchinner@redhat.com>
> > > > > Date:   Tue Mar 6 17:03:28 2018 -0800
> > > > >
> > > > >     xfs: fall back to vmalloc when allocation log vector buffers
> > > > >
> > > > >     When using large directory blocks, we regularly see memory
> > > > >     allocations of >64k being made for the shadow log vector buffer.
> > > > >     When we are under memory pressure, kmalloc() may not be able to
> > find
> > > > >     contiguous memory chunks large enough to satisfy these
> > allocations
> > > > >     easily, and if memory is fragmented we can potentially stall
> > here.
> > > > >
> > > > >     TO avoid this problem, switch the log vector buffer allocation to
> > > > >     use kmem_alloc_large(). This will allow failed allocations to
> > fall
> > > > >     back to vmalloc and so remove the dependency on large contiguous
> > > > >     regions of memory being available. This should prevent slowdowns
> > > > >     and potential stalls when memory is low and/or fragmented.
> > > > >
> > > > >     Signed-Off-By: Dave Chinner <dchinner@redhat.com>
> > > > >     Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>
> > > > >     Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> > > > >
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Dave.
> > > > > --
> > > > > Dave Chinner
> > > > > david@fromorbit.com
> > > >
> > > >
> > > >
> > > > --
> > > > With Regards,
> > > > Andrew Carr
> > > >
> > > > e. andrewlanecarr@gmail.com
> > > > w. andrew.carr@openlogic.com
> > > > c. 4239489206
> > > > a. P.O. Box 1231, Greeneville, TN, 37744
> > >
> > >
> > >
> > > --
> > > With Regards,
> > > Andrew Carr
> > >
> > > e. andrewlanecarr@gmail.com
> > > w. andrew.carr@openlogic.com
> > > c. 4239489206
> > > a. P.O. Box 1231, Greeneville, TN, 37744
> >