Question to RWLock & reverse DNS ip=>hostname

Question to RWLock & reverse DNS ip=>hostname

[Andreas Joachim Peters]

Hi, 
by chance I had a look to the RWLock class. To my best knowledge the way you create RW locks it defaults to writer-starvation e.g. all readers will always jump a head of a pending writer. I cannot imagine that you never have the opposite requirement in the CEPH multithreaded code but I didn't review where it is used. In case you are aware, you can just ignore this comment, otherwise one could add the option to create a writer-prefering RWLock at construction time using e.g.
pthread_rwlockattr_setkind_np(&attr, PTHREAD_RWLOCK_PREFER_WRITER_NP)

Another question: I couldn't find a base implementation to translate IPV4/6 ip's to host names?
Is there same configuration table/method allowing to get back from an OSD address retrieved via OSDMap::get_addr() the original host name avoiding the use of DNS lookups?

Thanks Andreas.

Re: Question to RWLock & reverse DNS ip=>hostname

[Sage Weil]

On Tue, 9 Sep 2014, Andreas Joachim Peters wrote:
> Hi, 
> by chance I had a look to the RWLock class. To my best knowledge the way 
> you create RW locks it defaults to writer-starvation e.g. all readers 
> will always jump a head of a pending writer. I cannot imagine that you 
> never have the opposite requirement in the CEPH multithreaded code but I 
> didn't review where it is used. In case you are aware, you can just 
> ignore this comment, otherwise one could add the option to create a 
> writer-prefering RWLock at construction time using e.g. 
> pthread_rwlockattr_setkind_np(&attr, PTHREAD_RWLOCK_PREFER_WRITER_NP)

Hmm, I was not aware.  Perhaps we should make the reader vs writer 
preference explicit during construction?

> Another question: I couldn't find a base implementation to translate 
> IPV4/6 ip's to host names? Is there same configuration table/method 
> allowing to get back from an OSD address retrieved via 
> OSDMap::get_addr() the original host name avoiding the use of DNS 
> lookups?

I don't think we are doing reverse DNS lookups anywhere in the ceph 
code ...

sage

Re: Question to RWLock & reverse DNS ip=>hostname

[Gregory Farnum]

On Tue, Sep 9, 2014 at 10:50 AM, Sage Weil <sweil@redhat.com> wrote:
> On Tue, 9 Sep 2014, Andreas Joachim Peters wrote:
>> Hi,
>> by chance I had a look to the RWLock class. To my best knowledge the way
>> you create RW locks it defaults to writer-starvation e.g. all readers
>> will always jump a head of a pending writer. I cannot imagine that you
>> never have the opposite requirement in the CEPH multithreaded code but I
>> didn't review where it is used. In case you are aware, you can just
>> ignore this comment, otherwise one could add the option to create a
>> writer-prefering RWLock at construction time using e.g.
>> pthread_rwlockattr_setkind_np(&attr, PTHREAD_RWLOCK_PREFER_WRITER_NP)
>
> Hmm, I was not aware.  Perhaps we should make the reader vs writer
> preference explicit during construction?

Yeah, this is interesting as I thought I'd learned somewhere that the
default behavior was to prefer writers. Poking around the internet
makes me wonder if we actually want to make any changes — best I can
tell PTHREAD_RWLOCK_PREFER_WRITER_NP does not actually behave the way
one wants (as part of the spec!) because pthreads requires recursive
locking, so we'd have to use
PTHREAD_RWLOCK_PREFER_WRITER_NONRECURSIVE_NP. But we'd probably want
to profile the performance changes (and, uh, run tests to make sure we
don't deadlock somehow) before putting any such code upstream.

>> Another question: I couldn't find a base implementation to translate
>> IPV4/6 ip's to host names? Is there same configuration table/method
>> allowing to get back from an OSD address retrieved via
>> OSDMap::get_addr() the original host name avoiding the use of DNS
>> lookups?
>
> I don't think we are doing reverse DNS lookups anywhere in the ceph
> code ...

In particular I think we've looked at doing this before and been
unable to reliably map to the *proper* hostname in cases where servers
have multiple NICs.
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Question to RWLock & reverse DNS ip=>hostname

[Andreas Joachim Peters]

We did a lot of testing of RW locks in another multi-threaded project and we use successfully "PTHREAD_RWLOCK_PREFER_WRITER_NP" which works perfectly fine if you avoid recursive read locks. It is probably good practice to avoid any recursive mutex in general.

If one stay's with the default RW mutex without setting attributes we have seen that you starve writers for long periods if there is a contention on the read mutex (n-reader >> n-writer). I agree this is a very 'sensitive' topic but could well be an origin for threading scalability problems - depends how and where you use the RW locks in CEPH.

Cheers Andreas.

________________________________________
From: Gregory Farnum [greg@inktank.com]
Sent: 09 September 2014 21:56
To: Sage Weil
Cc: Andreas Joachim Peters; ceph-devel@vger.kernel.org
Subject: Re: Question to RWLock & reverse DNS ip=>hostname

On Tue, Sep 9, 2014 at 10:50 AM, Sage Weil <sweil@redhat.com> wrote:
> On Tue, 9 Sep 2014, Andreas Joachim Peters wrote:
>> Hi,
>> by chance I had a look to the RWLock class. To my best knowledge the way
>> you create RW locks it defaults to writer-starvation e.g. all readers
>> will always jump a head of a pending writer. I cannot imagine that you
>> never have the opposite requirement in the CEPH multithreaded code but I
>> didn't review where it is used. In case you are aware, you can just
>> ignore this comment, otherwise one could add the option to create a
>> writer-prefering RWLock at construction time using e.g.
>> pthread_rwlockattr_setkind_np(&attr, PTHREAD_RWLOCK_PREFER_WRITER_NP)
>
> Hmm, I was not aware.  Perhaps we should make the reader vs writer
> preference explicit during construction?

Yeah, this is interesting as I thought I'd learned somewhere that the
default behavior was to prefer writers. Poking around the internet
makes me wonder if we actually want to make any changes — best I can
tell PTHREAD_RWLOCK_PREFER_WRITER_NP does not actually behave the way
one wants (as part of the spec!) because pthreads requires recursive
locking, so we'd have to use
PTHREAD_RWLOCK_PREFER_WRITER_NONRECURSIVE_NP. But we'd probably want
to profile the performance changes (and, uh, run tests to make sure we
don't deadlock somehow) before putting any such code upstream.

>> Another question: I couldn't find a base implementation to translate
>> IPV4/6 ip's to host names? Is there same configuration table/method
>> allowing to get back from an OSD address retrieved via
>> OSDMap::get_addr() the original host name avoiding the use of DNS
>> lookups?
>
> I don't think we are doing reverse DNS lookups anywhere in the ceph
> code ...

In particular I think we've looked at doing this before and been
unable to reliably map to the *proper* hostname in cases where servers
have multiple NICs.
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Question to RWLock & reverse DNS ip=>hostname

[Sage Weil]

On Wed, 10 Sep 2014, Andreas Joachim Peters wrote:
> We did a lot of testing of RW locks in another multi-threaded project 
> and we use successfully "PTHREAD_RWLOCK_PREFER_WRITER_NP" which works 
> perfectly fine if you avoid recursive read locks. It is probably good 
> practice to avoid any recursive mutex in general.

The good news is that there are (I believe) no recursive uses of the 
mutexes or rwlocks in the code base.  Our lockdep dependency checking 
system throws errors if it detects any recursion.  There might be a few 
exceptions that are explicitly marked recursive via the constructor.  
 
> If one stay's with the default RW mutex without setting attributes we 
> have seen that you starve writers for long periods if there is a 
> contention on the read mutex (n-reader >> n-writer). I agree this is a 
> very 'sensitive' topic but could well be an origin for threading 
> scalability problems - depends how and where you use the RW locks in 
> CEPH.

Yeah!

I suspect the way to approach this would be to make reader vs writer 
favortism explicit in the constructor arguments, which no change in 
behavior, and then a series of patches that switch to writer-favortism...

sage

> 
> Cheers Andreas.
> 
> ________________________________________
> From: Gregory Farnum [greg@inktank.com]
> Sent: 09 September 2014 21:56
> To: Sage Weil
> Cc: Andreas Joachim Peters; ceph-devel@vger.kernel.org
> Subject: Re: Question to RWLock & reverse DNS ip=>hostname
> 
> On Tue, Sep 9, 2014 at 10:50 AM, Sage Weil <sweil@redhat.com> wrote:
> > On Tue, 9 Sep 2014, Andreas Joachim Peters wrote:
> >> Hi,
> >> by chance I had a look to the RWLock class. To my best knowledge the way
> >> you create RW locks it defaults to writer-starvation e.g. all readers
> >> will always jump a head of a pending writer. I cannot imagine that you
> >> never have the opposite requirement in the CEPH multithreaded code but I
> >> didn't review where it is used. In case you are aware, you can just
> >> ignore this comment, otherwise one could add the option to create a
> >> writer-prefering RWLock at construction time using e.g.
> >> pthread_rwlockattr_setkind_np(&attr, PTHREAD_RWLOCK_PREFER_WRITER_NP)
> >
> > Hmm, I was not aware.  Perhaps we should make the reader vs writer
> > preference explicit during construction?
> 
> Yeah, this is interesting as I thought I'd learned somewhere that the
> default behavior was to prefer writers. Poking around the internet
> makes me wonder if we actually want to make any changes ? best I can
> tell PTHREAD_RWLOCK_PREFER_WRITER_NP does not actually behave the way
> one wants (as part of the spec!) because pthreads requires recursive
> locking, so we'd have to use
> PTHREAD_RWLOCK_PREFER_WRITER_NONRECURSIVE_NP. But we'd probably want
> to profile the performance changes (and, uh, run tests to make sure we
> don't deadlock somehow) before putting any such code upstream.
> 
> >> Another question: I couldn't find a base implementation to translate
> >> IPV4/6 ip's to host names? Is there same configuration table/method
> >> allowing to get back from an OSD address retrieved via
> >> OSDMap::get_addr() the original host name avoiding the use of DNS
> >> lookups?
> >
> > I don't think we are doing reverse DNS lookups anywhere in the ceph
> > code ...
> 
> In particular I think we've looked at doing this before and been
> unable to reliably map to the *proper* hostname in cases where servers
> have multiple NICs.
> -Greg
> Software Engineer #42 @ http://inktank.com | http://ceph.com
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
>