Re: Prioritize Heartbeat packets

Re: Prioritize Heartbeat packets

[Robert LeBlanc]

[-- Attachment #1.1: Type: text/plain, Size: 1446 bytes --]

I see that Jian Wen has done work on this for 0.94. I tried looking through
the code to see if I can figure out how to configure this new option, but
it all went over my head pretty quick.

Can I get a brief summary on how to set the priority of heartbeat packets
or where to look in the code to figure it out?

Thanks,

On Thu, Aug 28, 2014 at 2:01 AM, Daniel Swarbrick <
daniel.swarbrick-EIkl63zCoXaH+58JC4qpiA@public.gmane.org> wrote:

> On 28/08/14 02:56, Sage Weil wrote:
> > I seem to remember someone telling me there were hooks/hints you could
> > call that would tag either a socket or possibly data on that socket with
> a
> > label for use by iptables and such.. but I forget what it was.
> >
>
> Something like setsockopt() SO_MARK?
>
>        *SO_MARK *(since Linux 2.6.25)
>               Set the mark for each packet sent through this socket
> (similar
>               to the netfilter MARK target but socket-based).  Changing the
>               mark can be used for mark-based routing without netfilter or
>               for packet filtering.  Setting this option requires the
>               *CAP_NET_ADMIN *capability.
>
> Alternatively, directly set IP_TOS options on the socket, or SO_PRIORITY
> which sets the IP TOS bits as well.
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>

[-- Attachment #1.2: Type: text/html, Size: 2320 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Fwd: Prioritize Heartbeat packets

[Robert LeBlanc]

Hidden HTML ... trying agin...

---------- Forwarded message ----------
From: Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org>
Date: Fri, Mar 6, 2015 at 5:20 PM
Subject: Re: [ceph-users] Prioritize Heartbeat packets
To: "ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org" <ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org>,
ceph-devel <ceph-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>


I see that Jian Wen has done work on this for 0.94. I tried looking
through the code to see if I can figure out how to configure this new
option, but it all went over my head pretty quick.

Can I get a brief summary on how to set the priority of heartbeat
packets or where to look in the code to figure it out?

Thanks,

On Thu, Aug 28, 2014 at 2:01 AM, Daniel Swarbrick
<daniel.swarbrick-EIkl63zCoXaH+58JC4qpiA@public.gmane.org> wrote:
>
> On 28/08/14 02:56, Sage Weil wrote:
> > I seem to remember someone telling me there were hooks/hints you could
> > call that would tag either a socket or possibly data on that socket with a
> > label for use by iptables and such.. but I forget what it was.
> >
>
> Something like setsockopt() SO_MARK?
>
>        *SO_MARK *(since Linux 2.6.25)
>               Set the mark for each packet sent through this socket (similar
>               to the netfilter MARK target but socket-based).  Changing the
>               mark can be used for mark-based routing without netfilter or
>               for packet filtering.  Setting this option requires the
>               *CAP_NET_ADMIN *capability.
>
> Alternatively, directly set IP_TOS options on the socket, or SO_PRIORITY
> which sets the IP TOS bits as well.
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: Prioritize Heartbeat packets

[Daniel Swarbrick]

Judging by the commit, this ought to do the trick:

osd heartbeat use min delay socket = true

On 07/03/15 01:20, Robert LeBlanc wrote:
> I see that Jian Wen has done work on this for 0.94. I tried looking
> through the code to see if I can figure out how to configure this new
> option, but it all went over my head pretty quick.
>
> Can I get a brief summary on how to set the priority of heartbeat
> packets or where to look in the code to figure it out?
>
> Thanks,
>
> On Thu, Aug 28, 2014 at 2:01 AM, Daniel Swarbrick
> <daniel.swarbrick@profitbricks.com
> <mailto:daniel.swarbrick@profitbricks.com>> wrote:
>
>     On 28/08/14 02:56, Sage Weil wrote:
>     > I seem to remember someone telling me there were hooks/hints you could
>     > call that would tag either a socket or possibly data on that socket with a
>     > label for use by iptables and such.. but I forget what it was.
>     >
>
>     Something like setsockopt() SO_MARK?
>
>             *SO_MARK *(since Linux 2.6.25)
>                    Set the mark for each packet sent through this socket
>     (similar
>                    to the netfilter MARK target but socket-based).
>     Changing the
>                    mark can be used for mark-based routing without
>     netfilter or
>                    for packet filtering.  Setting this option requires the
>                    *CAP_NET_ADMIN *capability.
>
>     Alternatively, directly set IP_TOS options on the socket, or SO_PRIORITY
>     which sets the IP TOS bits as well.
>
>
>     _______________________________________________
>     ceph-users mailing list
>     ceph-users@lists.ceph.com
>     <mailto:ceph-users@lists.ceph.com>
>     http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>

Re: Prioritize Heartbeat packets

[Robert LeBlanc]

I've found commit 9b9a682fe035c985e416ee1c112fa58f9045a27c and I see
that when 'osd heartbeat use min delay socket = true' it will mark the
packet with DSCP CS6. Based on the setting of the socket in
msg/simple/Pipe.cc is it possible that this can apply to both OSD and
monitor? I don't understand the code enough to know how the
set_socket_options() is called from the OSD and monitor.

If this applies to both monitor and OSD, would it be better to rename
the option to a more generic name?

Thanks,

On Sat, Mar 7, 2015 at 4:23 PM, Daniel Swarbrick
<daniel.swarbrick-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Judging by the commit, this ought to do the trick:
>
> osd heartbeat use min delay socket = true
>
> On 07/03/15 01:20, Robert LeBlanc wrote:
>>
>> I see that Jian Wen has done work on this for 0.94. I tried looking
>> through the code to see if I can figure out how to configure this new
>> option, but it all went over my head pretty quick.
>>
>> Can I get a brief summary on how to set the priority of heartbeat
>> packets or where to look in the code to figure it out?
>>
>> Thanks,
>>
>> On Thu, Aug 28, 2014 at 2:01 AM, Daniel Swarbrick
>> <daniel.swarbrick-EIkl63zCoXaH+58JC4qpiA@public.gmane.org
>> <mailto:daniel.swarbrick-EIkl63zCoXaH+58JC4qpiA@public.gmane.org>> wrote:
>>
>>     On 28/08/14 02:56, Sage Weil wrote:
>>     > I seem to remember someone telling me there were hooks/hints you
>> could
>>     > call that would tag either a socket or possibly data on that socket
>> with a
>>     > label for use by iptables and such.. but I forget what it was.
>>     >
>>
>>     Something like setsockopt() SO_MARK?
>>
>>             *SO_MARK *(since Linux 2.6.25)
>>                    Set the mark for each packet sent through this socket
>>     (similar
>>                    to the netfilter MARK target but socket-based).
>>     Changing the
>>                    mark can be used for mark-based routing without
>>     netfilter or
>>                    for packet filtering.  Setting this option requires the
>>                    *CAP_NET_ADMIN *capability.
>>
>>     Alternatively, directly set IP_TOS options on the socket, or
>> SO_PRIORITY
>>     which sets the IP TOS bits as well.
>>
>>
>>     _______________________________________________
>>     ceph-users mailing list
>>     ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
>>     <mailto:ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org>
>>     http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>>
>>
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [ceph-users] Prioritize Heartbeat packets

[Jian Wen]

Only OSD calls set_socket_priority().
See  https://github.com/ceph/ceph/pull/3353

On Tue, Mar 10, 2015 at 3:36 AM, Robert LeBlanc <robert@leblancnet.us> wrote:
> I've found commit 9b9a682fe035c985e416ee1c112fa58f9045a27c and I see
> that when 'osd heartbeat use min delay socket = true' it will mark the
> packet with DSCP CS6. Based on the setting of the socket in
> msg/simple/Pipe.cc is it possible that this can apply to both OSD and
> monitor? I don't understand the code enough to know how the
> set_socket_options() is called from the OSD and monitor.
>
> If this applies to both monitor and OSD, would it be better to rename
> the option to a more generic name?
>
> Thanks,
>
> On Sat, Mar 7, 2015 at 4:23 PM, Daniel Swarbrick
> <daniel.swarbrick@gmail.com> wrote:
>> Judging by the commit, this ought to do the trick:
>>
>> osd heartbeat use min delay socket = true
>>
>> On 07/03/15 01:20, Robert LeBlanc wrote:
>>>
>>> I see that Jian Wen has done work on this for 0.94. I tried looking
>>> through the code to see if I can figure out how to configure this new
>>> option, but it all went over my head pretty quick.
>>>
>>> Can I get a brief summary on how to set the priority of heartbeat
>>> packets or where to look in the code to figure it out?
>>>
>>> Thanks,
>>>
>>> On Thu, Aug 28, 2014 at 2:01 AM, Daniel Swarbrick
>>> <daniel.swarbrick@profitbricks.com
>>> <mailto:daniel.swarbrick@profitbricks.com>> wrote:
>>>
>>>     On 28/08/14 02:56, Sage Weil wrote:
>>>     > I seem to remember someone telling me there were hooks/hints you
>>> could
>>>     > call that would tag either a socket or possibly data on that socket
>>> with a
>>>     > label for use by iptables and such.. but I forget what it was.
>>>     >
>>>
>>>     Something like setsockopt() SO_MARK?
>>>
>>>             *SO_MARK *(since Linux 2.6.25)
>>>                    Set the mark for each packet sent through this socket
>>>     (similar
>>>                    to the netfilter MARK target but socket-based).
>>>     Changing the
>>>                    mark can be used for mark-based routing without
>>>     netfilter or
>>>                    for packet filtering.  Setting this option requires the
>>>                    *CAP_NET_ADMIN *capability.
>>>
>>>     Alternatively, directly set IP_TOS options on the socket, or
>>> SO_PRIORITY
>>>     which sets the IP TOS bits as well.
>>>
>>>
>>>     _______________________________________________
>>>     ceph-users mailing list
>>>     ceph-users@lists.ceph.com
>>>     <mailto:ceph-users@lists.ceph.com>
>>>     http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



-- 
Best,

Jian

Re: [ceph-users] Prioritize Heartbeat packets

[Robert LeBlanc]

Jian,

Thanks for the clarification. I'll mark traffic destined for the
monitors as well. We are getting ready to put our first cluster into
production. If you are interested we will be testing the heartbeat
priority to see if we can saturate the network (not an easy task for
40 Gb) and keep the cluster from falling apart. Our network team is
marking COS based on the DSCP and enforcing priority. We have three
VLANs on bonded 40 GbE, management, storage (monitors, clients, OSDs),
and cluster (replication). We have three priority classes management
(heartbeats on all VLANs, SSH, DNS, etc), storage traffic (no
marking), and replication (scavenger class). We are interested to see
how things pan out.

Thanks,
Robert

On Mon, Mar 9, 2015 at 8:58 PM, Jian Wen <wenjianhn@gmail.com> wrote:
> Only OSD calls set_socket_priority().
> See  https://github.com/ceph/ceph/pull/3353
>
> On Tue, Mar 10, 2015 at 3:36 AM, Robert LeBlanc <robert@leblancnet.us> wrote:
>> I've found commit 9b9a682fe035c985e416ee1c112fa58f9045a27c and I see
>> that when 'osd heartbeat use min delay socket = true' it will mark the
>> packet with DSCP CS6. Based on the setting of the socket in
>> msg/simple/Pipe.cc is it possible that this can apply to both OSD and
>> monitor? I don't understand the code enough to know how the
>> set_socket_options() is called from the OSD and monitor.
>>
>> If this applies to both monitor and OSD, would it be better to rename
>> the option to a more generic name?
>>
>> Thanks,
>>
>> On Sat, Mar 7, 2015 at 4:23 PM, Daniel Swarbrick
>> <daniel.swarbrick@gmail.com> wrote:
>>> Judging by the commit, this ought to do the trick:
>>>
>>> osd heartbeat use min delay socket = true
>>>
>>> On 07/03/15 01:20, Robert LeBlanc wrote:
>>>>
>>>> I see that Jian Wen has done work on this for 0.94. I tried looking
>>>> through the code to see if I can figure out how to configure this new
>>>> option, but it all went over my head pretty quick.
>>>>
>>>> Can I get a brief summary on how to set the priority of heartbeat
>>>> packets or where to look in the code to figure it out?
>>>>
>>>> Thanks,
>>>>
>>>> On Thu, Aug 28, 2014 at 2:01 AM, Daniel Swarbrick
>>>> <daniel.swarbrick@profitbricks.com
>>>> <mailto:daniel.swarbrick@profitbricks.com>> wrote:
>>>>
>>>>     On 28/08/14 02:56, Sage Weil wrote:
>>>>     > I seem to remember someone telling me there were hooks/hints you
>>>> could
>>>>     > call that would tag either a socket or possibly data on that socket
>>>> with a
>>>>     > label for use by iptables and such.. but I forget what it was.
>>>>     >
>>>>
>>>>     Something like setsockopt() SO_MARK?
>>>>
>>>>             *SO_MARK *(since Linux 2.6.25)
>>>>                    Set the mark for each packet sent through this socket
>>>>     (similar
>>>>                    to the netfilter MARK target but socket-based).
>>>>     Changing the
>>>>                    mark can be used for mark-based routing without
>>>>     netfilter or
>>>>                    for packet filtering.  Setting this option requires the
>>>>                    *CAP_NET_ADMIN *capability.
>>>>
>>>>     Alternatively, directly set IP_TOS options on the socket, or
>>>> SO_PRIORITY
>>>>     which sets the IP TOS bits as well.
>>>>
>>>>
>>>>     _______________________________________________
>>>>     ceph-users mailing list
>>>>     ceph-users@lists.ceph.com
>>>>     <mailto:ceph-users@lists.ceph.com>
>>>>     http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> ceph-users mailing list
>>>> ceph-users@lists.ceph.com
>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>
>>>
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
> --
> Best,
>
> Jian

Re: Prioritize Heartbeat packets

[Jian Wen]

Robert,
    60 days ago, we enabled this feature in our production environment.
    It works for us.

    Did you see any flapping OSDs?

On Tue, Mar 10, 2015 at 1:16 PM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> wrote:
> Jian,
>
> Thanks for the clarification. I'll mark traffic destined for the
> monitors as well. We are getting ready to put our first cluster into
> production. If you are interested we will be testing the heartbeat
> priority to see if we can saturate the network (not an easy task for
> 40 Gb) and keep the cluster from falling apart. Our network team is
> marking COS based on the DSCP and enforcing priority. We have three
> VLANs on bonded 40 GbE, management, storage (monitors, clients, OSDs),
> and cluster (replication). We have three priority classes management
> (heartbeats on all VLANs, SSH, DNS, etc), storage traffic (no
> marking), and replication (scavenger class). We are interested to see
> how things pan out.
>
> Thanks,
> Robert
>
> On Mon, Mar 9, 2015 at 8:58 PM, Jian Wen <wenjianhn-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> Only OSD calls set_socket_priority().
>> See  https://github.com/ceph/ceph/pull/3353
>>
>> On Tue, Mar 10, 2015 at 3:36 AM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> wrote:
>>> I've found commit 9b9a682fe035c985e416ee1c112fa58f9045a27c and I see
>>> that when 'osd heartbeat use min delay socket = true' it will mark the
>>> packet with DSCP CS6. Based on the setting of the socket in
>>> msg/simple/Pipe.cc is it possible that this can apply to both OSD and
>>> monitor? I don't understand the code enough to know how the
>>> set_socket_options() is called from the OSD and monitor.
>>>
>>> If this applies to both monitor and OSD, would it be better to rename
>>> the option to a more generic name?
>>>
>>> Thanks,
>>>
>>> On Sat, Mar 7, 2015 at 4:23 PM, Daniel Swarbrick
>>> <daniel.swarbrick-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>>> Judging by the commit, this ought to do the trick:
>>>>
>>>> osd heartbeat use min delay socket = true
>>>>
>>>> On 07/03/15 01:20, Robert LeBlanc wrote:
>>>>>
>>>>> I see that Jian Wen has done work on this for 0.94. I tried looking
>>>>> through the code to see if I can figure out how to configure this new
>>>>> option, but it all went over my head pretty quick.
>>>>>
>>>>> Can I get a brief summary on how to set the priority of heartbeat
>>>>> packets or where to look in the code to figure it out?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> On Thu, Aug 28, 2014 at 2:01 AM, Daniel Swarbrick
>>>>> <daniel.swarbrick-EIkl63zCoXaH+58JC4qpiA@public.gmane.org
>>>>> <mailto:daniel.swarbrick-EIkl63zCoXaH+58JC4qpiA@public.gmane.org>> wrote:
>>>>>
>>>>>     On 28/08/14 02:56, Sage Weil wrote:
>>>>>     > I seem to remember someone telling me there were hooks/hints you
>>>>> could
>>>>>     > call that would tag either a socket or possibly data on that socket
>>>>> with a
>>>>>     > label for use by iptables and such.. but I forget what it was.
>>>>>     >
>>>>>
>>>>>     Something like setsockopt() SO_MARK?
>>>>>
>>>>>             *SO_MARK *(since Linux 2.6.25)
>>>>>                    Set the mark for each packet sent through this socket
>>>>>     (similar
>>>>>                    to the netfilter MARK target but socket-based).
>>>>>     Changing the
>>>>>                    mark can be used for mark-based routing without
>>>>>     netfilter or
>>>>>                    for packet filtering.  Setting this option requires the
>>>>>                    *CAP_NET_ADMIN *capability.
>>>>>
>>>>>     Alternatively, directly set IP_TOS options on the socket, or
>>>>> SO_PRIORITY
>>>>>     which sets the IP TOS bits as well.
>>>>>
>>>>>
>>>>>     _______________________________________________
>>>>>     ceph-users mailing list
>>>>>     ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
>>>>>     <mailto:ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org>
>>>>>     http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> ceph-users mailing list
>>>>> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>>
>>>>
>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>> _______________________________________________
>>> ceph-users mailing list
>>> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>>
>> --
>> Best,
>>
>> Jian



-- 
Best,

Jian

Re: Prioritize Heartbeat packets

[Robert LeBlanc]

[-- Attachment #1.1: Type: text/plain, Size: 5340 bytes --]

We have had some delays. Our to be production cluster is currently on
firefly but should be on hammer next week. Once upgraded we will preform
the network saturation test and verify results.

Robert LeBlanc

Sent from a mobile device please excuse any typos.
On Apr 10, 2015 12:39 AM, "Jian Wen" <wenjianhn-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

> Robert,
>     60 days ago, we enabled this feature in our production environment.
>     It works for us.
>
>     Did you see any flapping OSDs?
>
> On Tue, Mar 10, 2015 at 1:16 PM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org>
> wrote:
> > Jian,
> >
> > Thanks for the clarification. I'll mark traffic destined for the
> > monitors as well. We are getting ready to put our first cluster into
> > production. If you are interested we will be testing the heartbeat
> > priority to see if we can saturate the network (not an easy task for
> > 40 Gb) and keep the cluster from falling apart. Our network team is
> > marking COS based on the DSCP and enforcing priority. We have three
> > VLANs on bonded 40 GbE, management, storage (monitors, clients, OSDs),
> > and cluster (replication). We have three priority classes management
> > (heartbeats on all VLANs, SSH, DNS, etc), storage traffic (no
> > marking), and replication (scavenger class). We are interested to see
> > how things pan out.
> >
> > Thanks,
> > Robert
> >
> > On Mon, Mar 9, 2015 at 8:58 PM, Jian Wen <wenjianhn-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> >> Only OSD calls set_socket_priority().
> >> See  https://github.com/ceph/ceph/pull/3353
> >>
> >> On Tue, Mar 10, 2015 at 3:36 AM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org>
> wrote:
> >>> I've found commit 9b9a682fe035c985e416ee1c112fa58f9045a27c and I see
> >>> that when 'osd heartbeat use min delay socket = true' it will mark the
> >>> packet with DSCP CS6. Based on the setting of the socket in
> >>> msg/simple/Pipe.cc is it possible that this can apply to both OSD and
> >>> monitor? I don't understand the code enough to know how the
> >>> set_socket_options() is called from the OSD and monitor.
> >>>
> >>> If this applies to both monitor and OSD, would it be better to rename
> >>> the option to a more generic name?
> >>>
> >>> Thanks,
> >>>
> >>> On Sat, Mar 7, 2015 at 4:23 PM, Daniel Swarbrick
> >>> <daniel.swarbrick-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> >>>> Judging by the commit, this ought to do the trick:
> >>>>
> >>>> osd heartbeat use min delay socket = true
> >>>>
> >>>> On 07/03/15 01:20, Robert LeBlanc wrote:
> >>>>>
> >>>>> I see that Jian Wen has done work on this for 0.94. I tried looking
> >>>>> through the code to see if I can figure out how to configure this new
> >>>>> option, but it all went over my head pretty quick.
> >>>>>
> >>>>> Can I get a brief summary on how to set the priority of heartbeat
> >>>>> packets or where to look in the code to figure it out?
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> On Thu, Aug 28, 2014 at 2:01 AM, Daniel Swarbrick
> >>>>> <daniel.swarbrick-EIkl63zCoXaH+58JC4qpiA@public.gmane.org
> >>>>> <mailto:daniel.swarbrick-EIkl63zCoXaH+58JC4qpiA@public.gmane.org>> wrote:
> >>>>>
> >>>>>     On 28/08/14 02:56, Sage Weil wrote:
> >>>>>     > I seem to remember someone telling me there were hooks/hints
> you
> >>>>> could
> >>>>>     > call that would tag either a socket or possibly data on that
> socket
> >>>>> with a
> >>>>>     > label for use by iptables and such.. but I forget what it was.
> >>>>>     >
> >>>>>
> >>>>>     Something like setsockopt() SO_MARK?
> >>>>>
> >>>>>             *SO_MARK *(since Linux 2.6.25)
> >>>>>                    Set the mark for each packet sent through this
> socket
> >>>>>     (similar
> >>>>>                    to the netfilter MARK target but socket-based).
> >>>>>     Changing the
> >>>>>                    mark can be used for mark-based routing without
> >>>>>     netfilter or
> >>>>>                    for packet filtering.  Setting this option
> requires the
> >>>>>                    *CAP_NET_ADMIN *capability.
> >>>>>
> >>>>>     Alternatively, directly set IP_TOS options on the socket, or
> >>>>> SO_PRIORITY
> >>>>>     which sets the IP TOS bits as well.
> >>>>>
> >>>>>
> >>>>>     _______________________________________________
> >>>>>     ceph-users mailing list
> >>>>>     ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> >>>>>     <mailto:ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org>
> >>>>>     http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> _______________________________________________
> >>>>> ceph-users mailing list
> >>>>> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> >>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in
> >>>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> >>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>> _______________________________________________
> >>> ceph-users mailing list
> >>> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >>
> >>
> >>
> >> --
> >> Best,
> >>
> >> Jian
>
>
>
> --
> Best,
>
> Jian
>

[-- Attachment #1.2: Type: text/html, Size: 8471 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Prioritize Heartbeat packets

[Robert LeBlanc]

We have had some delays. Our to be production cluster is currently on
firefly but should be on hammer next week. Once upgraded we will
preform the network saturation test and verify results.

On Fri, Apr 10, 2015 at 12:39 AM, Jian Wen <wenjianhn@gmail.com> wrote:
> Robert,
>     60 days ago, we enabled this feature in our production environment.
>     It works for us.
>
>     Did you see any flapping OSDs?
>
> On Tue, Mar 10, 2015 at 1:16 PM, Robert LeBlanc <robert@leblancnet.us> wrote:
>> Jian,
>>
>> Thanks for the clarification. I'll mark traffic destined for the
>> monitors as well. We are getting ready to put our first cluster into
>> production. If you are interested we will be testing the heartbeat
>> priority to see if we can saturate the network (not an easy task for
>> 40 Gb) and keep the cluster from falling apart. Our network team is
>> marking COS based on the DSCP and enforcing priority. We have three
>> VLANs on bonded 40 GbE, management, storage (monitors, clients, OSDs),
>> and cluster (replication). We have three priority classes management
>> (heartbeats on all VLANs, SSH, DNS, etc), storage traffic (no
>> marking), and replication (scavenger class). We are interested to see
>> how things pan out.
>>
>> Thanks,
>> Robert
>>
>> On Mon, Mar 9, 2015 at 8:58 PM, Jian Wen <wenjianhn@gmail.com> wrote:
>>> Only OSD calls set_socket_priority().
>>> See  https://github.com/ceph/ceph/pull/3353
>>>
>>> On Tue, Mar 10, 2015 at 3:36 AM, Robert LeBlanc <robert@leblancnet.us> wrote:
>>>> I've found commit 9b9a682fe035c985e416ee1c112fa58f9045a27c and I see
>>>> that when 'osd heartbeat use min delay socket = true' it will mark the
>>>> packet with DSCP CS6. Based on the setting of the socket in
>>>> msg/simple/Pipe.cc is it possible that this can apply to both OSD and
>>>> monitor? I don't understand the code enough to know how the
>>>> set_socket_options() is called from the OSD and monitor.
>>>>
>>>> If this applies to both monitor and OSD, would it be better to rename
>>>> the option to a more generic name?
>>>>
>>>> Thanks,
>>>>
>>>> On Sat, Mar 7, 2015 at 4:23 PM, Daniel Swarbrick
>>>> <daniel.swarbrick@gmail.com> wrote:
>>>>> Judging by the commit, this ought to do the trick:
>>>>>
>>>>> osd heartbeat use min delay socket = true
>>>>>
>>>>> On 07/03/15 01:20, Robert LeBlanc wrote:
>>>>>>
>>>>>> I see that Jian Wen has done work on this for 0.94. I tried looking
>>>>>> through the code to see if I can figure out how to configure this new
>>>>>> option, but it all went over my head pretty quick.
>>>>>>
>>>>>> Can I get a brief summary on how to set the priority of heartbeat
>>>>>> packets or where to look in the code to figure it out?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> On Thu, Aug 28, 2014 at 2:01 AM, Daniel Swarbrick
>>>>>> <daniel.swarbrick@profitbricks.com
>>>>>> <mailto:daniel.swarbrick@profitbricks.com>> wrote:
>>>>>>
>>>>>>     On 28/08/14 02:56, Sage Weil wrote:
>>>>>>     > I seem to remember someone telling me there were hooks/hints you
>>>>>> could
>>>>>>     > call that would tag either a socket or possibly data on that socket
>>>>>> with a
>>>>>>     > label for use by iptables and such.. but I forget what it was.
>>>>>>     >
>>>>>>
>>>>>>     Something like setsockopt() SO_MARK?
>>>>>>
>>>>>>             *SO_MARK *(since Linux 2.6.25)
>>>>>>                    Set the mark for each packet sent through this socket
>>>>>>     (similar
>>>>>>                    to the netfilter MARK target but socket-based).
>>>>>>     Changing the
>>>>>>                    mark can be used for mark-based routing without
>>>>>>     netfilter or
>>>>>>                    for packet filtering.  Setting this option requires the
>>>>>>                    *CAP_NET_ADMIN *capability.
>>>>>>
>>>>>>     Alternatively, directly set IP_TOS options on the socket, or
>>>>>> SO_PRIORITY
>>>>>>     which sets the IP TOS bits as well.
>>>>>>
>>>>>>
>>>>>>     _______________________________________________
>>>>>>     ceph-users mailing list
>>>>>>     ceph-users@lists.ceph.com
>>>>>>     <mailto:ceph-users@lists.ceph.com>
>>>>>>     http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> ceph-users mailing list
>>>>>> ceph-users@lists.ceph.com
>>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>> _______________________________________________
>>>> ceph-users mailing list
>>>> ceph-users@lists.ceph.com
>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>>>
>>>
>>> --
>>> Best,
>>>
>>> Jian
>
>
>
> --
> Best,
>
> Jian