monitor port closed

monitor port closed

[Ming Lin]

Hi,

[mon]
mon initial members = a

[mon.a]
mon addr = 192.168.0.1:7000

jewel release with 1 monitor and 3 OSDs.

After hours of fio rbd test, the monitor binded port(7000) was closed
for unknown reason.
But monitor is still running,

# pidof ceph-mon
21980

Another strange thing is there are more than 1000 threads for the
ceph-mon process.

# ls /proc/21980/task/ |wc -l
1022

Any possible reason?

Thanks,
Ming

Re: monitor port closed

[kefu chai]

On Mon, Jan 23, 2017 at 1:32 PM, Ming Lin <minggr@gmail.com> wrote:
> Hi,
>
> [mon]
> mon initial members = a
>
> [mon.a]
> mon addr = 192.168.0.1:7000
>
> jewel release with 1 monitor and 3 OSDs.
>
> After hours of fio rbd test, the monitor binded port(7000) was closed
> for unknown reason.

is this reproduciable? can you enable ms-debug=1 and attach the log?

> But monitor is still running,
>
> # pidof ceph-mon
> 21980
>
> Another strange thing is there are more than 1000 threads for the
> ceph-mon process.
>
> # ls /proc/21980/task/ |wc -l
> 1022
>
> Any possible reason?

are you using the simple messenger? maybe you can use pstree to check
the thread names?

>
> Thanks,
> Ming
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Regards
Kefu Chai

Re: monitor port closed

[Willem Jan Withagen]

On 23-1-2017 14:12, kefu chai wrote:
> On Mon, Jan 23, 2017 at 1:32 PM, Ming Lin <minggr@gmail.com> wrote:
>> Hi,
>>
>> [mon]
>> mon initial members = a
>>
>> [mon.a]
>> mon addr = 192.168.0.1:7000
>>
>> jewel release with 1 monitor and 3 OSDs.
>>
>> After hours of fio rbd test, the monitor binded port(7000) was closed
>> for unknown reason.
> 
> is this reproduciable? can you enable ms-debug=1 and attach the log?
> 
>> But monitor is still running,
>>
>> # pidof ceph-mon
>> 21980
>>
>> Another strange thing is there are more than 1000 threads for the
>> ceph-mon process.
>>
>> # ls /proc/21980/task/ |wc -l
>> 1022
>>
>> Any possible reason?
> 
> are you using the simple messenger? maybe you can use pstree to check
> the thread names?

Other nice tools to help you here to see where the socket has gone:
	sockstat
	lsof

They can tell you if the socket is somewhere else opened.
And what the state is.

Or they can tell you what other port(s) the monitor has opened.

Also note that recycling thru large numbers of ports does not always
goes well with the TCPstack. That is why there is the options:
     SO_REUSEADDR       enables local address reuse
     SO_REUSEPORT       enables duplicate address and port bindings

But then still you might run out of ports if too many are left in a WAIT
state.

--WjW

Re: monitor port closed

[Ming Lin]

On Mon, Jan 23, 2017 at 5:12 AM, kefu chai <tchaikov@gmail.com> wrote:
> On Mon, Jan 23, 2017 at 1:32 PM, Ming Lin <minggr@gmail.com> wrote:
>> Hi,
>>
>> [mon]
>> mon initial members = a
>>
>> [mon.a]
>> mon addr = 192.168.0.1:7000
>>
>> jewel release with 1 monitor and 3 OSDs.
>>
>> After hours of fio rbd test, the monitor binded port(7000) was closed
>> for unknown reason.
>
> is this reproduciable? can you enable ms-debug=1 and attach the log?

Will enable it and try to reproduce.

>
>> But monitor is still running,
>>
>> # pidof ceph-mon
>> 21980
>>
>> Another strange thing is there are more than 1000 threads for the
>> ceph-mon process.
>>
>> # ls /proc/21980/task/ |wc -l
>> 1022
>>
>> Any possible reason?
>
> are you using the simple messenger? maybe you can use pstree to check
> the thread names?

Yes, simple messenger.

# ceph daemon ./run/ceph-mon.a.asok config show |grep ms_type
    "ms_type": "simple"

All the thread names are "ceph-mon"

# pstree 21980
ceph-mon───1021*[{ceph-mon}]

Re: monitor port closed

[Ming Lin]

On Mon, Jan 23, 2017 at 6:28 AM, Willem Jan Withagen <wjw@digiware.nl> wrote:

>
> Other nice tools to help you here to see where the socket has gone:
>         sockstat
>         lsof
>
> They can tell you if the socket is somewhere else opened.
> And what the state is.
>
> Or they can tell you what other port(s) the monitor has opened.

# netstat -putnl |grep 7000
<no output>

# netstat -putna |grep 7000
tcp        0      0 192.168.0.1:7000        192.168.122.131:53240
ESTABLISHED 21980/ceph-mon

Seems port 7000 is not listening, but still established with rbd client.

root@rbd-client:~# netstat -putna |grep 53240
tcp        0      0 192.168.122.131:53240   192.168.0.1:7000
ESTABLISHED -

Re: monitor port closed

[Willem Jan Withagen]

On 23-1-2017 20:08, Ming Lin wrote:
> On Mon, Jan 23, 2017 at 6:28 AM, Willem Jan Withagen <wjw@digiware.nl> wrote:
> 
>>
>> Other nice tools to help you here to see where the socket has gone:
>>         sockstat
>>         lsof
>>
>> They can tell you if the socket is somewhere else opened.
>> And what the state is.
>>
>> Or they can tell you what other port(s) the monitor has opened.
> 
> # netstat -putnl |grep 7000
> <no output>

> # netstat -putna |grep 7000
> tcp        0      0 192.168.0.1:7000        192.168.122.131:53240
> ESTABLISHED 21980/ceph-mon

Is process 21980 listening on any other ports? the normal range would be
6800-7300 (as defined in config_opts.h)
And it would start on 6789, so quite a few rebinds have occurred.
unless have configured it differently.

> 
> Seems port 7000 is not listening, but still established with rbd client.
> 
> root@rbd-client:~# netstat -putna |grep 53240
> tcp        0      0 192.168.122.131:53240   192.168.0.1:7000
> ESTABLISHED -

Seems the OS thinks there is still a fully open connection, since both
ends are in ESTABLISHED.

--WjW

Re: monitor port closed

[Ming Lin]

On Mon, Jan 23, 2017 at 12:38 PM, Willem Jan Withagen <wjw@digiware.nl> wrote:
> On 23-1-2017 20:08, Ming Lin wrote:
>> On Mon, Jan 23, 2017 at 6:28 AM, Willem Jan Withagen <wjw@digiware.nl> wrote:
>>
>>>
>>> Other nice tools to help you here to see where the socket has gone:
>>>         sockstat
>>>         lsof
>>>
>>> They can tell you if the socket is somewhere else opened.
>>> And what the state is.
>>>
>>> Or they can tell you what other port(s) the monitor has opened.
>>
>> # netstat -putnl |grep 7000
>> <no output>
>
>> # netstat -putna |grep 7000
>> tcp        0      0 192.168.0.1:7000        192.168.122.131:53240
>> ESTABLISHED 21980/ceph-mon
>
> Is process 21980 listening on any other ports? the normal range would be
> 6800-7300 (as defined in config_opts.h)

No, that's the only one with ESTABLISHED state.

# netstat -putna |grep 21980
tcp        0      0 192.168.0.1:7000        192.168.122.131:53240
ESTABLISHED 21980/ceph-mon

> And it would start on 6789, so quite a few rebinds have occurred.
> unless have configured it differently.

I configured with 7000 for test purpose.

[mon]
mon initial members = a

[mon.a]
mon addr = 192.168.0.1:7000

Re: monitor port closed

[Willem Jan Withagen]

On 23-1-2017 22:01, Ming Lin wrote:
> On Mon, Jan 23, 2017 at 12:38 PM, Willem Jan Withagen <wjw@digiware.nl> wrote:
>> On 23-1-2017 20:08, Ming Lin wrote:
>>> On Mon, Jan 23, 2017 at 6:28 AM, Willem Jan Withagen <wjw@digiware.nl> wrote:
>>>
>>>>
>>>> Other nice tools to help you here to see where the socket has gone:
>>>>         sockstat
>>>>         lsof
>>>>
>>>> They can tell you if the socket is somewhere else opened.
>>>> And what the state is.
>>>>
>>>> Or they can tell you what other port(s) the monitor has opened.
>>>
>>> # netstat -putnl |grep 7000
>>> <no output>
>>
>>> # netstat -putna |grep 7000
>>> tcp        0      0 192.168.0.1:7000        192.168.122.131:53240
>>> ESTABLISHED 21980/ceph-mon
>>
>> Is process 21980 listening on any other ports? the normal range would be
>> 6800-7300 (as defined in config_opts.h)
> 
> No, that's the only one with ESTABLISHED state.
> 
> # netstat -putna |grep 21980
> tcp        0      0 192.168.0.1:7000        192.168.122.131:53240
> ESTABLISHED 21980/ceph-mon
> 
>> And it would start on 6789, so quite a few rebinds have occurred.
>> unless have configured it differently.
> 
> I configured with 7000 for test purpose.
> 
> [mon]
> mon initial members = a
> 
> [mon.a]
> mon addr = 192.168.0.1:7000
> 
Then the process is really broken somewhere, and you will have to refer
to either logfiles and or gdb-debugging.

--WjW