ceph mon_data_size_warn limits for large cluster

ceph mon_data_size_warn limits for large cluster

[M Ranga Swami Reddy]

Hello -  Are the any limits for mon_data_size for cluster with 2PB
(with 2000+ OSDs)?

Currently it set as 15G. What is logic behind this? Can we increase
when we get the mon_data_size_warn messages?

I am getting the mon_data_size_warn message even though there a ample
of free space on the disk (around 300G free disk)

Earlier thread on the same discusion:
https://www.spinics.net/lists/ceph-users/msg42456.html

Thanks
Swami

Re: ceph mon_data_size_warn limits for large cluster

[Sage Weil]

Hi Swami

The limit is somewhat arbitrary, based on cluster sizes we had seen when 
we picked it.  In your case it should be perfectly safe to increase it.

sage


On Wed, 6 Feb 2019, M Ranga Swami Reddy wrote:

> Hello -  Are the any limits for mon_data_size for cluster with 2PB
> (with 2000+ OSDs)?
> 
> Currently it set as 15G. What is logic behind this? Can we increase
> when we get the mon_data_size_warn messages?
> 
> I am getting the mon_data_size_warn message even though there a ample
> of free space on the disk (around 300G free disk)
> 
> Earlier thread on the same discusion:
> https://www.spinics.net/lists/ceph-users/msg42456.html
> 
> Thanks
> Swami
> 
> 
> 

Re: ceph mon_data_size_warn limits for large cluster

[Dan van der Ster]

Hi,

With HEALTH_OK a mon data dir should be under 2GB for even such a large cluster.

During backfilling scenarios, the mons keep old maps and grow quite
quickly. So if you have balancing, pg splitting, etc. ongoing for
awhile, the mon stores will eventually trigger that 15GB alarm.
But the intended behavior is that once the PGs are all active+clean,
the old maps should be trimmed and the disk space freed.

However, several people have noted that (at least in luminous
releases) the old maps are not trimmed until after HEALTH_OK *and* all
mons are restarted. This ticket seems related:
http://tracker.ceph.com/issues/37875

(Over here we're restarting mons every ~2-3 weeks, resulting in the
mon stores dropping from >15GB to ~700MB each time).

-- Dan


On Wed, Feb 6, 2019 at 1:26 PM Sage Weil <sage-BnTBU8nroG7k1uMJSBkQmQ@public.gmane.org> wrote:
>
> Hi Swami
>
> The limit is somewhat arbitrary, based on cluster sizes we had seen when
> we picked it.  In your case it should be perfectly safe to increase it.
>
> sage
>
>
> On Wed, 6 Feb 2019, M Ranga Swami Reddy wrote:
>
> > Hello -  Are the any limits for mon_data_size for cluster with 2PB
> > (with 2000+ OSDs)?
> >
> > Currently it set as 15G. What is logic behind this? Can we increase
> > when we get the mon_data_size_warn messages?
> >
> > I am getting the mon_data_size_warn message even though there a ample
> > of free space on the disk (around 300G free disk)
> >
> > Earlier thread on the same discusion:
> > https://www.spinics.net/lists/ceph-users/msg42456.html
> >
> > Thanks
> > Swami
> >
> >
> >
> _______________________________________________
> ceph-users mailing list
> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: ceph mon_data_size_warn limits for large cluster

[M Ranga Swami Reddy]

Hi Dan,
>During backfilling scenarios, the mons keep old maps and grow quite
>quickly. So if you have balancing, pg splitting, etc. ongoing for
>awhile, the mon stores will eventually trigger that 15GB alarm.
>But the intended behavior is that once the PGs are all active+clean,
>the old maps should be trimmed and the disk space freed.

old maps not trimmed after cluster reached to "all+clean" state for all PGs.
Is there (known) bug here?
As the size of dB showing > 15G, do I need to run the compact commands
to do the trimming?

Thanks
Swami

On Wed, Feb 6, 2019 at 6:24 PM Dan van der Ster <dan-EOCVfBHj35C+XT7JhA+gdA@public.gmane.org> wrote:
>
> Hi,
>
> With HEALTH_OK a mon data dir should be under 2GB for even such a large cluster.
>
> During backfilling scenarios, the mons keep old maps and grow quite
> quickly. So if you have balancing, pg splitting, etc. ongoing for
> awhile, the mon stores will eventually trigger that 15GB alarm.
> But the intended behavior is that once the PGs are all active+clean,
> the old maps should be trimmed and the disk space freed.
>
> However, several people have noted that (at least in luminous
> releases) the old maps are not trimmed until after HEALTH_OK *and* all
> mons are restarted. This ticket seems related:
> http://tracker.ceph.com/issues/37875
>
> (Over here we're restarting mons every ~2-3 weeks, resulting in the
> mon stores dropping from >15GB to ~700MB each time).
>
> -- Dan
>
>
> On Wed, Feb 6, 2019 at 1:26 PM Sage Weil <sage-BnTBU8nroG7k1uMJSBkQmQ@public.gmane.org> wrote:
> >
> > Hi Swami
> >
> > The limit is somewhat arbitrary, based on cluster sizes we had seen when
> > we picked it.  In your case it should be perfectly safe to increase it.
> >
> > sage
> >
> >
> > On Wed, 6 Feb 2019, M Ranga Swami Reddy wrote:
> >
> > > Hello -  Are the any limits for mon_data_size for cluster with 2PB
> > > (with 2000+ OSDs)?
> > >
> > > Currently it set as 15G. What is logic behind this? Can we increase
> > > when we get the mon_data_size_warn messages?
> > >
> > > I am getting the mon_data_size_warn message even though there a ample
> > > of free space on the disk (around 300G free disk)
> > >
> > > Earlier thread on the same discusion:
> > > https://www.spinics.net/lists/ceph-users/msg42456.html
> > >
> > > Thanks
> > > Swami
> > >
> > >
> > >
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: ceph mon_data_size_warn limits for large cluster

[M Ranga Swami Reddy]

Hi Sage

Sure, we will increase the mon_data_size to 30G to avoid this type of
warning. And currently we are using 500G disk here. I guss, which
should good enough here.

Thanks
Swami

On Wed, Feb 6, 2019 at 5:56 PM Sage Weil <sage-BnTBU8nroG7k1uMJSBkQmQ@public.gmane.org> wrote:
>
> Hi Swami
>
> The limit is somewhat arbitrary, based on cluster sizes we had seen when
> we picked it.  In your case it should be perfectly safe to increase it.
>
> sage
>
>
> On Wed, 6 Feb 2019, M Ranga Swami Reddy wrote:
>
> > Hello -  Are the any limits for mon_data_size for cluster with 2PB
> > (with 2000+ OSDs)?
> >
> > Currently it set as 15G. What is logic behind this? Can we increase
> > when we get the mon_data_size_warn messages?
> >
> > I am getting the mon_data_size_warn message even though there a ample
> > of free space on the disk (around 300G free disk)
> >
> > Earlier thread on the same discusion:
> > https://www.spinics.net/lists/ceph-users/msg42456.html
> >
> > Thanks
> > Swami
> >
> >
> >

Re: ceph mon_data_size_warn limits for large cluster

[Dan van der Ster]

On Thu, Feb 7, 2019 at 12:17 PM M Ranga Swami Reddy
<swamireddy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
> Hi Dan,
> >During backfilling scenarios, the mons keep old maps and grow quite
> >quickly. So if you have balancing, pg splitting, etc. ongoing for
> >awhile, the mon stores will eventually trigger that 15GB alarm.
> >But the intended behavior is that once the PGs are all active+clean,
> >the old maps should be trimmed and the disk space freed.
>
> old maps not trimmed after cluster reached to "all+clean" state for all PGs.
> Is there (known) bug here?
> As the size of dB showing > 15G, do I need to run the compact commands
> to do the trimming?

Compaction isn't necessary -- you should only need to restart all
peon's then the leader. A few minutes later the db's should start
trimming.

-- dan


>
> Thanks
> Swami
>
> On Wed, Feb 6, 2019 at 6:24 PM Dan van der Ster <dan-EOCVfBHj35C+XT7JhA+gdA@public.gmane.org> wrote:
> >
> > Hi,
> >
> > With HEALTH_OK a mon data dir should be under 2GB for even such a large cluster.
> >
> > During backfilling scenarios, the mons keep old maps and grow quite
> > quickly. So if you have balancing, pg splitting, etc. ongoing for
> > awhile, the mon stores will eventually trigger that 15GB alarm.
> > But the intended behavior is that once the PGs are all active+clean,
> > the old maps should be trimmed and the disk space freed.
> >
> > However, several people have noted that (at least in luminous
> > releases) the old maps are not trimmed until after HEALTH_OK *and* all
> > mons are restarted. This ticket seems related:
> > http://tracker.ceph.com/issues/37875
> >
> > (Over here we're restarting mons every ~2-3 weeks, resulting in the
> > mon stores dropping from >15GB to ~700MB each time).
> >
> > -- Dan
> >
> >
> > On Wed, Feb 6, 2019 at 1:26 PM Sage Weil <sage-BnTBU8nroG7k1uMJSBkQmQ@public.gmane.org> wrote:
> > >
> > > Hi Swami
> > >
> > > The limit is somewhat arbitrary, based on cluster sizes we had seen when
> > > we picked it.  In your case it should be perfectly safe to increase it.
> > >
> > > sage
> > >
> > >
> > > On Wed, 6 Feb 2019, M Ranga Swami Reddy wrote:
> > >
> > > > Hello -  Are the any limits for mon_data_size for cluster with 2PB
> > > > (with 2000+ OSDs)?
> > > >
> > > > Currently it set as 15G. What is logic behind this? Can we increase
> > > > when we get the mon_data_size_warn messages?
> > > >
> > > > I am getting the mon_data_size_warn message even though there a ample
> > > > of free space on the disk (around 300G free disk)
> > > >
> > > > Earlier thread on the same discusion:
> > > > https://www.spinics.net/lists/ceph-users/msg42456.html
> > > >
> > > > Thanks
> > > > Swami
> > > >
> > > >
> > > >
> > > _______________________________________________
> > > ceph-users mailing list
> > > ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: ceph mon_data_size_warn limits for large cluster

[M Ranga Swami Reddy]

>Compaction isn't necessary -- you should only need to restart all
>peon's then the leader. A few minutes later the db's should start
>trimming.

As we on production cluster, which may not be safe to restart the
ceph-mon, instead prefer to do the compact on non-leader mons.
Is this ok?

Thanks
Swami

On Thu, Feb 7, 2019 at 6:30 PM Dan van der Ster <dan-EOCVfBHj35C+XT7JhA+gdA@public.gmane.org> wrote:
>
> On Thu, Feb 7, 2019 at 12:17 PM M Ranga Swami Reddy
> <swamireddy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> >
> > Hi Dan,
> > >During backfilling scenarios, the mons keep old maps and grow quite
> > >quickly. So if you have balancing, pg splitting, etc. ongoing for
> > >awhile, the mon stores will eventually trigger that 15GB alarm.
> > >But the intended behavior is that once the PGs are all active+clean,
> > >the old maps should be trimmed and the disk space freed.
> >
> > old maps not trimmed after cluster reached to "all+clean" state for all PGs.
> > Is there (known) bug here?
> > As the size of dB showing > 15G, do I need to run the compact commands
> > to do the trimming?
>
> Compaction isn't necessary -- you should only need to restart all
> peon's then the leader. A few minutes later the db's should start
> trimming.
>
> -- dan
>
>
> >
> > Thanks
> > Swami
> >
> > On Wed, Feb 6, 2019 at 6:24 PM Dan van der Ster <dan-EOCVfBHj35C+XT7JhA+gdA@public.gmane.org> wrote:
> > >
> > > Hi,
> > >
> > > With HEALTH_OK a mon data dir should be under 2GB for even such a large cluster.
> > >
> > > During backfilling scenarios, the mons keep old maps and grow quite
> > > quickly. So if you have balancing, pg splitting, etc. ongoing for
> > > awhile, the mon stores will eventually trigger that 15GB alarm.
> > > But the intended behavior is that once the PGs are all active+clean,
> > > the old maps should be trimmed and the disk space freed.
> > >
> > > However, several people have noted that (at least in luminous
> > > releases) the old maps are not trimmed until after HEALTH_OK *and* all
> > > mons are restarted. This ticket seems related:
> > > http://tracker.ceph.com/issues/37875
> > >
> > > (Over here we're restarting mons every ~2-3 weeks, resulting in the
> > > mon stores dropping from >15GB to ~700MB each time).
> > >
> > > -- Dan
> > >
> > >
> > > On Wed, Feb 6, 2019 at 1:26 PM Sage Weil <sage-BnTBU8nroG7k1uMJSBkQmQ@public.gmane.org> wrote:
> > > >
> > > > Hi Swami
> > > >
> > > > The limit is somewhat arbitrary, based on cluster sizes we had seen when
> > > > we picked it.  In your case it should be perfectly safe to increase it.
> > > >
> > > > sage
> > > >
> > > >
> > > > On Wed, 6 Feb 2019, M Ranga Swami Reddy wrote:
> > > >
> > > > > Hello -  Are the any limits for mon_data_size for cluster with 2PB
> > > > > (with 2000+ OSDs)?
> > > > >
> > > > > Currently it set as 15G. What is logic behind this? Can we increase
> > > > > when we get the mon_data_size_warn messages?
> > > > >
> > > > > I am getting the mon_data_size_warn message even though there a ample
> > > > > of free space on the disk (around 300G free disk)
> > > > >
> > > > > Earlier thread on the same discusion:
> > > > > https://www.spinics.net/lists/ceph-users/msg42456.html
> > > > >
> > > > > Thanks
> > > > > Swami
> > > > >
> > > > >
> > > > >
> > > > _______________________________________________
> > > > ceph-users mailing list
> > > > ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: ceph mon_data_size_warn limits for large cluster

[Dan van der Ster]

On Thu, Feb 7, 2019 at 4:12 PM M Ranga Swami Reddy <swamireddy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
> >Compaction isn't necessary -- you should only need to restart all
> >peon's then the leader. A few minutes later the db's should start
> >trimming.
>
> As we on production cluster, which may not be safe to restart the
> ceph-mon, instead prefer to do the compact on non-leader mons.
> Is this ok?
>

Compaction doesn't solve this particular problem, because the maps
have not yet been deleted by the ceph-mon process.

-- dan


> Thanks
> Swami
>
> On Thu, Feb 7, 2019 at 6:30 PM Dan van der Ster <dan-EOCVfBHj35C+XT7JhA+gdA@public.gmane.org> wrote:
> >
> > On Thu, Feb 7, 2019 at 12:17 PM M Ranga Swami Reddy
> > <swamireddy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > >
> > > Hi Dan,
> > > >During backfilling scenarios, the mons keep old maps and grow quite
> > > >quickly. So if you have balancing, pg splitting, etc. ongoing for
> > > >awhile, the mon stores will eventually trigger that 15GB alarm.
> > > >But the intended behavior is that once the PGs are all active+clean,
> > > >the old maps should be trimmed and the disk space freed.
> > >
> > > old maps not trimmed after cluster reached to "all+clean" state for all PGs.
> > > Is there (known) bug here?
> > > As the size of dB showing > 15G, do I need to run the compact commands
> > > to do the trimming?
> >
> > Compaction isn't necessary -- you should only need to restart all
> > peon's then the leader. A few minutes later the db's should start
> > trimming.
> >
> > -- dan
> >
> >
> > >
> > > Thanks
> > > Swami
> > >
> > > On Wed, Feb 6, 2019 at 6:24 PM Dan van der Ster <dan-EOCVfBHj35C+XT7JhA+gdA@public.gmane.org> wrote:
> > > >
> > > > Hi,
> > > >
> > > > With HEALTH_OK a mon data dir should be under 2GB for even such a large cluster.
> > > >
> > > > During backfilling scenarios, the mons keep old maps and grow quite
> > > > quickly. So if you have balancing, pg splitting, etc. ongoing for
> > > > awhile, the mon stores will eventually trigger that 15GB alarm.
> > > > But the intended behavior is that once the PGs are all active+clean,
> > > > the old maps should be trimmed and the disk space freed.
> > > >
> > > > However, several people have noted that (at least in luminous
> > > > releases) the old maps are not trimmed until after HEALTH_OK *and* all
> > > > mons are restarted. This ticket seems related:
> > > > http://tracker.ceph.com/issues/37875
> > > >
> > > > (Over here we're restarting mons every ~2-3 weeks, resulting in the
> > > > mon stores dropping from >15GB to ~700MB each time).
> > > >
> > > > -- Dan
> > > >
> > > >
> > > > On Wed, Feb 6, 2019 at 1:26 PM Sage Weil <sage-BnTBU8nroG7k1uMJSBkQmQ@public.gmane.org> wrote:
> > > > >
> > > > > Hi Swami
> > > > >
> > > > > The limit is somewhat arbitrary, based on cluster sizes we had seen when
> > > > > we picked it.  In your case it should be perfectly safe to increase it.
> > > > >
> > > > > sage
> > > > >
> > > > >
> > > > > On Wed, 6 Feb 2019, M Ranga Swami Reddy wrote:
> > > > >
> > > > > > Hello -  Are the any limits for mon_data_size for cluster with 2PB
> > > > > > (with 2000+ OSDs)?
> > > > > >
> > > > > > Currently it set as 15G. What is logic behind this? Can we increase
> > > > > > when we get the mon_data_size_warn messages?
> > > > > >
> > > > > > I am getting the mon_data_size_warn message even though there a ample
> > > > > > of free space on the disk (around 300G free disk)
> > > > > >
> > > > > > Earlier thread on the same discusion:
> > > > > > https://www.spinics.net/lists/ceph-users/msg42456.html
> > > > > >
> > > > > > Thanks
> > > > > > Swami
> > > > > >
> > > > > >
> > > > > >
> > > > > _______________________________________________
> > > > > ceph-users mailing list
> > > > > ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: ceph mon_data_size_warn limits for large cluster

[M Ranga Swami Reddy]

Alternatively, will increase the mon_data_size to 30G (from 15G)..

Thanks
Swami

On Thu, Feb 7, 2019 at 8:44 PM Dan van der Ster <dan-EOCVfBHj35C+XT7JhA+gdA@public.gmane.org> wrote:
>
> On Thu, Feb 7, 2019 at 4:12 PM M Ranga Swami Reddy <swamireddy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> >
> > >Compaction isn't necessary -- you should only need to restart all
> > >peon's then the leader. A few minutes later the db's should start
> > >trimming.
> >
> > As we on production cluster, which may not be safe to restart the
> > ceph-mon, instead prefer to do the compact on non-leader mons.
> > Is this ok?
> >
>
> Compaction doesn't solve this particular problem, because the maps
> have not yet been deleted by the ceph-mon process.
>
> -- dan
>
>
> > Thanks
> > Swami
> >
> > On Thu, Feb 7, 2019 at 6:30 PM Dan van der Ster <dan-EOCVfBHj35C+XT7JhA+gdA@public.gmane.org> wrote:
> > >
> > > On Thu, Feb 7, 2019 at 12:17 PM M Ranga Swami Reddy
> > > <swamireddy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > > >
> > > > Hi Dan,
> > > > >During backfilling scenarios, the mons keep old maps and grow quite
> > > > >quickly. So if you have balancing, pg splitting, etc. ongoing for
> > > > >awhile, the mon stores will eventually trigger that 15GB alarm.
> > > > >But the intended behavior is that once the PGs are all active+clean,
> > > > >the old maps should be trimmed and the disk space freed.
> > > >
> > > > old maps not trimmed after cluster reached to "all+clean" state for all PGs.
> > > > Is there (known) bug here?
> > > > As the size of dB showing > 15G, do I need to run the compact commands
> > > > to do the trimming?
> > >
> > > Compaction isn't necessary -- you should only need to restart all
> > > peon's then the leader. A few minutes later the db's should start
> > > trimming.
> > >
> > > -- dan
> > >
> > >
> > > >
> > > > Thanks
> > > > Swami
> > > >
> > > > On Wed, Feb 6, 2019 at 6:24 PM Dan van der Ster <dan-EOCVfBHj35C+XT7JhA+gdA@public.gmane.org> wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > With HEALTH_OK a mon data dir should be under 2GB for even such a large cluster.
> > > > >
> > > > > During backfilling scenarios, the mons keep old maps and grow quite
> > > > > quickly. So if you have balancing, pg splitting, etc. ongoing for
> > > > > awhile, the mon stores will eventually trigger that 15GB alarm.
> > > > > But the intended behavior is that once the PGs are all active+clean,
> > > > > the old maps should be trimmed and the disk space freed.
> > > > >
> > > > > However, several people have noted that (at least in luminous
> > > > > releases) the old maps are not trimmed until after HEALTH_OK *and* all
> > > > > mons are restarted. This ticket seems related:
> > > > > http://tracker.ceph.com/issues/37875
> > > > >
> > > > > (Over here we're restarting mons every ~2-3 weeks, resulting in the
> > > > > mon stores dropping from >15GB to ~700MB each time).
> > > > >
> > > > > -- Dan
> > > > >
> > > > >
> > > > > On Wed, Feb 6, 2019 at 1:26 PM Sage Weil <sage-BnTBU8nroG7k1uMJSBkQmQ@public.gmane.org> wrote:
> > > > > >
> > > > > > Hi Swami
> > > > > >
> > > > > > The limit is somewhat arbitrary, based on cluster sizes we had seen when
> > > > > > we picked it.  In your case it should be perfectly safe to increase it.
> > > > > >
> > > > > > sage
> > > > > >
> > > > > >
> > > > > > On Wed, 6 Feb 2019, M Ranga Swami Reddy wrote:
> > > > > >
> > > > > > > Hello -  Are the any limits for mon_data_size for cluster with 2PB
> > > > > > > (with 2000+ OSDs)?
> > > > > > >
> > > > > > > Currently it set as 15G. What is logic behind this? Can we increase
> > > > > > > when we get the mon_data_size_warn messages?
> > > > > > >
> > > > > > > I am getting the mon_data_size_warn message even though there a ample
> > > > > > > of free space on the disk (around 300G free disk)
> > > > > > >
> > > > > > > Earlier thread on the same discusion:
> > > > > > > https://www.spinics.net/lists/ceph-users/msg42456.html
> > > > > > >
> > > > > > > Thanks
> > > > > > > Swami
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > _______________________________________________
> > > > > > ceph-users mailing list
> > > > > > ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> > > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: ceph mon_data_size_warn limits for large cluster

[Sage Weil]

On Thu, 7 Feb 2019, Dan van der Ster wrote:
> On Thu, Feb 7, 2019 at 12:17 PM M Ranga Swami Reddy
> <swamireddy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> >
> > Hi Dan,
> > >During backfilling scenarios, the mons keep old maps and grow quite
> > >quickly. So if you have balancing, pg splitting, etc. ongoing for
> > >awhile, the mon stores will eventually trigger that 15GB alarm.
> > >But the intended behavior is that once the PGs are all active+clean,
> > >the old maps should be trimmed and the disk space freed.
> >
> > old maps not trimmed after cluster reached to "all+clean" state for all PGs.
> > Is there (known) bug here?
> > As the size of dB showing > 15G, do I need to run the compact commands
> > to do the trimming?
> 
> Compaction isn't necessary -- you should only need to restart all
> peon's then the leader. A few minutes later the db's should start
> trimming.

The next time someone sees this behavior, can you please

- enable debug_mon = 20 on all mons (*before* restarting)
   ceph tell mon.* injectargs '--debug-mon 20'
- wait for 10 minutes or so to generate some logs
- add 'debug mon = 20' to ceph.conf (on mons only)
- restart the monitors
- wait for them to start trimming
- remove 'debug mon = 20' from ceph.conf (on mons only)
- tar up the log files, ceph-post-file them, and share them with ticket 
http://tracker.ceph.com/issues/38322

Thanks!
sage



 
> -- dan
> 
> 
> >
> > Thanks
> > Swami
> >
> > On Wed, Feb 6, 2019 at 6:24 PM Dan van der Ster <dan-EOCVfBHj35C+XT7JhA+gdA@public.gmane.org> wrote:
> > >
> > > Hi,
> > >
> > > With HEALTH_OK a mon data dir should be under 2GB for even such a large cluster.
> > >
> > > During backfilling scenarios, the mons keep old maps and grow quite
> > > quickly. So if you have balancing, pg splitting, etc. ongoing for
> > > awhile, the mon stores will eventually trigger that 15GB alarm.
> > > But the intended behavior is that once the PGs are all active+clean,
> > > the old maps should be trimmed and the disk space freed.
> > >
> > > However, several people have noted that (at least in luminous
> > > releases) the old maps are not trimmed until after HEALTH_OK *and* all
> > > mons are restarted. This ticket seems related:
> > > http://tracker.ceph.com/issues/37875
> > >
> > > (Over here we're restarting mons every ~2-3 weeks, resulting in the
> > > mon stores dropping from >15GB to ~700MB each time).
> > >
> > > -- Dan
> > >
> > >
> > > On Wed, Feb 6, 2019 at 1:26 PM Sage Weil <sage-BnTBU8nroG7k1uMJSBkQmQ@public.gmane.org> wrote:
> > > >
> > > > Hi Swami
> > > >
> > > > The limit is somewhat arbitrary, based on cluster sizes we had seen when
> > > > we picked it.  In your case it should be perfectly safe to increase it.
> > > >
> > > > sage
> > > >
> > > >
> > > > On Wed, 6 Feb 2019, M Ranga Swami Reddy wrote:
> > > >
> > > > > Hello -  Are the any limits for mon_data_size for cluster with 2PB
> > > > > (with 2000+ OSDs)?
> > > > >
> > > > > Currently it set as 15G. What is logic behind this? Can we increase
> > > > > when we get the mon_data_size_warn messages?
> > > > >
> > > > > I am getting the mon_data_size_warn message even though there a ample
> > > > > of free space on the disk (around 300G free disk)
> > > > >
> > > > > Earlier thread on the same discusion:
> > > > > https://www.spinics.net/lists/ceph-users/msg42456.html
> > > > >
> > > > > Thanks
> > > > > Swami
> > > > >
> > > > >
> > > > >
> > > > _______________________________________________
> > > > ceph-users mailing list
> > > > ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 
> 

Re: ceph mon_data_size_warn limits for large cluster

[M Ranga Swami Reddy]

Sure, will this. For now I have creased the size to 30G (from 15G).

On Thu, Feb 14, 2019 at 7:39 PM Sage Weil <sage-BnTBU8nroG7k1uMJSBkQmQ@public.gmane.org> wrote:
>
> On Thu, 7 Feb 2019, Dan van der Ster wrote:
> > On Thu, Feb 7, 2019 at 12:17 PM M Ranga Swami Reddy
> > <swamireddy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > >
> > > Hi Dan,
> > > >During backfilling scenarios, the mons keep old maps and grow quite
> > > >quickly. So if you have balancing, pg splitting, etc. ongoing for
> > > >awhile, the mon stores will eventually trigger that 15GB alarm.
> > > >But the intended behavior is that once the PGs are all active+clean,
> > > >the old maps should be trimmed and the disk space freed.
> > >
> > > old maps not trimmed after cluster reached to "all+clean" state for all PGs.
> > > Is there (known) bug here?
> > > As the size of dB showing > 15G, do I need to run the compact commands
> > > to do the trimming?
> >
> > Compaction isn't necessary -- you should only need to restart all
> > peon's then the leader. A few minutes later the db's should start
> > trimming.
>
> The next time someone sees this behavior, can you please
>
> - enable debug_mon = 20 on all mons (*before* restarting)
>    ceph tell mon.* injectargs '--debug-mon 20'
> - wait for 10 minutes or so to generate some logs
> - add 'debug mon = 20' to ceph.conf (on mons only)
> - restart the monitors
> - wait for them to start trimming
> - remove 'debug mon = 20' from ceph.conf (on mons only)
> - tar up the log files, ceph-post-file them, and share them with ticket
> http://tracker.ceph.com/issues/38322
>
> Thanks!
> sage
>
>
>
>
> > -- dan
> >
> >
> > >
> > > Thanks
> > > Swami
> > >
> > > On Wed, Feb 6, 2019 at 6:24 PM Dan van der Ster <dan-EOCVfBHj35C+XT7JhA+gdA@public.gmane.org> wrote:
> > > >
> > > > Hi,
> > > >
> > > > With HEALTH_OK a mon data dir should be under 2GB for even such a large cluster.
> > > >
> > > > During backfilling scenarios, the mons keep old maps and grow quite
> > > > quickly. So if you have balancing, pg splitting, etc. ongoing for
> > > > awhile, the mon stores will eventually trigger that 15GB alarm.
> > > > But the intended behavior is that once the PGs are all active+clean,
> > > > the old maps should be trimmed and the disk space freed.
> > > >
> > > > However, several people have noted that (at least in luminous
> > > > releases) the old maps are not trimmed until after HEALTH_OK *and* all
> > > > mons are restarted. This ticket seems related:
> > > > http://tracker.ceph.com/issues/37875
> > > >
> > > > (Over here we're restarting mons every ~2-3 weeks, resulting in the
> > > > mon stores dropping from >15GB to ~700MB each time).
> > > >
> > > > -- Dan
> > > >
> > > >
> > > > On Wed, Feb 6, 2019 at 1:26 PM Sage Weil <sage-BnTBU8nroG7k1uMJSBkQmQ@public.gmane.org> wrote:
> > > > >
> > > > > Hi Swami
> > > > >
> > > > > The limit is somewhat arbitrary, based on cluster sizes we had seen when
> > > > > we picked it.  In your case it should be perfectly safe to increase it.
> > > > >
> > > > > sage
> > > > >
> > > > >
> > > > > On Wed, 6 Feb 2019, M Ranga Swami Reddy wrote:
> > > > >
> > > > > > Hello -  Are the any limits for mon_data_size for cluster with 2PB
> > > > > > (with 2000+ OSDs)?
> > > > > >
> > > > > > Currently it set as 15G. What is logic behind this? Can we increase
> > > > > > when we get the mon_data_size_warn messages?
> > > > > >
> > > > > > I am getting the mon_data_size_warn message even though there a ample
> > > > > > of free space on the disk (around 300G free disk)
> > > > > >
> > > > > > Earlier thread on the same discusion:
> > > > > > https://www.spinics.net/lists/ceph-users/msg42456.html
> > > > > >
> > > > > > Thanks
> > > > > > Swami
> > > > > >
> > > > > >
> > > > > >
> > > > > _______________________________________________
> > > > > ceph-users mailing list
> > > > > ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> >
> >

Re: ceph mon_data_size_warn limits for large cluster

[M Ranga Swami Reddy]

today I again hit the warn with 30G also...

On Thu, Feb 14, 2019 at 7:39 PM Sage Weil <sage-BnTBU8nroG7k1uMJSBkQmQ@public.gmane.org> wrote:
>
> On Thu, 7 Feb 2019, Dan van der Ster wrote:
> > On Thu, Feb 7, 2019 at 12:17 PM M Ranga Swami Reddy
> > <swamireddy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > >
> > > Hi Dan,
> > > >During backfilling scenarios, the mons keep old maps and grow quite
> > > >quickly. So if you have balancing, pg splitting, etc. ongoing for
> > > >awhile, the mon stores will eventually trigger that 15GB alarm.
> > > >But the intended behavior is that once the PGs are all active+clean,
> > > >the old maps should be trimmed and the disk space freed.
> > >
> > > old maps not trimmed after cluster reached to "all+clean" state for all PGs.
> > > Is there (known) bug here?
> > > As the size of dB showing > 15G, do I need to run the compact commands
> > > to do the trimming?
> >
> > Compaction isn't necessary -- you should only need to restart all
> > peon's then the leader. A few minutes later the db's should start
> > trimming.
>
> The next time someone sees this behavior, can you please
>
> - enable debug_mon = 20 on all mons (*before* restarting)
>    ceph tell mon.* injectargs '--debug-mon 20'
> - wait for 10 minutes or so to generate some logs
> - add 'debug mon = 20' to ceph.conf (on mons only)
> - restart the monitors
> - wait for them to start trimming
> - remove 'debug mon = 20' from ceph.conf (on mons only)
> - tar up the log files, ceph-post-file them, and share them with ticket
> http://tracker.ceph.com/issues/38322
>
> Thanks!
> sage
>
>
>
>
> > -- dan
> >
> >
> > >
> > > Thanks
> > > Swami
> > >
> > > On Wed, Feb 6, 2019 at 6:24 PM Dan van der Ster <dan-EOCVfBHj35C+XT7JhA+gdA@public.gmane.org> wrote:
> > > >
> > > > Hi,
> > > >
> > > > With HEALTH_OK a mon data dir should be under 2GB for even such a large cluster.
> > > >
> > > > During backfilling scenarios, the mons keep old maps and grow quite
> > > > quickly. So if you have balancing, pg splitting, etc. ongoing for
> > > > awhile, the mon stores will eventually trigger that 15GB alarm.
> > > > But the intended behavior is that once the PGs are all active+clean,
> > > > the old maps should be trimmed and the disk space freed.
> > > >
> > > > However, several people have noted that (at least in luminous
> > > > releases) the old maps are not trimmed until after HEALTH_OK *and* all
> > > > mons are restarted. This ticket seems related:
> > > > http://tracker.ceph.com/issues/37875
> > > >
> > > > (Over here we're restarting mons every ~2-3 weeks, resulting in the
> > > > mon stores dropping from >15GB to ~700MB each time).
> > > >
> > > > -- Dan
> > > >
> > > >
> > > > On Wed, Feb 6, 2019 at 1:26 PM Sage Weil <sage-BnTBU8nroG7k1uMJSBkQmQ@public.gmane.org> wrote:
> > > > >
> > > > > Hi Swami
> > > > >
> > > > > The limit is somewhat arbitrary, based on cluster sizes we had seen when
> > > > > we picked it.  In your case it should be perfectly safe to increase it.
> > > > >
> > > > > sage
> > > > >
> > > > >
> > > > > On Wed, 6 Feb 2019, M Ranga Swami Reddy wrote:
> > > > >
> > > > > > Hello -  Are the any limits for mon_data_size for cluster with 2PB
> > > > > > (with 2000+ OSDs)?
> > > > > >
> > > > > > Currently it set as 15G. What is logic behind this? Can we increase
> > > > > > when we get the mon_data_size_warn messages?
> > > > > >
> > > > > > I am getting the mon_data_size_warn message even though there a ample
> > > > > > of free space on the disk (around 300G free disk)
> > > > > >
> > > > > > Earlier thread on the same discusion:
> > > > > > https://www.spinics.net/lists/ceph-users/msg42456.html
> > > > > >
> > > > > > Thanks
> > > > > > Swami
> > > > > >
> > > > > >
> > > > > >
> > > > > _______________________________________________
> > > > > ceph-users mailing list
> > > > > ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> >
> >

Re: ceph mon_data_size_warn limits for large cluster

[M Ranga Swami Reddy]

Hi Sage - If the mon data increases, is this impacts the ceph cluster
performance (ie on ceph osd bench, etc)?

On Fri, Feb 15, 2019 at 3:13 PM M Ranga Swami Reddy
<swamireddy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
> today I again hit the warn with 30G also...
>
> On Thu, Feb 14, 2019 at 7:39 PM Sage Weil <sage-BnTBU8nroG7k1uMJSBkQmQ@public.gmane.org> wrote:
> >
> > On Thu, 7 Feb 2019, Dan van der Ster wrote:
> > > On Thu, Feb 7, 2019 at 12:17 PM M Ranga Swami Reddy
> > > <swamireddy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > > >
> > > > Hi Dan,
> > > > >During backfilling scenarios, the mons keep old maps and grow quite
> > > > >quickly. So if you have balancing, pg splitting, etc. ongoing for
> > > > >awhile, the mon stores will eventually trigger that 15GB alarm.
> > > > >But the intended behavior is that once the PGs are all active+clean,
> > > > >the old maps should be trimmed and the disk space freed.
> > > >
> > > > old maps not trimmed after cluster reached to "all+clean" state for all PGs.
> > > > Is there (known) bug here?
> > > > As the size of dB showing > 15G, do I need to run the compact commands
> > > > to do the trimming?
> > >
> > > Compaction isn't necessary -- you should only need to restart all
> > > peon's then the leader. A few minutes later the db's should start
> > > trimming.
> >
> > The next time someone sees this behavior, can you please
> >
> > - enable debug_mon = 20 on all mons (*before* restarting)
> >    ceph tell mon.* injectargs '--debug-mon 20'
> > - wait for 10 minutes or so to generate some logs
> > - add 'debug mon = 20' to ceph.conf (on mons only)
> > - restart the monitors
> > - wait for them to start trimming
> > - remove 'debug mon = 20' from ceph.conf (on mons only)
> > - tar up the log files, ceph-post-file them, and share them with ticket
> > http://tracker.ceph.com/issues/38322
> >
> > Thanks!
> > sage
> >
> >
> >
> >
> > > -- dan
> > >
> > >
> > > >
> > > > Thanks
> > > > Swami
> > > >
> > > > On Wed, Feb 6, 2019 at 6:24 PM Dan van der Ster <dan-EOCVfBHj35C+XT7JhA+gdA@public.gmane.org> wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > With HEALTH_OK a mon data dir should be under 2GB for even such a large cluster.
> > > > >
> > > > > During backfilling scenarios, the mons keep old maps and grow quite
> > > > > quickly. So if you have balancing, pg splitting, etc. ongoing for
> > > > > awhile, the mon stores will eventually trigger that 15GB alarm.
> > > > > But the intended behavior is that once the PGs are all active+clean,
> > > > > the old maps should be trimmed and the disk space freed.
> > > > >
> > > > > However, several people have noted that (at least in luminous
> > > > > releases) the old maps are not trimmed until after HEALTH_OK *and* all
> > > > > mons are restarted. This ticket seems related:
> > > > > http://tracker.ceph.com/issues/37875
> > > > >
> > > > > (Over here we're restarting mons every ~2-3 weeks, resulting in the
> > > > > mon stores dropping from >15GB to ~700MB each time).
> > > > >
> > > > > -- Dan
> > > > >
> > > > >
> > > > > On Wed, Feb 6, 2019 at 1:26 PM Sage Weil <sage-BnTBU8nroG7k1uMJSBkQmQ@public.gmane.org> wrote:
> > > > > >
> > > > > > Hi Swami
> > > > > >
> > > > > > The limit is somewhat arbitrary, based on cluster sizes we had seen when
> > > > > > we picked it.  In your case it should be perfectly safe to increase it.
> > > > > >
> > > > > > sage
> > > > > >
> > > > > >
> > > > > > On Wed, 6 Feb 2019, M Ranga Swami Reddy wrote:
> > > > > >
> > > > > > > Hello -  Are the any limits for mon_data_size for cluster with 2PB
> > > > > > > (with 2000+ OSDs)?
> > > > > > >
> > > > > > > Currently it set as 15G. What is logic behind this? Can we increase
> > > > > > > when we get the mon_data_size_warn messages?
> > > > > > >
> > > > > > > I am getting the mon_data_size_warn message even though there a ample
> > > > > > > of free space on the disk (around 300G free disk)
> > > > > > >
> > > > > > > Earlier thread on the same discusion:
> > > > > > > https://www.spinics.net/lists/ceph-users/msg42456.html
> > > > > > >
> > > > > > > Thanks
> > > > > > > Swami
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > _______________________________________________
> > > > > > ceph-users mailing list
> > > > > > ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> > > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > >
> > >
> > >

Re: ceph mon_data_size_warn limits for large cluster

[Dan van der Ster]

On Thu, Feb 14, 2019 at 2:31 PM Sage Weil <sage-BnTBU8nroG7k1uMJSBkQmQ@public.gmane.org> wrote:
>
> On Thu, 7 Feb 2019, Dan van der Ster wrote:
> > On Thu, Feb 7, 2019 at 12:17 PM M Ranga Swami Reddy
> > <swamireddy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > >
> > > Hi Dan,
> > > >During backfilling scenarios, the mons keep old maps and grow quite
> > > >quickly. So if you have balancing, pg splitting, etc. ongoing for
> > > >awhile, the mon stores will eventually trigger that 15GB alarm.
> > > >But the intended behavior is that once the PGs are all active+clean,
> > > >the old maps should be trimmed and the disk space freed.
> > >
> > > old maps not trimmed after cluster reached to "all+clean" state for all PGs.
> > > Is there (known) bug here?
> > > As the size of dB showing > 15G, do I need to run the compact commands
> > > to do the trimming?
> >
> > Compaction isn't necessary -- you should only need to restart all
> > peon's then the leader. A few minutes later the db's should start
> > trimming.
>
> The next time someone sees this behavior, can you please
>
> - enable debug_mon = 20 on all mons (*before* restarting)
>    ceph tell mon.* injectargs '--debug-mon 20'
> - wait for 10 minutes or so to generate some logs
> - add 'debug mon = 20' to ceph.conf (on mons only)
> - restart the monitors
> - wait for them to start trimming
> - remove 'debug mon = 20' from ceph.conf (on mons only)
> - tar up the log files, ceph-post-file them, and share them with ticket
> http://tracker.ceph.com/issues/38322
>

Not sure if you noticed, but we sent some logs Friday.

-- dan

> Thanks!
> sage
>
>
>
>
> > -- dan
> >
> >
> > >
> > > Thanks
> > > Swami
> > >
> > > On Wed, Feb 6, 2019 at 6:24 PM Dan van der Ster <dan-EOCVfBHj35C+XT7JhA+gdA@public.gmane.org> wrote:
> > > >
> > > > Hi,
> > > >
> > > > With HEALTH_OK a mon data dir should be under 2GB for even such a large cluster.
> > > >
> > > > During backfilling scenarios, the mons keep old maps and grow quite
> > > > quickly. So if you have balancing, pg splitting, etc. ongoing for
> > > > awhile, the mon stores will eventually trigger that 15GB alarm.
> > > > But the intended behavior is that once the PGs are all active+clean,
> > > > the old maps should be trimmed and the disk space freed.
> > > >
> > > > However, several people have noted that (at least in luminous
> > > > releases) the old maps are not trimmed until after HEALTH_OK *and* all
> > > > mons are restarted. This ticket seems related:
> > > > http://tracker.ceph.com/issues/37875
> > > >
> > > > (Over here we're restarting mons every ~2-3 weeks, resulting in the
> > > > mon stores dropping from >15GB to ~700MB each time).
> > > >
> > > > -- Dan
> > > >
> > > >
> > > > On Wed, Feb 6, 2019 at 1:26 PM Sage Weil <sage-BnTBU8nroG7k1uMJSBkQmQ@public.gmane.org> wrote:
> > > > >
> > > > > Hi Swami
> > > > >
> > > > > The limit is somewhat arbitrary, based on cluster sizes we had seen when
> > > > > we picked it.  In your case it should be perfectly safe to increase it.
> > > > >
> > > > > sage
> > > > >
> > > > >
> > > > > On Wed, 6 Feb 2019, M Ranga Swami Reddy wrote:
> > > > >
> > > > > > Hello -  Are the any limits for mon_data_size for cluster with 2PB
> > > > > > (with 2000+ OSDs)?
> > > > > >
> > > > > > Currently it set as 15G. What is logic behind this? Can we increase
> > > > > > when we get the mon_data_size_warn messages?
> > > > > >
> > > > > > I am getting the mon_data_size_warn message even though there a ample
> > > > > > of free space on the disk (around 300G free disk)
> > > > > >
> > > > > > Earlier thread on the same discusion:
> > > > > > https://www.spinics.net/lists/ceph-users/msg42456.html
> > > > > >
> > > > > > Thanks
> > > > > > Swami
> > > > > >
> > > > > >
> > > > > >
> > > > > _______________________________________________
> > > > > ceph-users mailing list
> > > > > ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> >
> >

Re: ceph mon_data_size_warn limits for large cluster

[Dan van der Ster]

Not really.

You should just restart your mons though -- if done one at a time it
has zero impact on your clients.

-- dan


On Mon, Feb 18, 2019 at 12:11 PM M Ranga Swami Reddy
<swamireddy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
> Hi Sage - If the mon data increases, is this impacts the ceph cluster
> performance (ie on ceph osd bench, etc)?
>
> On Fri, Feb 15, 2019 at 3:13 PM M Ranga Swami Reddy
> <swamireddy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> >
> > today I again hit the warn with 30G also...
> >
> > On Thu, Feb 14, 2019 at 7:39 PM Sage Weil <sage-BnTBU8nroG7k1uMJSBkQmQ@public.gmane.org> wrote:
> > >
> > > On Thu, 7 Feb 2019, Dan van der Ster wrote:
> > > > On Thu, Feb 7, 2019 at 12:17 PM M Ranga Swami Reddy
> > > > <swamireddy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > > > >
> > > > > Hi Dan,
> > > > > >During backfilling scenarios, the mons keep old maps and grow quite
> > > > > >quickly. So if you have balancing, pg splitting, etc. ongoing for
> > > > > >awhile, the mon stores will eventually trigger that 15GB alarm.
> > > > > >But the intended behavior is that once the PGs are all active+clean,
> > > > > >the old maps should be trimmed and the disk space freed.
> > > > >
> > > > > old maps not trimmed after cluster reached to "all+clean" state for all PGs.
> > > > > Is there (known) bug here?
> > > > > As the size of dB showing > 15G, do I need to run the compact commands
> > > > > to do the trimming?
> > > >
> > > > Compaction isn't necessary -- you should only need to restart all
> > > > peon's then the leader. A few minutes later the db's should start
> > > > trimming.
> > >
> > > The next time someone sees this behavior, can you please
> > >
> > > - enable debug_mon = 20 on all mons (*before* restarting)
> > >    ceph tell mon.* injectargs '--debug-mon 20'
> > > - wait for 10 minutes or so to generate some logs
> > > - add 'debug mon = 20' to ceph.conf (on mons only)
> > > - restart the monitors
> > > - wait for them to start trimming
> > > - remove 'debug mon = 20' from ceph.conf (on mons only)
> > > - tar up the log files, ceph-post-file them, and share them with ticket
> > > http://tracker.ceph.com/issues/38322
> > >
> > > Thanks!
> > > sage
> > >
> > >
> > >
> > >
> > > > -- dan
> > > >
> > > >
> > > > >
> > > > > Thanks
> > > > > Swami
> > > > >
> > > > > On Wed, Feb 6, 2019 at 6:24 PM Dan van der Ster <dan-EOCVfBHj35C+XT7JhA+gdA@public.gmane.org> wrote:
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > With HEALTH_OK a mon data dir should be under 2GB for even such a large cluster.
> > > > > >
> > > > > > During backfilling scenarios, the mons keep old maps and grow quite
> > > > > > quickly. So if you have balancing, pg splitting, etc. ongoing for
> > > > > > awhile, the mon stores will eventually trigger that 15GB alarm.
> > > > > > But the intended behavior is that once the PGs are all active+clean,
> > > > > > the old maps should be trimmed and the disk space freed.
> > > > > >
> > > > > > However, several people have noted that (at least in luminous
> > > > > > releases) the old maps are not trimmed until after HEALTH_OK *and* all
> > > > > > mons are restarted. This ticket seems related:
> > > > > > http://tracker.ceph.com/issues/37875
> > > > > >
> > > > > > (Over here we're restarting mons every ~2-3 weeks, resulting in the
> > > > > > mon stores dropping from >15GB to ~700MB each time).
> > > > > >
> > > > > > -- Dan
> > > > > >
> > > > > >
> > > > > > On Wed, Feb 6, 2019 at 1:26 PM Sage Weil <sage-BnTBU8nroG7k1uMJSBkQmQ@public.gmane.org> wrote:
> > > > > > >
> > > > > > > Hi Swami
> > > > > > >
> > > > > > > The limit is somewhat arbitrary, based on cluster sizes we had seen when
> > > > > > > we picked it.  In your case it should be perfectly safe to increase it.
> > > > > > >
> > > > > > > sage
> > > > > > >
> > > > > > >
> > > > > > > On Wed, 6 Feb 2019, M Ranga Swami Reddy wrote:
> > > > > > >
> > > > > > > > Hello -  Are the any limits for mon_data_size for cluster with 2PB
> > > > > > > > (with 2000+ OSDs)?
> > > > > > > >
> > > > > > > > Currently it set as 15G. What is logic behind this? Can we increase
> > > > > > > > when we get the mon_data_size_warn messages?
> > > > > > > >
> > > > > > > > I am getting the mon_data_size_warn message even though there a ample
> > > > > > > > of free space on the disk (around 300G free disk)
> > > > > > > >
> > > > > > > > Earlier thread on the same discusion:
> > > > > > > > https://www.spinics.net/lists/ceph-users/msg42456.html
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > > Swami
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > _______________________________________________
> > > > > > > ceph-users mailing list
> > > > > > > ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> > > > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > > >
> > > >
> > > >

Re: ceph mon_data_size_warn limits for large cluster

[M Ranga Swami Reddy]

OK, sure will restart the ceph-mon (starting from non leader first,
and then last leader node).


On Mon, Feb 18, 2019 at 4:59 PM Dan van der Ster <dan-EOCVfBHj35C+XT7JhA+gdA@public.gmane.org> wrote:
>
> Not really.
>
> You should just restart your mons though -- if done one at a time it
> has zero impact on your clients.
>
> -- dan
>
>
> On Mon, Feb 18, 2019 at 12:11 PM M Ranga Swami Reddy
> <swamireddy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> >
> > Hi Sage - If the mon data increases, is this impacts the ceph cluster
> > performance (ie on ceph osd bench, etc)?
> >
> > On Fri, Feb 15, 2019 at 3:13 PM M Ranga Swami Reddy
> > <swamireddy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > >
> > > today I again hit the warn with 30G also...
> > >
> > > On Thu, Feb 14, 2019 at 7:39 PM Sage Weil <sage-BnTBU8nroG7k1uMJSBkQmQ@public.gmane.org> wrote:
> > > >
> > > > On Thu, 7 Feb 2019, Dan van der Ster wrote:
> > > > > On Thu, Feb 7, 2019 at 12:17 PM M Ranga Swami Reddy
> > > > > <swamireddy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > > > > >
> > > > > > Hi Dan,
> > > > > > >During backfilling scenarios, the mons keep old maps and grow quite
> > > > > > >quickly. So if you have balancing, pg splitting, etc. ongoing for
> > > > > > >awhile, the mon stores will eventually trigger that 15GB alarm.
> > > > > > >But the intended behavior is that once the PGs are all active+clean,
> > > > > > >the old maps should be trimmed and the disk space freed.
> > > > > >
> > > > > > old maps not trimmed after cluster reached to "all+clean" state for all PGs.
> > > > > > Is there (known) bug here?
> > > > > > As the size of dB showing > 15G, do I need to run the compact commands
> > > > > > to do the trimming?
> > > > >
> > > > > Compaction isn't necessary -- you should only need to restart all
> > > > > peon's then the leader. A few minutes later the db's should start
> > > > > trimming.
> > > >
> > > > The next time someone sees this behavior, can you please
> > > >
> > > > - enable debug_mon = 20 on all mons (*before* restarting)
> > > >    ceph tell mon.* injectargs '--debug-mon 20'
> > > > - wait for 10 minutes or so to generate some logs
> > > > - add 'debug mon = 20' to ceph.conf (on mons only)
> > > > - restart the monitors
> > > > - wait for them to start trimming
> > > > - remove 'debug mon = 20' from ceph.conf (on mons only)
> > > > - tar up the log files, ceph-post-file them, and share them with ticket
> > > > http://tracker.ceph.com/issues/38322
> > > >
> > > > Thanks!
> > > > sage
> > > >
> > > >
> > > >
> > > >
> > > > > -- dan
> > > > >
> > > > >
> > > > > >
> > > > > > Thanks
> > > > > > Swami
> > > > > >
> > > > > > On Wed, Feb 6, 2019 at 6:24 PM Dan van der Ster <dan-EOCVfBHj35C+XT7JhA+gdA@public.gmane.org> wrote:
> > > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > With HEALTH_OK a mon data dir should be under 2GB for even such a large cluster.
> > > > > > >
> > > > > > > During backfilling scenarios, the mons keep old maps and grow quite
> > > > > > > quickly. So if you have balancing, pg splitting, etc. ongoing for
> > > > > > > awhile, the mon stores will eventually trigger that 15GB alarm.
> > > > > > > But the intended behavior is that once the PGs are all active+clean,
> > > > > > > the old maps should be trimmed and the disk space freed.
> > > > > > >
> > > > > > > However, several people have noted that (at least in luminous
> > > > > > > releases) the old maps are not trimmed until after HEALTH_OK *and* all
> > > > > > > mons are restarted. This ticket seems related:
> > > > > > > http://tracker.ceph.com/issues/37875
> > > > > > >
> > > > > > > (Over here we're restarting mons every ~2-3 weeks, resulting in the
> > > > > > > mon stores dropping from >15GB to ~700MB each time).
> > > > > > >
> > > > > > > -- Dan
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Feb 6, 2019 at 1:26 PM Sage Weil <sage-BnTBU8nroG7k1uMJSBkQmQ@public.gmane.org> wrote:
> > > > > > > >
> > > > > > > > Hi Swami
> > > > > > > >
> > > > > > > > The limit is somewhat arbitrary, based on cluster sizes we had seen when
> > > > > > > > we picked it.  In your case it should be perfectly safe to increase it.
> > > > > > > >
> > > > > > > > sage
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, 6 Feb 2019, M Ranga Swami Reddy wrote:
> > > > > > > >
> > > > > > > > > Hello -  Are the any limits for mon_data_size for cluster with 2PB
> > > > > > > > > (with 2000+ OSDs)?
> > > > > > > > >
> > > > > > > > > Currently it set as 15G. What is logic behind this? Can we increase
> > > > > > > > > when we get the mon_data_size_warn messages?
> > > > > > > > >
> > > > > > > > > I am getting the mon_data_size_warn message even though there a ample
> > > > > > > > > of free space on the disk (around 300G free disk)
> > > > > > > > >
> > > > > > > > > Earlier thread on the same discusion:
> > > > > > > > > https://www.spinics.net/lists/ceph-users/msg42456.html
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > > Swami
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > _______________________________________________
> > > > > > > > ceph-users mailing list
> > > > > > > > ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> > > > > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > > > >
> > > > >
> > > > >