More writes on filestore than on journal ?

More writes on filestore than on journal ?

[Olivier Bonvalet]

Hi,

I'm still trying to find why there is much more write operations on
filestore since Emperor/Firefly than from Dumpling.

So, I add monitoring of all perf counters values from OSD.

From what I see : «filestore.ops» reports an average of 78 operations
per seconds. But, block device monitoring reports an average of 113
operations per seconds (+45%).
please thoses 2 graphs :
- https://daevel.fr/img/firefly/osd-70.filestore-ops.png
- https://daevel.fr/img/firefly/osd-70.sda-ops.png

Do you see what can explain this difference ? (this OSD use XFS)

Thanks,
Olivier

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: More writes on blockdevice than on filestore ?

[Olivier Bonvalet]

Erg... I sent to fast. Bad title, please read «More writes on
blockdevice than on filestore)


Le lundi 23 mars 2015 à 14:21 +0100, Olivier Bonvalet a écrit :
> Hi,
> 
> I'm still trying to find why there is much more write operations on
> filestore since Emperor/Firefly than from Dumpling.
> 
> So, I add monitoring of all perf counters values from OSD.
> 
> From what I see : «filestore.ops» reports an average of 78 operations
> per seconds. But, block device monitoring reports an average of 113
> operations per seconds (+45%).
> please thoses 2 graphs :
> - https://daevel.fr/img/firefly/osd-70.filestore-ops.png
> - https://daevel.fr/img/firefly/osd-70.sda-ops.png
> 
> Do you see what can explain this difference ? (this OSD use XFS)
> 
> Thanks,
> Olivier
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] More writes on filestore than on journal ?

[Gregory Farnum]

On Mon, Mar 23, 2015 at 6:21 AM, Olivier Bonvalet <ceph.list@daevel.fr> wrote:
> Hi,
>
> I'm still trying to find why there is much more write operations on
> filestore since Emperor/Firefly than from Dumpling.

Do you have any history around this? It doesn't sound familiar,
although I bet it's because of the WBThrottle and flushing changes.

>
> So, I add monitoring of all perf counters values from OSD.
>
> From what I see : «filestore.ops» reports an average of 78 operations
> per seconds. But, block device monitoring reports an average of 113
> operations per seconds (+45%).
> please thoses 2 graphs :
> - https://daevel.fr/img/firefly/osd-70.filestore-ops.png
> - https://daevel.fr/img/firefly/osd-70.sda-ops.png

That's unfortunate but perhaps not surprising — any filestore op can
change a backing file (which requires hitting both the file and the
inode: potentially two disk seeks), as well as adding entries to the
leveldb instance.
-Greg

>
> Do you see what can explain this difference ? (this OSD use XFS)
>
> Thanks,
> Olivier
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [ceph-users] More writes on filestore than on journal ?

[Olivier Bonvalet]

Hi,

Le lundi 23 mars 2015 à 07:29 -0700, Gregory Farnum a écrit :
> On Mon, Mar 23, 2015 at 6:21 AM, Olivier Bonvalet <ceph.list@daevel.fr> wrote:
> > Hi,
> >
> > I'm still trying to find why there is much more write operations on
> > filestore since Emperor/Firefly than from Dumpling.
> 
> Do you have any history around this? It doesn't sound familiar,
> although I bet it's because of the WBThrottle and flushing changes.

I only have history for block device stats and global stats reports by
«ceph status».
When I have upgrade from Dumpling to Firefly (via Emperor), write
operations increased a lot on OSD.
I suppose it's because of WBThrottle too, but can't find any parameter
able to confirm that.


> >
> > So, I add monitoring of all perf counters values from OSD.
> >
> > From what I see : «filestore.ops» reports an average of 78 operations
> > per seconds. But, block device monitoring reports an average of 113
> > operations per seconds (+45%).
> > please thoses 2 graphs :
> > - https://daevel.fr/img/firefly/osd-70.filestore-ops.png
> > - https://daevel.fr/img/firefly/osd-70.sda-ops.png
> 
> That's unfortunate but perhaps not surprising — any filestore op can
> change a backing file (which requires hitting both the file and the
> inode: potentially two disk seeks), as well as adding entries to the
> leveldb instance.
> -Greg
> 

Ok thanks, so this part can be «normal».

> >
> > Do you see what can explain this difference ? (this OSD use XFS)
> >
> > Thanks,
> > Olivier
> >
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html