“GenericFileStoreBackend::syncfs" takes long time

“GenericFileStoreBackend::syncfs" takes long time

[崔凯]

Hi Guys,

I find an interesting thing in the Ceph practice - under high incoming writing traffic, "GenericFileStoreBackend::syncfs" operation takes long time, e.g., approximately 35 seconds. As a result, it causes journal operation to stop working temporarily and degrade the writing performance gradually.  
I use rados bench with 4M block size for writing test. Network speed is 10000Mb, journal size is 5GB,  filestore backend is XFS, max sync interval is 5 second and min sync interval is 0.1 second.  The trace excerpt list below.
my question is : If any way or configuration could reduce the syncfs time so as to improve the final writing performance ?  Thanks in advance.

trace excerpt:
2017-03-17 18:43:49.136061 7fa6f670d700 15 filestore(/var/lib/ceph/osd/ucsm-18) sync_entry committing 1634920
2017-03-17 18:43:49.735962 7fa6f670d700 10 filestore(/var/lib/ceph/osd/ucsm-18) sync_entry commit took 0.599921, interval was 5.600013
2017-03-17 18:43:49.736072 7fa6f670d700 15 filestore(/var/lib/ceph/osd/ucsm-18) sync_entry committed to op_seq 1634920
2017-03-17 18:43:54.736266 7fa6f670d700 15 filestore(/var/lib/ceph/osd/ucsm-18) sync_entry committing 1635037
2017-03-17 18:44:02.433323 7fa6f670d700 10 filestore(/var/lib/ceph/osd/ucsm-18) sync_entry commit took 7.697065, interval was 12.697185
2017-03-17 18:44:02.433372 7fa6f670d700 15 filestore(/var/lib/ceph/osd/ucsm-18) sync_entry committed to op_seq 1635037
2017-03-17 18:44:07.949692 7fa6f670d700 15 filestore(/var/lib/ceph/osd/ucsm-18) sync_entry committing 1635455
2017-03-17 18:44:24.610648 7fa6f670d700 10 filestore(/var/lib/ceph/osd/ucsm-18) sync_entry commit took 16.660962, interval was 22.177266
2017-03-17 18:44:24.610867 7fa6f670d700 15 filestore(/var/lib/ceph/osd/ucsm-18) sync_entry committed to op_seq 1635455
2017-03-17 18:44:29.856190 7fa6f670d700 15 filestore(/var/lib/ceph/osd/ucsm-18) sync_entry committing 1636081
2017-03-17 18:45:01.173941 7fa6f670d700 10 filestore(/var/lib/ceph/osd/ucsm-18) sync_entry commit took 31.317749, interval was 36.563052
2017-03-17 18:45:01.174117 7fa6f670d700 15 filestore(/var/lib/ceph/osd/ucsm-18) sync_entry committed to op_seq 1636081
2017-03-17 18:45:06.174246 7fa6f670d700 15 filestore(/var/lib/ceph/osd/ucsm-18) sync_entry committing 1636789
2017-03-17 18:45:12.364669 7fa6f670d700 10 filestore(/var/lib/ceph/osd/ucsm-18) sync_entry commit took 6.190430, interval was 11.190543
2017-03-17 18:45:12.364977 7fa6f670d700 15 filestore(/var/lib/ceph/osd/ucsm-18) sync_entry committed to op_seq 1636789
2017-03-17 18:45:32.365431 7fa6f670d700 15 filestore(/var/lib/ceph/osd/ucsm-18) sync_entry committing 1636791
2017-03-17 18:45:32.421757 7fa6f670d700 10 filestore(/var/lib/ceph/osd/ucsm-18) sync_entry commit took 0.056331, interval was 5.056418
2017-03-17 18:45:32.421819 7fa6f670d700 15 filestore(/var/lib/ceph/osd/ucsm-18) sync_entry committed to op_seq 1636791
2017-03-17 18:45:42.422096 7fa6f670d700 15 filestore(/var/lib/ceph/osd/ucsm-18) sync_entry committing 1636793

Cheers,
Klaus