* sleeps and waits during io_submit
@ 2015-11-28 2:43 Glauber Costa
2015-11-30 14:10 ` Brian Foster
2015-11-30 23:10 ` Dave Chinner
0 siblings, 2 replies; 58+ messages in thread
From: Glauber Costa @ 2015-11-28 2:43 UTC (permalink / raw)
To: xfs, Avi Kivity, david
[-- Attachment #1: Type: text/plain, Size: 3130 bytes --]
Hello my dear XFSers,
For those of you who don't know, we at ScyllaDB produce a modern NoSQL
data store that, at the moment, runs on top of XFS only. We deal
exclusively with asynchronous and direct IO, due to our
thread-per-core architecture. Due to that, we avoid issuing any
operation that will sleep.
While debugging an extreme case of bad performance (most likely
related to a not-so-great disk), I have found a variety of cases in
which XFS blocks. To find those, I have used perf record -e
sched:sched_switch -p <pid_of_db>, and I am attaching the perf report
as xfs-sched_switch.log. Please note that this doesn't tell me for how
long we block, but as mentioned before, blocking operations outside
our control are detrimental to us regardless of the elapsed time.
For those who are not acquainted to our internals, please ignore
everything in that file but the xfs functions. For the xfs symbols,
there are two kinds of events: the ones that are a children of
io_submit, where we don't tolerate blocking, and the ones that are
children of our helper IO thread, to where we push big operations that
we know will block until we can get rid of them all. We care about the
former and ignore the latter.
Please allow me to ask you a couple of questions about those findings.
If we are doing anything wrong, advise on best practices is truly
welcome.
1) xfs_buf_lock -> xfs_log_force.
I've started wondering what would make xfs_log_force sleep. But then I
have noticed that xfs_log_force will only be called when a buffer is
marked stale. Most of the times a buffer is marked stale seems to be
due to errors. Although that is not my case (more on that), it got me
thinking that maybe the right thing to do would be to avoid hitting
this case altogether?
The file example-stale.txt contains a backtrace of the case where we
are being marked as stale. It seems to be happening when we convert
the the inode's extents from unwritten to real. Can this case be
avoided? I won't pretend I know the intricacies of this, but couldn't
we be keeping extents from the very beginning to avoid creating stale
buffers?
2) xfs_buf_lock -> down
This is one I truly don't understand. What can be causing contention
in this lock? We never have two different cores writing to the same
buffer, nor should we have the same core doingCAP_FOWNER so.
3) xfs_file_aio_write_checks -> file_update_time -> xfs_vn_update_time
You guys seem to have an interface to avoid that, by setting the
FMODE_NOCMTIME flag. This is done by issuing the open by handle ioctl,
which will set this flag for all regular files. That's great, but that
ioctl required CAP_SYS_ADMIN, which is a big no for us, since we run
our server as an unprivileged user. I don't understand, however, why
such an strict check is needed. If we have full rights on the
filesystem, why can't we issue this operation? In my view, CAP_FOWNER
should already be enough.I do understand the handles have to be stable
and a file can have its ownership changed, in which case the previous
owner would keep the handle valid. Is that the reason you went with
the most restrictive capability ?
[-- Attachment #2: xfs-sched_switch.log --]
[-- Type: text/plain, Size: 171763 bytes --]
# To display the perf.data header info, please use --header/--header-only options.
#
#
# Total Lost Samples: 0
#
# Samples: 2K of event 'sched:sched_switch'
# Event count (approx.): 2669
#
# Overhead Command Shared Object Symbol
# ........ ....... ................. ..............
#
100.00% scylla [kernel.kallsyms] [k] __schedule
|
---__schedule
|
|--96.18%-- schedule
| |
| |--56.14%-- schedule_user
| | |
| | |--53.30%-- int_careful
| | | |
| | | |--45.05%-- 0x7f4ade6f74ed
| | | | reactor_backend_epoll::make_reactor_notifier
| | | | |
| | | | |--67.63%-- syscall_work_queue::submit_item
| | | | | |
| | | | | |--32.05%-- posix_file_impl::truncate
| | | | | | |
| | | | | | |--65.33%-- _ZN12continuationIZN6futureIJEE4thenIZN19file_data_sink_impl5flushEvEUlvE_S1_EET0_OT_EUlS7_E_JEE3runEv
| | | | | | | reactor::del_timer
| | | | | | | 0x60b0000e2040
| | | | | | |
| | | | | | |--20.00%-- db::commitlog::segment::flush(unsigned long)::{lambda()#1}::operator()
| | | | | | | |
| | | | | | | |--73.33%-- future<>::then<db::commitlog::segment::flush(unsigned long)::{lambda()#1}, future<lw_shared_ptr<db::commitlog::segment> > >
| | | | | | | | _ZN12continuationIZN6futureIJ13lw_shared_ptrIN2db9commitlog7segmentEEEE4thenIZNS4_4syncEvEUlT_E_S6_EET0_OS8_EUlSB_E_JS5_EE3runEv
| | | | | | | | reactor::del_timer
| | | | | | | | 0x60e0000e2040
| | | | | | | |
| | | | | | | --26.67%-- _ZN12continuationIZN6futureIJEE4thenIZN2db9commitlog7segment5flushEmEUlvE_S0_IJ13lw_shared_ptrIS5_EEEEET0_OT_EUlSC_E_JEE3runEv
| | | | | | | reactor::del_timer
| | | | | | | 0x6090000e2040
| | | | | | |
| | | | | | |--10.67%-- sstables::sstable::seal_sstable
| | | | | | | std::_Function_handler<void (), futurize<std::result_of<std::decay<sstables::sstable::write_components(mutation_reader, unsigned long, lw_shared_ptr<schema const>, unsigned long)::{lambda()#1}>::type ()>::type>::type seastar::async<sstables::sstable::write_components(mutation_reader, unsigned long, lw_shared_ptr<schema const>, unsigned long)::{lambda()#1}>(std::decay&&, (std::decay<sstables::sstable::write_components(mutation_reader, unsigned long, lw_shared_ptr<schema const>, unsigned long)::{lambda()#1}>::type&&)...)::{lambda(futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type> seastar::async<{lambda()#1}>(futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type>::type, std::decay<{lambda()#1}>::type&&)::work&)#1}::operator()(futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type> seastar::async<{lambda()#1}>(futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type>::type, std::decay<{lambda()#1}>::type&&)::work)::{lambda()#1}>::_M_invoke
| | | | | | | _GLOBAL__sub_I__ZN12app_templateC2Ev
| | | | | | |
| | | | | | --4.00%-- sstables::sstable::write_toc
| | | | | | sstables::sstable::prepare_write_components
| | | | | | |
| | | | | | |--50.00%-- 0x4d3a4f6ec4e8cd75
| | | | | | |
| | | | | | --50.00%-- 0x3ebf3dd80e3b174d
| | | | | |
| | | | | |--23.93%-- posix_file_impl::discard
| | | | | | |
| | | | | | |--82.14%-- _ZN12continuationIZN6futureIImEE4thenIZN19file_data_sink_impl6do_putEm16temporary_bufferIcEEUlmE_S0_IIEEEET0_OT_EUlSA_E_ImEE3runEv
| | | | | | | reactor::del_timer
| | | | | | | 0x6080000e2040
| | | | | | |
| | | | | | --17.86%-- futurize<future<lw_shared_ptr<db::commitlog::segment> > >::apply<db::commitlog::segment_manager::allocate_segment(bool)::{lambda(file)#1}, file>
| | | | | | _ZN12continuationIZN6futureIJ4fileEE4thenIZN2db9commitlog15segment_manager16allocate_segmentEbEUlS1_E_S0_IJ13lw_shared_ptrINS5_7segmentEEEEEET0_OT_EUlSE_E_JS1_EE3runEv
| | | | | |
| | | | | |--20.94%-- reactor::open_file_dma
| | | | | | |
| | | | | | |--20.41%-- db::commitlog::segment_manager::allocate_segment
| | | | | | | db::commitlog::segment_manager::on_timer()::{lambda()#1}::operator()
| | | | | | | 0xb8c264
| | | | | | |
| | | | | | |--14.29%-- sstables::sstable::write_simple<(sstables::sstable::component_type)8, sstables::statistics>
| | | | | | | std::_Function_handler<void (), futurize<std::result_of<std::decay<sstables::sstable::write_components(mutation_reader, unsigned long, lw_shared_ptr<schema const>, unsigned long)::{lambda()#1}>::type ()>::type>::type seastar::async<sstables::sstable::write_components(mutation_reader, unsigned long, lw_shared_ptr<schema const>, unsigned long)::{lambda()#1}>(std::decay&&, (std::decay<sstables::sstable::write_components(mutation_reader, unsigned long, lw_shared_ptr<schema const>, unsigned long)::{lambda()#1}>::type&&)...)::{lambda(futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type> seastar::async<{lambda()#1}>(futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type>::type, std::decay<{lambda()#1}>::type&&)::work&)#1}::operator()(futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type> seastar::async<{lambda()#1}>(futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type>::type, std::decay<{lambda()#1}>::type&&)::work)::{lambda()#1}>::_M_invoke
| | | | | | | _GLOBAL__sub_I__ZN12app_templateC2Ev
| | | | | | |
| | | | | | |--12.24%-- sstables::write_crc
| | | | | | | |
| | | | | | | |--16.67%-- 0x313532343536002f
| | | | | | | |
| | | | | | | |--16.67%-- 0x373633323533002f
| | | | | | | |
| | | | | | | |--16.67%-- 0x363139333232002f
| | | | | | | |
| | | | | | | |--16.67%-- 0x353933303330002f
| | | | | | | |
| | | | | | | |--16.67%-- 0x383930383133002f
| | | | | | | |
| | | | | | | --16.67%-- 0x323338303037002f
| | | | | | |
| | | | | | |--12.24%-- sstables::write_digest
| | | | | | |
| | | | | | |--10.20%-- sstables::sstable::write_simple<(sstables::sstable::component_type)7, sstables::filter>
| | | | | | | sstables::sstable::write_filter
| | | | | | | std::_Function_handler<void (), futurize<std::result_of<std::decay<sstables::sstable::write_components(mutation_reader, unsigned long, lw_shared_ptr<schema const>, unsigned long)::{lambda()#1}>::type ()>::type>::type seastar::async<sstables::sstable::write_components(mutation_reader, unsigned long, lw_shared_ptr<schema const>, unsigned long)::{lambda()#1}>(std::decay&&, (std::decay<sstables::sstable::write_components(mutation_reader, unsigned long, lw_shared_ptr<schema const>, unsigned long)::{lambda()#1}>::type&&)...)::{lambda(futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type> seastar::async<{lambda()#1}>(futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type>::type, std::decay<{lambda()#1}>::type&&)::work&)#1}::operator()(futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type> seastar::async<{lambda()#1}>(futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type>::type, std::decay<{lambda()#1}>::type&&)::work)::{lambda()#1}>::_M_invoke
| | | | | | | _GLOBAL__sub_I__ZN12app_templateC2Ev
| | | | | | |
| | | | | | |--10.20%-- sstables::sstable::write_simple<(sstables::sstable::component_type)4, sstables::summary_ka>
| | | | | | | std::_Function_handler<void (), futurize<std::result_of<std::decay<sstables::sstable::write_components(mutation_reader, unsigned long, lw_shared_ptr<schema const>, unsigned long)::{lambda()#1}>::type ()>::type>::type seastar::async<sstables::sstable::write_components(mutation_reader, unsigned long, lw_shared_ptr<schema const>, unsigned long)::{lambda()#1}>(std::decay&&, (std::decay<sstables::sstable::write_components(mutation_reader, unsigned long, lw_shared_ptr<schema const>, unsigned long)::{lambda()#1}>::type&&)...)::{lambda(futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type> seastar::async<{lambda()#1}>(futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type>::type, std::decay<{lambda()#1}>::type&&)::work&)#1}::operator()(futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type> seastar::async<{lambda()#1}>(futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type>::type, std::decay<{lambda()#1}>::type&&)::work)::{lambda()#1}>::_M_invoke
| | | | | | | _GLOBAL__sub_I__ZN12app_templateC2Ev
| | | | | | |
| | | | | | |--10.20%-- 0x78d93b
| | | | | | |
| | | | | | |--6.12%-- sstables::sstable::open_data
| | | | | | | |
| | | | | | | --100.00%-- 0x8000000004000000
| | | | | | |
| | | | | | --4.08%-- sstables::sstable::write_toc
| | | | | | sstables::sstable::prepare_write_components
| | | | | | |
| | | | | | --100.00%-- 0x6100206690ef
| | | | | |
| | | | | |--18.38%-- syscall_work_queue::submit_item
| | | | | | |
| | | | | | |--10.00%-- 0x7f4ad89f8fe0
| | | | | | |
| | | | | | |--7.50%-- 0x7f4ad83f8fe0
| | | | | | |
| | | | | | |--7.50%-- 0x7f4ad6bf8fe0
| | | | | | |
| | | | | | |--7.50%-- 0x7f4ad65f8fe0
| | | | | | |
| | | | | | |--5.00%-- 0x60b015e8cd90
| | | | | | |
| | | | | | |--5.00%-- 0x60100acaed90
| | | | | | |
| | | | | | |--5.00%-- 0x607006f04d90
| | | | | | |
| | | | | | |--5.00%-- 0xffffffffffffa5d0
| | | | | | |
| | | | | | |--2.50%-- 0x60e01acbed90
| | | | | | |
| | | | | | |--2.50%-- 0x60e01acbec60
| | | | | | |
| | | | | | |--2.50%-- 0x60a018d7ad90
| | | | | | |
| | | | | | |--2.50%-- 0x60a018d7ac60
| | | | | | |
| | | | | | |--2.50%-- 0x60b015e8cc60
| | | | | | |
| | | | | | |--2.50%-- 0x60900bb8ad60
| | | | | | |
| | | | | | |--2.50%-- 0x60100acaec60
| | | | | | |
| | | | | | |--2.50%-- 0x60800951dd90
| | | | | | |
| | | | | | |--2.50%-- 0x60800951dc60
| | | | | | |
| | | | | | |--2.50%-- 0x60d009089d90
| | | | | | |
| | | | | | |--2.50%-- 0x60d009089c60
| | | | | | |
| | | | | | |--2.50%-- 0x607006f04c60
| | | | | | |
| | | | | | |--2.50%-- 0x60f005984d60
| | | | | | |
| | | | | | |--2.50%-- 0x7f4ad77f8fe0
| | | | | | |
| | | | | | |--2.50%-- 0x7f4adb9f8fe0
| | | | | | |
| | | | | | |--2.50%-- 0x7f4ad9bf8fe0
| | | | | | |
| | | | | | |--2.50%-- 0x7f4ad7df8fe0
| | | | | | |
| | | | | | |--2.50%-- 0x7f4ad77f8fe0
| | | | | | |
| | | | | | --2.50%-- 0x7f4ad5ff8fe0
| | | | | |
| | | | | |--2.99%-- reactor::open_directory
| | | | | | |
| | | | | | |--57.14%-- sstables::sstable::filename
| | | | | | |
| | | | | | --42.86%-- sstables::sstable::write_toc
| | | | | | sstables::sstable::prepare_write_components
| | | | | | |
| | | | | | |--50.00%-- 0x4d3a4f6ec4e8cd75
| | | | | | |
| | | | | | --50.00%-- 0x3ebf3dd80e3b174d
| | | | | |
| | | | | --1.71%-- reactor::rename_file
| | | | | sstables::sstable::seal_sstable
| | | | | std::_Function_handler<void (), futurize<std::result_of<std::decay<sstables::sstable::write_components(mutation_reader, unsigned long, lw_shared_ptr<schema const>, unsigned long)::{lambda()#1}>::type ()>::type>::type seastar::async<sstables::sstable::write_components(mutation_reader, unsigned long, lw_shared_ptr<schema const>, unsigned long)::{lambda()#1}>(std::decay&&, (std::decay<sstables::sstable::write_components(mutation_reader, unsigned long, lw_shared_ptr<schema const>, unsigned long)::{lambda()#1}>::type&&)...)::{lambda(futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type> seastar::async<{lambda()#1}>(futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type>::type, std::decay<{lambda()#1}>::type&&)::work&)#1}::operator()(futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type> seastar::async<{lambda()#1}>(futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type>::type, std::decay<{lambda()#1}>::type&&)::work)::{lambda()#1}>::_M_invoke
| | | | | _GLOBAL__sub_I__ZN12app_templateC2Ev
| | | | |
| | | | --32.37%-- _ZN12continuationIZN6futureIJEE4thenIZN18syscall_work_queue11submit_itemEPNS3_9work_itemEEUlvE_S1_EET0_OT_EUlS9_E_JEE3runEv
| | | | reactor::del_timer
| | | | 0x60d0000e2040
| | | |
| | | |--29.04%-- __vdso_clock_gettime
| | | |
| | | |--19.66%-- 0x7f4ade42b193
| | | | reactor_backend_epoll::complete_epoll_event
| | | | |
| | | | |--41.61%-- smp_message_queue::async_work_item<future<> seastar::sharded<database>::invoke_on<service::storage_proxy::mutate_locally(frozen_mutation const&)::{lambda(database&)#1}, future<> >(unsigned int, service::storage_proxy::mutate_locally(frozen_mutation const&)::{lambda(database&)#1}&&)::{lambda()#1}>::process
| | | | | |
| | | | | |--79.03%-- smp_message_queue::process_queue<2ul, smp_message_queue::process_incoming()::{lambda(smp_message_queue::work_item*)#1}>
| | | | | | |
| | | | | | |--95.92%-- 0x6070000c3000
| | | | | | |
| | | | | | |--2.04%-- 0x61d0000c1000
| | | | | | |
| | | | | | --2.04%-- 0x61d0000c1000
| | | | | |
| | | | | |--3.23%-- 0x14dd51
| | | | | |
| | | | | |--1.61%-- 0x162a54
| | | | | |
| | | | | |--1.61%-- 0x161dca
| | | | | |
| | | | | |--1.61%-- 0x159c8b
| | | | | |
| | | | | |--1.61%-- 0x1598b5
| | | | | |
| | | | | |--1.61%-- 0x14dd3e
| | | | | |
| | | | | |--1.61%-- 0x14bad8
| | | | | |
| | | | | |--1.61%-- 0x14a880
| | | | | |
| | | | | |--1.61%-- 0x127105
| | | | | |
| | | | | |--1.61%-- 0x6070000e2040
| | | | | |
| | | | | |--1.61%-- smp_message_queue::process_queue<2ul, smp_message_queue::process_incoming()::{lambda(smp_message_queue::work_item*)#1}>
| | | | | | 0x60d0000c3000
| | | | | |
| | | | | --1.61%-- __vdso_clock_gettime
| | | | | 0x7f4ad77f9160
| | | | |
| | | | |--30.20%-- __restore_rt
| | | | | |
| | | | | |--57.14%-- __vdso_clock_gettime
| | | | | | 0x1d
| | | | | |
| | | | | |--9.52%-- smp_message_queue::smp_message_queue
| | | | | | 0x6070000c3000
| | | | | |
| | | | | |--4.76%-- 0x600000357240
| | | | | |
| | | | | |--4.76%-- 0x60000031a640
| | | | | |
| | | | | |--2.38%-- posix_file_impl::list_directory
| | | | | | 0x609000044730
| | | | | |
| | | | | |--2.38%-- 0x46efbf
| | | | | |
| | | | | |--2.38%-- 0x600000442e40
| | | | | |
| | | | | |--2.38%-- 0x600000376440
| | | | | |
| | | | | |--2.38%-- 0x6000002bac40
| | | | | |
| | | | | |--2.38%-- 0x600000295640
| | | | | |
| | | | | |--2.38%-- 0x600000289e40
| | | | | |
| | | | | |--2.38%-- 0x60000031a640
| | | | | |
| | | | | |--2.38%-- 0x7f4ade6f74ed
| | | | | | __libc_siglongjmp
| | | | | | 0x60000047be40
| | | | | |
| | | | | --2.38%-- 0x7f4adb3f7fd0
| | | | |
| | | | |--14.09%-- 0x33
| | | | |
| | | | |--12.08%-- promise<temporary_buffer<char> >::promise
| | | | | _ZN6futureIJ16temporary_bufferIcEEE4thenIZN12input_streamIcE12read_exactlyEmEUlT_E_S2_EET0_OS6_
| | | | | |
| | | | | |--44.44%-- input_stream<char>::read_exactly
| | | | | | 0x8
| | | | | |
| | | | | |--11.11%-- 0x7f4adb3f8ea0
| | | | | |
| | | | | |--11.11%-- 0x7f4ad9bf8ea0
| | | | | |
| | | | | |--11.11%-- 0x7f4ad89f8ea0
| | | | | |
| | | | | |--11.11%-- 0x7f4ad83f8ea0
| | | | | |
| | | | | |--5.56%-- 0x7f4ad77f8ea0
| | | | | |
| | | | | --5.56%-- 0x7f4ad7df8ea0
| | | | |
| | | | |--1.34%-- 0x7f4ad6bf8d80
| | | | |
| | | | --0.67%-- 0x7f4adadf8d80
| | | |
| | | |--4.43%-- __libc_send
| | | | _ZN12continuationIZN6futureIJmEE4thenIZN7reactor14write_all_partER17pollable_fd_statePKvmmEUlmE_S0_IJEEEET0_OT_EUlSC_E_JmEE3runEv
| | | | |
| | | | |--14.71%-- 0x4
| | | | |
| | | | |--11.76%-- 0x7f4ad89f8de0
| | | | |
| | | | |--8.82%-- 0x7f4adb3f8de0
| | | | |
| | | | |--8.82%-- 0x7f4ad9bf8de0
| | | | |
| | | | |--8.82%-- 0x7f4ad77f8de0
| | | | |
| | | | |--8.82%-- 0x7f4ad6bf8de0
| | | | |
| | | | |--5.88%-- 0x7f4ad83f8de0
| | | | |
| | | | |--5.88%-- 0x7f4ad7df8de0
| | | | |
| | | | |--5.88%-- 0x7f4ad53f8de0
| | | | |
| | | | |--2.94%-- 0x7f4acc9f8de0
| | | | |
| | | | |--2.94%-- continuation<future<file>::wait()::{lambda(future_state<file>&&)#1}, file>::~continuation
| | | | | 0x611003c8e9b8
| | | | |
| | | | |--2.94%-- 0x7f4adb9f8de0
| | | | |
| | | | |--2.94%-- 0x7f4ad71f8de0
| | | | |
| | | | |--2.94%-- 0x7f4ad65f8de0
| | | | |
| | | | |--2.94%-- 0x7f4ad59f8de0
| | | | |
| | | | --2.94%-- 0x7f4ad35f8de0
| | | |
| | | |--1.56%-- 0x7f4ade6f754d
| | | | reactor::read_some
| | | | |
| | | | |--66.67%-- _ZN12continuationIZN6futureIJEE4thenIZZN7service13storage_proxy22send_to_live_endpointsEmENKUlRSt4pairIK13basic_sstringIcjLj15EESt6vectorIN3gms12inet_addressESaISB_EEEE_clESF_EUlvE_S1_EET0_OT_EUlSK_E_JEE3runEv
| | | | | reactor::del_timer
| | | | | 0x6070000e2040
| | | | |
| | | | |--8.33%-- _ZN12continuationIZN6futureIIEE4thenIZ5sleepINSt6chrono3_V212system_clockEmSt5ratioILl1ELl1000000EEES1_NS4_8durationIT0_T1_EEEUlvE_S1_EESA_OT_EUlSF_E_IEE3runEv
| | | | | reactor::del_timer
| | | | | 0x6080000e2040
| | | | |
| | | | |--8.33%-- 0x600000483640
| | | | |
| | | | |--8.33%-- 0x600000480440
| | | | |
| | | | --8.33%-- 0x36
| | | --0.26%-- [...]
| | |
| | --46.70%-- retint_careful
| | |
| | |--6.24%-- posix_file_impl::list_directory
| | | |
| | | |--80.00%-- 0x60f0000e2020
| | | |
| | | |--5.00%-- 0x601000044730
| | | |
| | | |--5.00%-- 0x60e000044720
| | | |
| | | |--2.50%-- 0x60f000135500
| | | |
| | | |--2.50%-- 0x6190000e2098
| | | |
| | | |--2.50%-- 0x60d0000c3000
| | | |
| | | --2.50%-- 0x1
| | |
| | |--3.42%-- boost::lockfree::detail::ringbuffer_base<smp_message_queue::work_item*>::pop
| | | |
| | | |--95.65%-- boost::program_options::variables_map::get
| | | |
| | | --4.35%-- 0x618000044680
| | |
| | |--3.12%-- memory::small_pool::add_more_objects
| | | |
| | | |--10.53%-- managed_vector<atomic_cell_or_collection, 5u, unsigned int>::clear_and_release
| | | | mutation_partition::clustered_row
| | | | mutation::set_clustered_cell
| | | | cql3::constants::setter::execute
| | | | cql3::statements::update_statement::add_update_for_key
| | | | _ZN8futurizeI6futureIJSt6vectorI8mutationSaIS2_EEEEE5applyIZN4cql310statements22modification_statement13get_mutationsERN7seastar7shardedIN7service13storage_proxyEEERKNS8_13query_optionsEblEUlT_E_JSt10unique_ptrINS8_17update_parametersESt14default_deleteISN_EEEEES5_OSK_OSt5tupleIJDpT0_EE
| | | | cql3::statements::modification_statement::get_mutations
| | | | cql3::statements::modification_statement::execute_without_condition
| | | | cql3::query_options::query_options
| | | | |
| | | | |--50.00%-- 0x7f4ad77f80e0
| | | | |
| | | | --50.00%-- 0x7f4ad6bf80e0
| | | |
| | | |--10.53%-- memory::small_pool::add_more_objects
| | | | |
| | | | |--50.00%-- 0x60e00015d000
| | | | |
| | | | --50.00%-- 0x60b00af6c758
| | | |
| | | |--5.26%-- 0x60a018ee3867
| | | |
| | | |--5.26%-- 0x60d00d41f680
| | | |
| | | |--5.26%-- 0x61400c6bb4d0
| | | |
| | | |--5.26%-- 0x60e007c918d6
| | | |
| | | |--5.26%-- 0x60e0078294ce
| | | |
| | | |--5.26%-- 0x607006ee4da0
| | | |
| | | |--5.26%-- _ZN12continuationIZN6futureIJEE12then_wrappedIZNS1_16handle_exceptionIZN7service13storage_proxy22send_to_live_endpointsEmEUlNSt15__exception_ptr13exception_ptrEE0_EES1_OT_EUlSA_E_S1_EET0_SA_EUlSA_E_JEE3runEv
| | | | reactor::del_timer
| | | | 0x6030000e2040
| | | |
| | | |--5.26%-- service::storage_proxy::mutate_locally
| | | | service::storage_proxy::send_to_live_endpoints
| | | | parallel_for_each<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, service::storage_proxy::mutate_begin(std::vector<unsigned long, std::allocator<unsigned long> >, db::consistency_level)::{lambda(unsigned long)#1}>
| | | | 0x601000136d00
| | | |
| | | |--5.26%-- 0x60a0001900e0
| | | |
| | | |--5.26%-- 0x60e00015d040
| | | |
| | | |--5.26%-- 0x61300015d000
| | | |
| | | |--5.26%-- 0x60e00013bde0
| | | |
| | | |--5.26%-- 0x60b00010f308
| | | |
| | | |--5.26%-- 0x6010000e4808
| | | |
| | | --5.26%-- 0x7f4ad65f7f50
| | |
| | |--2.82%-- std::unique_ptr<reactor::pollfn, std::default_delete<std::unique_ptr> > reactor::make_pollfn<reactor::run()::{lambda()#3}>(reactor::run()::{lambda()#3}&&)::the_pollfn::poll_and_check_more_work
| | | |
| | | |--25.00%-- boost::lockfree::detail::ringbuffer_base<smp_message_queue::work_item*>::pop
| | | | boost::program_options::variables_map::get
| | | |
| | | |--25.00%-- 0x1
| | | |
| | | |--12.50%-- 0x53
| | | |
| | | |--12.50%-- 0x3e
| | | |
| | | |--12.50%-- 0x24
| | | |
| | | --12.50%-- 0xb958000000000000
| | |
| | |--2.67%-- std::_Function_handler<partition_presence_checker_result (partition_key const&), column_family::make_partition_presence_checker(lw_shared_ptr<std::map<long, lw_shared_ptr<sstables::sstable>, std::less<long>, std::allocator<std::pair<long const, lw_shared_ptr<sstables::sstable> > > > >)::{lambda(partition_key const&)#1}>::_M_invoke
| | | |
| | | |--66.67%-- 0x1b5c280
| | | |
| | | |--27.78%-- managed_vector<atomic_cell_or_collection, 5u, unsigned int>::resize
| | | | row::apply
| | | | mutation_partition_applier::accept_row_cell
| | | | mutation_partition_view::accept
| | | |
| | | --5.56%-- 0x2a4399
| | |
| | |--2.08%-- smp_message_queue::smp_message_queue
| | | |
| | | |--60.00%-- 0x60f0000c3000
| | | |
| | | |--10.00%-- 0x6000002d7240
| | | |
| | | |--10.00%-- 0x19
| | | |
| | | |--10.00%-- 0xb
| | | |
| | | --10.00%-- 0x7
| | |
| | |--1.93%-- smp_message_queue::process_queue<4ul, smp_message_queue::process_completions()::{lambda(smp_message_queue::work_item*)#1}>
| | |
| | |--1.63%-- __vdso_clock_gettime
| | | |
| | | --100.00%-- __clock_gettime
| | | std::chrono::_V2::system_clock::now
| | | 0xa63209
| | |
| | |--1.49%-- memory::small_pool::deallocate
| | | |
| | | |--40.00%-- managed_vector<atomic_cell_or_collection, 5u, unsigned int>::emplace_back<atomic_cell_or_collection>
| | | |
| | | |--20.00%-- std::_Hashtable<unsigned long, std::pair<unsigned long const, service::storage_proxy::rh_entry>, std::allocator<std::pair<unsigned long const, service::storage_proxy::rh_entry> >, std::__detail::_Select1st, std::equal_to<unsigned long>, std::hash<unsigned long>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_erase
| | | | service::storage_proxy::got_response
| | | | _ZN12continuationIZN6futureIJEE4thenIZZN7service13storage_proxy22send_to_live_endpointsEmENKUlRSt4pairIK13basic_sstringIcjLj15EESt6vectorIN3gms12inet_addressESaISB_EEEE_clESF_EUlvE_S1_EET0_OT_EUlSK_E_JEE3runEv
| | | | reactor::del_timer
| | | | 0x6100000e2040
| | | |
| | | |--10.00%-- cql3::statements::modification_statement::get_mutations
| | | |
| | | |--10.00%-- cql3::statements::modification_statement::build_partition_keys
| | | | cql3::statements::modification_statement::create_exploded_clustering_prefix
| | | | 0x60c014be0b00
| | | |
| | | |--10.00%-- mutation_partition::~mutation_partition
| | | | std::vector<mutation, std::allocator<mutation> >::~vector
| | | | service::storage_proxy::mutate_with_triggers
| | | | cql3::statements::modification_statement::execute_without_condition
| | | | cql3::statements::modification_statement::execute
| | | | cql3::query_processor::process_statement
| | | | transport::cql_server::connection::process_execute
| | | | transport::cql_server::connection::process_request_one
| | | | futurize<future<std::pair<foreign_ptr<shared_ptr<transport::cql_server::response> >, service::client_state> > >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}::operator()(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&) const::{lambda(temporary_buffer<char>)#1}::operator()(temporary_buffer) const::{lambda()#1}::operator()()::{lambda()#1}&>
| | | | futurize<future<> >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}::operator()(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&) const::{lambda(temporary_buffer<char>)#1}, temporary_buffer>
| | | | futurize<future<> >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}, future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> > >
| | | | 0x8961de
| | | |
| | | --10.00%-- object_deleter_impl<deleter>::~object_deleter_impl
| | | _ZN12continuationIZN6futureIJEE12then_wrappedIZZNS1_7finallyIZ7do_withI11foreign_ptrI10shared_ptrIN9transport10cql_server8responseEEEZZNS8_10connection14write_responseEOSB_ENUlvE_clEvEUlRT_E_EDaOSF_OT0_EUlvE_EES1_SI_ENUlS1_E_clES1_EUlSF_E_S1_EESJ_SI_EUlSI_E_JEED0Ev
| | | 0x61a0000c3db0
| | |
| | |--1.34%-- dht::decorated_key::equal
| | | |
| | | |--83.33%-- 0x607000138f00
| | | |
| | | --16.67%-- 0x60a0000e0f40
| | |
| | |--1.34%-- service::storage_proxy::send_to_live_endpoints
| | |
| | |--1.19%-- transport::cql_server::connection::process_execute
| | | transport::cql_server::connection::process_request_one
| | | futurize<future<std::pair<foreign_ptr<shared_ptr<transport::cql_server::response> >, service::client_state> > >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}::operator()(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&) const::{lambda(temporary_buffer<char>)#1}::operator()(temporary_buffer) const::{lambda()#1}::operator()()::{lambda()#1}&>
| | | futurize<future<> >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}::operator()(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&) const::{lambda(temporary_buffer<char>)#1}, temporary_buffer>
| | | futurize<future<> >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}, future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> > >
| | | |
| | | |--87.50%-- transport::cql_server::connection::process_request
| | | | do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>
| | | | do_void_futurize_apply<void do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>(transport::cql_server::connection::process()::{lambda()#1}&, transport::cql_server::connection::process()::{lambda()#2}&&, promise<>)::{lambda(future<>)#1}, promise<> >
| | | | do_void_futurize_apply<void do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>(transport::cql_server::connection::process()::{lambda()#1}&, transport::cql_server::connection::process()::{lambda()#2}&&, promise<>)::{lambda(future<>)#1}, promise<> >
| | | | 0x60e0000c3000
| | | |
| | | --12.50%-- 0x8961de
| | |
| | |--1.19%-- reactor::run
| | | |
| | | |--87.50%-- smp::configure(boost::program_options::variables_map)::{lambda()#1}::operator()
| | | | continuation<future<temporary_buffer<char> > future<>::then<future<temporary_buffer<char> > file::dma_read_bulk<char>(unsigned long, unsigned long)::{lambda(unsigned long)#1}::operator()(unsigned long)::{lambda()#3}, future<temporary_buffer<char> > >(future<temporary_buffer<char> > file::dma_read_bulk<char>(unsigned long, unsigned long)::{lambda(unsigned long)#1}::operator()(unsigned long)::{lambda()#3}&&)::{lambda(future<temporary_buffer<char> >)#1}>::run
| | | | 0x600000043d00
| | | |
| | | --12.50%-- app_template::run_deprecated
| | | main
| | | __libc_start_main
| | | _GLOBAL__sub_I__ZN3org6apache9cassandra21g_cassandra_constantsE
| | | 0x7f4ae20c9fa0
| | |
| | |--1.04%-- __clock_gettime
| | | std::chrono::_V2::system_clock::now
| | | |
| | | |--42.86%-- reactor::run
| | | | smp::configure(boost::program_options::variables_map)::{lambda()#1}::operator()
| | | | continuation<future<temporary_buffer<char> > future<>::then<future<temporary_buffer<char> > file::dma_read_bulk<char>(unsigned long, unsigned long)::{lambda(unsigned long)#1}::operator()(unsigned long)::{lambda()#3}, future<temporary_buffer<char> > >(future<temporary_buffer<char> > file::dma_read_bulk<char>(unsigned long, unsigned long)::{lambda(unsigned long)#1}::operator()(unsigned long)::{lambda()#3}&&)::{lambda(future<temporary_buffer<char> >)#1}>::run
| | | | 0x600000043d00
| | | |
| | | |--14.29%-- 0xa63209
| | | |
| | | |--14.29%-- continuation<future<> future<>::finally<auto do_with<std::vector<frozen_mutation, std::allocator<frozen_mutation> >, shared_ptr<service::storage_proxy>, service::storage_proxy::init_messaging_service()::{lambda(std::vector<frozen_mutation, std::allocator<frozen_mutation> >)#1}::operator()(std::vector<frozen_mutation, std::allocator<frozen_mutation> >) const::{lambda(std::vector<frozen_mutation, std::allocator<frozen_mutation> > const&, shared_ptr<service::storage_proxy>&)#1}>(std::vector<frozen_mutation, std::allocator<frozen_mutation> >&&, shared_ptr<service::storage_proxy>&&, service::storage_proxy::init_messaging_service()::{lambda(std::vector<frozen_mutation, std::allocator<frozen_mutation> >)#1}::operator()(std::vector<frozen_mutation, std::allocator<frozen_mutation> >) const::{lambda(std::vector<frozen_mutation, std::allocator<frozen_mutation> > const&, shared_ptr<service::storage_proxy>&)#1}&&)::{lambda()#1}>(service::storage_proxy::init_messaging_service()::{lambda(std::vector<frozen_mutation, std::a
| | | | 0x2b7434
| | | |
| | | |--14.29%-- _ZN8futurizeI6futureIJSt10unique_ptrIN4cql317update_parametersESt14default_deleteIS3_EEEEE5applyIZNS2_10statements22modification_statement22make_update_parametersERN7seastar7shardedIN7service13storage_proxyEEE13lw_shared_ptrISt6vectorI13partition_keySaISK_EEESI_I26exploded_clustering_prefixERKNS2_13query_optionsEblEUlT_E_JNSt12experimental15fundamentals_v18optionalINS3_13prefetch_dataEEEEEES7_OST_OSt5tupleIJDpT0_EE
| | | | cql3::statements::modification_statement::make_update_parameters
| | | | cql3::statements::modification_statement::get_mutations
| | | | cql3::statements::modification_statement::execute_without_condition
| | | | cql3::query_options::query_options
| | | | 0x7f4ad6bf80e0
| | | |
| | | --14.29%-- database::apply_in_memory
| | | database::do_apply
| | | _ZN12continuationIZN6futureIJEE4thenIZN8database5applyERK15frozen_mutationEUlvE_S1_EET0_OT_EUlSA_E_JEE3runEv
| | | reactor::del_timer
| | | 0x6090000e2040
| | |
| | |--1.04%-- memory::small_pool::allocate
| | | |
| | | |--14.29%-- 0x5257c379469d9
| | | |
| | | |--14.29%-- 0x609002b9fe98
| | | |
| | | |--14.29%-- 0x13c8b90
| | | |
| | | |--14.29%-- 0x60f000190710
| | | |
| | | |--14.29%-- 0x25
| | | |
| | | |--14.29%-- 0x7f4ad6bf84c0
| | | |
| | | --14.29%-- 0x7f4ad53f81f0
| | |
| | |--0.89%-- db::serializer<atomic_cell_view>::serializer
| | | mutation_partition_serializer::write_without_framing
| | | frozen_mutation::frozen_mutation
| | | frozen_mutation::frozen_mutation
| | |
| | |--0.89%-- do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>
| | | do_void_futurize_apply<void do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>(transport::cql_server::connection::process()::{lambda()#1}&, transport::cql_server::connection::process()::{lambda()#2}&&, promise<>)::{lambda(future<>)#1}, promise<> >
| | | do_void_futurize_apply<void do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>(transport::cql_server::connection::process()::{lambda()#1}&, transport::cql_server::connection::process()::{lambda()#2}&&, promise<>)::{lambda(future<>)#1}, promise<> >
| | | 0x60f0000c3000
| | |
| | |--0.89%-- futurize<future<> >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}::operator()(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&) const::{lambda(temporary_buffer<char>)#1}, temporary_buffer>
| | | futurize<future<> >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}, future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> > >
| | | transport::cql_server::connection::process_request
| | | do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>
| | | do_void_futurize_apply<void do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>(transport::cql_server::connection::process()::{lambda()#1}&, transport::cql_server::connection::process()::{lambda()#2}&&, promise<>)::{lambda(future<>)#1}, promise<> >
| | | do_void_futurize_apply<void do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>(transport::cql_server::connection::process()::{lambda()#1}&, transport::cql_server::connection::process()::{lambda()#2}&&, promise<>)::{lambda(future<>)#1}, promise<> >
| | | |
| | | |--83.33%-- 0x6090000c3000
| | | |
| | | --16.67%-- 0x600000044400
| | |
| | |--0.89%-- std::_Function_handler<void (), reactor::run()::{lambda()#8}>::_M_invoke
| | | |
| | | |--50.00%-- reactor::run
| | | | smp::configure(boost::program_options::variables_map)::{lambda()#1}::operator()
| | | | continuation<future<temporary_buffer<char> > future<>::then<future<temporary_buffer<char> > file::dma_read_bulk<char>(unsigned long, unsigned long)::{lambda(unsigned long)#1}::operator()(unsigned long)::{lambda()#3}, future<temporary_buffer<char> > >(future<temporary_buffer<char> > file::dma_read_bulk<char>(unsigned long, unsigned long)::{lambda(unsigned long)#1}::operator()(unsigned long)::{lambda()#3}&&)::{lambda(future<temporary_buffer<char> >)#1}>::run
| | | | 0x600000043d00
| | | |
| | | --50.00%-- reactor::signals::signal_handler::signal_handler
| | | 0x3e8
| | |
| | |--0.74%-- db::commitlog::segment::allocate
| | | |
| | | --100.00%-- db::commitlog::add
| | | database::do_apply
| | | |
| | | |--75.00%-- database::apply
| | | | smp_message_queue::async_work_item<future<> seastar::sharded<database>::invoke_on<service::storage_proxy::mutate_locally(mutation const&)::{lambda(database&)#1}, future<> >(unsigned int, service::storage_proxy::mutate_locally(mutation const&)::{lambda(database&)#1}&&)::{lambda()#1}>::process
| | | | smp_message_queue::process_queue<2ul, smp_message_queue::process_incoming()::{lambda(smp_message_queue::work_item*)#1}>
| | | | boost::lockfree::detail::ringbuffer_base<smp_message_queue::work_item*>::pop
| | | | boost::program_options::variables_map::get
| | | |
| | | --25.00%-- _ZN12continuationIZN6futureIJEE4thenIZN8database5applyERK15frozen_mutationEUlvE_S1_EET0_OT_EUlSA_E_JEE3runEv
| | | reactor::del_timer
| | | 0x60b0000e2040
| | |
| | |--0.74%-- service::storage_proxy::create_write_response_handler
| | |
| | |--0.74%-- transport::cql_server::connection::process_request_one
| | | futurize<future<std::pair<foreign_ptr<shared_ptr<transport::cql_server::response> >, service::client_state> > >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}::operator()(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&) const::{lambda(temporary_buffer<char>)#1}::operator()(temporary_buffer) const::{lambda()#1}::operator()()::{lambda()#1}&>
| | | futurize<future<> >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}::operator()(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&) const::{lambda(temporary_buffer<char>)#1}, temporary_buffer>
| | | futurize<future<> >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}, future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> > >
| | | |
| | | |--80.00%-- transport::cql_server::connection::process_request
| | | | do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>
| | | | do_void_futurize_apply<void do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>(transport::cql_server::connection::process()::{lambda()#1}&, transport::cql_server::connection::process()::{lambda()#2}&&, promise<>)::{lambda(future<>)#1}, promise<> >
| | | | do_void_futurize_apply<void do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>(transport::cql_server::connection::process()::{lambda()#1}&, transport::cql_server::connection::process()::{lambda()#2}&&, promise<>)::{lambda(future<>)#1}, promise<> >
| | | | 0x60a0000c3000
| | | |
| | | --20.00%-- 0x8961de
| | |
| | |--0.74%-- compound_type<(allow_prefixes)0>::compare
| | | |
| | | |--20.00%-- 0x6030056c0f20
| | | |
| | | |--20.00%-- boost::intrusive::bstbase2<boost::intrusive::mhtraits<rows_entry, boost::intrusive::set_member_hook<void, void, void, void>, &rows_entry::_link>, rows_entry::compare, (boost::intrusive::algo_types)5, boost::intrusive::detail::default_header_holder<boost::intrusive::rbtree_node_traits<void*, false> > >::find
| | | | mutation_partition::clustered_row
| | | | mutation::set_clustered_cell
| | | | cql3::constants::setter::execute
| | | | cql3::statements::update_statement::add_update_for_key
| | | | _ZN8futurizeI6futureIJSt6vectorI8mutationSaIS2_EEEEE5applyIZN4cql310statements22modification_statement13get_mutationsERN7seastar7shardedIN7service13storage_proxyEEERKNS8_13query_optionsEblEUlT_E_JSt10unique_ptrINS8_17update_parametersESt14default_deleteISN_EEEEES5_OSK_OSt5tupleIJDpT0_EE
| | | | cql3::statements::modification_statement::get_mutations
| | | | cql3::statements::modification_statement::execute_without_condition
| | | | cql3::query_options::query_options
| | | | 0x7f4adb3f80e0
| | | |
| | | |--20.00%-- compound_type<(allow_prefixes)0>::compare
| | | |
| | | |--20.00%-- mutation_partition::clustered_row
| | | | boost::intrusive::bstree_impl<boost::intrusive::mhtraits<rows_entry, boost::intrusive::set_member_hook<void, void, void, void>, &rows_entry::_link>, rows_entry::compare, unsigned long, true, (boost::intrusive::algo_types)5, boost::intrusive::detail::default_header_holder<boost::intrusive::rbtree_node_traits<void*, false> > >::insert_unique
| | | | boost::intrusive::bstree_algorithms<boost::intrusive::rbtree_node_traits<void*, false> >::prev_node
| | | | 0x12d
| | | |
| | | --20.00%-- 0x60f00052daf0
| | |
| | |--0.74%-- __memmove_ssse3_back
| | | |
| | | |--40.00%-- output_stream<char>::write
| | | | |
| | | | |--50.00%-- transport::cql_server::response::output
| | | | | futurize<future<> >::apply<transport::cql_server::connection::write_response(foreign_ptr<shared_ptr<transport::cql_server::response> >&&)::{lambda()#1}>
| | | | |
| | | | --50.00%-- 0x7c7fb2
| | | | 0x5257c37847fa0
| | | |
| | | |--20.00%-- transport::cql_server::connection::read_short_bytes
| | | | transport::cql_server::connection::process_query
| | | | 0x7f4ada7f86f0
| | | |
| | | |--20.00%-- transport::cql_server::response::output
| | | | futurize<future<> >::apply<transport::cql_server::connection::write_response(foreign_ptr<shared_ptr<transport::cql_server::response> >&&)::{lambda()#1}>
| | | | 0x2
| | | |
| | | --20.00%-- smp_message_queue::flush_response_batch
| | | boost::lockfree::detail::ringbuffer_base<smp_message_queue::work_item*>::pop
| | | boost::program_options::variables_map::get
| | |
| | |--0.74%-- syscall_work_queue::work_item_returning<syscall_result_extra<stat>, reactor::file_size(basic_sstring<char, unsigned int, 15u>)::{lambda()#1}>::~work_item_returning
| | | |
| | | |--60.00%-- 0x6130000c3000
| | | |
| | | |--20.00%-- 0x608001fe59a0
| | | |
| | | --20.00%-- 0x16
| | |
| | |--0.74%-- __memset_sse2
| | | |
| | | |--40.00%-- std::_Hashtable<range<dht::token>, std::pair<range<dht::token> const, std::unordered_set<gms::inet_address, std::hash<gms::inet_address>, std::equal_to<gms::inet_address>, std::allocator<gms::inet_address> > >, std::allocator<std::pair<range<dht::token> const, std::unordered_set<gms::inet_address, std::hash<gms::inet_address>, std::equal_to<gms::inet_address>, std::allocator<gms::inet_address> > > >, std::__detail::_Select1st, std::equal_to<range<dht::token> >, std::hash<range<dht::token> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::~_Hashtable
| | | | locator::token_metadata::pending_endpoints_for
| | | | service::storage_proxy::create_write_response_handler
| | | | service::storage_proxy::mutate_prepare<std::vector<mutation, std::allocator<mutation> >, service::storage_proxy::mutate_prepare(std::vector<mutation, std::allocator<mutation> >&, db::consistency_level, db::write_type)::{lambda(mutation const&, db::consistency_level, db::write_type)#1}>
| | | | service::storage_proxy::mutate
| | | | service::storage_proxy::mutate_with_triggers
| | | | cql3::statements::modification_statement::execute_without_condition
| | | | cql3::statements::modification_statement::execute
| | | | cql3::query_processor::process_statement
| | | | transport::cql_server::connection::process_execute
| | | | transport::cql_server::connection::process_request_one
| | | | futurize<future<std::pair<foreign_ptr<shared_ptr<transport::cql_server::response> >, service::client_state> > >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}::operator()(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&) const::{lambda(temporary_buffer<char>)#1}::operator()(temporary_buffer) const::{lambda()#1}::operator()()::{lambda()#1}&>
| | | | futurize<future<> >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}::operator()(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&) const::{lambda(temporary_buffer<char>)#1}, temporary_buffer>
| | | | futurize<future<> >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}, future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> > >
| | | | transport::cql_server::connection::process_request
| | | | do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>
| | | | do_void_futurize_apply<void do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>(transport::cql_server::connection::process()::{lambda()#1}&, transport::cql_server::connection::process()::{lambda()#2}&&, promise<>)::{lambda(future<>)#1}, promise<> >
| | | | do_void_futurize_apply<void do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>(transport::cql_server::connection::process()::{lambda()#1}&, transport::cql_server::connection::process()::{lambda()#2}&&, promise<>)::{lambda(future<>)#1}, promise<> >
| | | | 0x6020000c3000
| | | |
| | | |--40.00%-- service::digest_read_resolver::~digest_read_resolver
| | | | |
| | | | --100.00%-- 0x610002612b50
| | | |
| | | --20.00%-- std::_Hashtable<basic_sstring<char, unsigned int, 15u>, std::pair<basic_sstring<char, unsigned int, 15u> const, std::vector<gms::inet_address, std::allocator<gms::inet_address> > >, std::allocator<std::pair<basic_sstring<char, unsigned int, 15u> const, std::vector<gms::inet_address, std::allocator<gms::inet_address> > > >, std::__detail::_Select1st, std::equal_to<basic_sstring<char, unsigned int, 15u> >, std::hash<basic_sstring<char, unsigned int, 15u> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::~_Hashtable
| | | service::storage_proxy::send_to_live_endpoints
| | | parallel_for_each<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, service::storage_proxy::mutate_begin(std::vector<unsigned long, std::allocator<unsigned long> >, db::consistency_level)::{lambda(unsigned long)#1}>
| | | service::storage_proxy::mutate
| | | service::storage_proxy::mutate_with_triggers
| | | cql3::statements::modification_statement::execute_without_condition
| | | cql3::statements::modification_statement::execute
| | | cql3::query_processor::process_statement
| | | transport::cql_server::connection::process_execute
| | | transport::cql_server::connection::process_request_one
| | | futurize<future<std::pair<foreign_ptr<shared_ptr<transport::cql_server::response> >, service::client_state> > >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}::operator()(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&) const::{lambda(temporary_buffer<char>)#1}::operator()(temporary_buffer) const::{lambda()#1}::operator()()::{lambda()#1}&>
| | | futurize<future<> >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}::operator()(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&) const::{lambda(temporary_buffer<char>)#1}, temporary_buffer>
| | | futurize<future<> >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}, future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> > >
| | | transport::cql_server::connection::process_request
| | | do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>
| | | do_void_futurize_apply<void do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>(transport::cql_server::connection::process()::{lambda()#1}&, transport::cql_server::connection::process()::{lambda()#2}&&, promise<>)::{lambda(future<>)#1}, promise<> >
| | | do_void_futurize_apply<void do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>(transport::cql_server::connection::process()::{lambda()#1}&, transport::cql_server::connection::process()::{lambda()#2}&&, promise<>)::{lambda(future<>)#1}, promise<> >
| | | 0x6070000c3000
| | |
| | |--0.74%-- reactor::del_timer
| | | |
| | | |--80.00%-- 0x60a0000e2040
| | | |
| | | --20.00%-- 0x6080000c3db0
| | |
| | |--0.59%-- unimplemented::operator<<
| | | |
| | | |--25.00%-- _ZN12continuationIZN6futureIJ10shared_ptrIN9transport8messages14result_messageEEEE4thenIZN4cql315query_processor17process_statementES1_INS8_13cql_statementEERN7service11query_stateERKNS8_13query_optionsEEUlT_E_S6_EET0_OSI_EUlSL_E_JS5_EED2Ev
| | | | 0x600100000008
| | | |
| | | |--25.00%-- floating_type_impl<float>::from_string
| | | |
| | | |--25.00%-- 0x60e0000e4c10
| | | |
| | | --25.00%-- _ZN12continuationIZN6futureIJ10shared_ptrIN9transport8messages14result_messageEEEE4thenIZN4cql315query_processor17process_statementES1_INS8_13cql_statementEERN7service11query_stateERKNS8_13query_optionsEEUlT_E_S6_EET0_OSI_EUlSL_E_JS5_EED2Ev
| | | 0x600100000008
| | |
| | |--0.59%-- std::_Hashtable<unsigned long, std::pair<unsigned long const, service::storage_proxy::rh_entry>, std::allocator<std::pair<unsigned long const, service::storage_proxy::rh_entry> >, std::__detail::_Select1st, std::equal_to<unsigned long>, std::hash<unsigned long>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_insert_unique_node
| | | service::storage_proxy::register_response_handler
| | | service::storage_proxy::create_write_response_handler
| | | service::storage_proxy::create_write_response_handler
| | | service::storage_proxy::mutate_prepare<std::vector<mutation, std::allocator<mutation> >, service::storage_proxy::mutate_prepare(std::vector<mutation, std::allocator<mutation> >&, db::consistency_level, db::write_type)::{lambda(mutation const&, db::consistency_level, db::write_type)#1}>
| | | service::storage_proxy::mutate
| | | service::storage_proxy::mutate_with_triggers
| | | cql3::statements::modification_statement::execute_without_condition
| | | cql3::statements::modification_statement::execute
| | | cql3::query_processor::process_statement
| | | transport::cql_server::connection::process_execute
| | | transport::cql_server::connection::process_request_one
| | | futurize<future<std::pair<foreign_ptr<shared_ptr<transport::cql_server::response> >, service::client_state> > >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}::operator()(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&) const::{lambda(temporary_buffer<char>)#1}::operator()(temporary_buffer) const::{lambda()#1}::operator()()::{lambda()#1}&>
| | | futurize<future<> >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}::operator()(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&) const::{lambda(temporary_buffer<char>)#1}, temporary_buffer>
| | | futurize<future<> >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}, future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> > >
| | | transport::cql_server::connection::process_request
| | | do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>
| | | do_void_futurize_apply<void do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>(transport::cql_server::connection::process()::{lambda()#1}&, transport::cql_server::connection::process()::{lambda()#2}&&, promise<>)::{lambda(future<>)#1}, promise<> >
| | | do_void_futurize_apply<void do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>(transport::cql_server::connection::process()::{lambda()#1}&, transport::cql_server::connection::process()::{lambda()#2}&&, promise<>)::{lambda(future<>)#1}, promise<> >
| | | 0x60b0000c3000
| | |
| | |--0.59%-- mutation::set_clustered_cell
| | | |
| | | |--75.00%-- 0xa
| | | |
| | | --25.00%-- cql3::constants::setter::execute
| | | cql3::statements::update_statement::add_update_for_key
| | | _ZN8futurizeI6futureIJSt6vectorI8mutationSaIS2_EEEEE5applyIZN4cql310statements22modification_statement13get_mutationsERN7seastar7shardedIN7service13storage_proxyEEERKNS8_13query_optionsEblEUlT_E_JSt10unique_ptrINS8_17update_parametersESt14default_deleteISN_EEEEES5_OSK_OSt5tupleIJDpT0_EE
| | | cql3::statements::modification_statement::get_mutations
| | | cql3::statements::modification_statement::execute_without_condition
| | | cql3::query_options::query_options
| | | 0x7f4ad89f80e0
| | |
| | |--0.59%-- memory::small_pool::small_pool
| | | |
| | | |--25.00%-- memory::stats
| | | | boost::program_options::variables_map::get
| | | |
| | | |--25.00%-- memory::reclaimer::~reclaimer
| | | | 0x1e
| | | |
| | | |--25.00%-- memory::allocate_aligned
| | | |
| | | --25.00%-- memory::small_pool::add_more_objects
| | | memory::small_pool::add_more_objects
| | | 0x6100000e0310
| | |
| | |--0.59%-- __memcpy_sse2_unaligned
| | | |
| | | |--50.00%-- mutation_partition_applier::accept_row_cell
| | | | mutation_partition_view::accept
| | | | boost::intrusive::bstree_algorithms<boost::intrusive::rbtree_node_traits<void*, false> >::prev_node
| | | | 0x12d
| | | |
| | | |--25.00%-- scanning_reader::operator()
| | | | sstables::sstable::do_write_components
| | | | sstables::sstable::prepare_write_components
| | | | std::_Function_handler<void (), futurize<std::result_of<std::decay<sstables::sstable::write_components(mutation_reader, unsigned long, lw_shared_ptr<schema const>, unsigned long)::{lambda()#1}>::type ()>::type>::type seastar::async<sstables::sstable::write_components(mutation_reader, unsigned long, lw_shared_ptr<schema const>, unsigned long)::{lambda()#1}>(std::decay&&, (std::decay<sstables::sstable::write_components(mutation_reader, unsigned long, lw_shared_ptr<schema const>, unsigned long)::{lambda()#1}>::type&&)...)::{lambda(futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type> seastar::async<{lambda()#1}>(futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type>::type, std::decay<{lambda()#1}>::type&&)::work&)#1}::operator()(futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type> seastar::async<{lambda()#1}>(futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type>::type, std::decay<{lambda()#1}>::type&&)::work)::{lambda()#1}>::_M_invoke
| | | | _GLOBAL__sub_I__ZN12app_templateC2Ev
| | | |
| | | --25.00%-- memtable::find_or_create_partition_slow
| | | memtable::apply
| | | database::apply_in_memory
| | | database::do_apply
| | | database::apply
| | | smp_message_queue::async_work_item<future<> seastar::sharded<database>::invoke_on<service::storage_proxy::mutate_locally(mutation const&)::{lambda(database&)#1}, future<> >(unsigned int, service::storage_proxy::mutate_locally(mutation const&)::{lambda(database&)#1}&&)::{lambda()#1}>::process
| | | smp_message_queue::process_queue<2ul, smp_message_queue::process_incoming()::{lambda(smp_message_queue::work_item*)#1}>
| | | boost::lockfree::detail::ringbuffer_base<smp_message_queue::work_item*>::pop
| | | boost::program_options::variables_map::get
| | |
| | |--0.59%-- smp_message_queue::flush_response_batch
| | | |
| | | |--25.00%-- boost::lockfree::detail::ringbuffer_base<smp_message_queue::work_item*>::pop
| | | | boost::program_options::variables_map::get
| | | |
| | | |--25.00%-- 0x13
| | | |
| | | |--25.00%-- 0x7f4ad5ff8f40
| | | |
| | | --25.00%-- reactor::run
| | | smp::configure(boost::program_options::variables_map)::{lambda()#1}::operator()
| | | continuation<future<temporary_buffer<char> > future<>::then<future<temporary_buffer<char> > file::dma_read_bulk<char>(unsigned long, unsigned long)::{lambda(unsigned long)#1}::operator()(unsigned long)::{lambda()#3}, future<temporary_buffer<char> > >(future<temporary_buffer<char> > file::dma_read_bulk<char>(unsigned long, unsigned long)::{lambda(unsigned long)#1}::operator()(unsigned long)::{lambda()#3}&&)::{lambda(future<temporary_buffer<char> >)#1}>::run
| | | 0x600000043d00
| | --54.38%-- [...]
| |
| |--14.26%-- schedule_timeout
| | |
| | |--38.52%-- wait_for_completion
| | | |
| | | |--90.07%-- flush_work
| | | | xlog_cil_force_lsn
| | | | |
| | | | |--96.85%-- _xfs_log_force_lsn
| | | | | |
| | | | | |--79.67%-- xfs_file_fsync
| | | | | | vfs_fsync_range
| | | | | | do_fsync
| | | | | | sys_fdatasync
| | | | | | entry_SYSCALL_64_fastpath
| | | | | | |
| | | | | | --100.00%-- 0x7f4ade4212ad
| | | | | | syscall_work_queue::work_item_returning<syscall_result<int>, posix_file_impl::flush()::{lambda()#1}>::process
| | | | | | 0x6030000c3ec0
| | | | | |
| | | | | --20.33%-- xfs_dir_fsync
| | | | | vfs_fsync_range
| | | | | do_fsync
| | | | | sys_fdatasync
| | | | | entry_SYSCALL_64_fastpath
| | | | | |
| | | | | --100.00%-- 0x7f4ade4212ad
| | | | | syscall_work_queue::work_item_returning<syscall_result<int>, posix_file_impl::flush()::{lambda()#1}>::process
| | | | | 0x6040000c3ec0
| | | | |
| | | | --3.15%-- _xfs_log_force
| | | | xfs_log_force
| | | | xfs_buf_lock
| | | | _xfs_buf_find
| | | | xfs_buf_get_map
| | | | xfs_trans_get_buf_map
| | | | xfs_btree_get_bufl
| | | | xfs_bmap_extents_to_btree
| | | | xfs_bmap_add_extent_hole_real
| | | | xfs_bmapi_write
| | | | xfs_iomap_write_direct
| | | | __xfs_get_blocks
| | | | xfs_get_blocks_direct
| | | | do_blockdev_direct_IO
| | | | __blockdev_direct_IO
| | | | xfs_vm_direct_IO
| | | | xfs_file_dio_aio_write
| | | | xfs_file_write_iter
| | | | aio_run_iocb
| | | | do_io_submit
| | | | sys_io_submit
| | | | entry_SYSCALL_64_fastpath
| | | | io_submit
| | | | 0x46d98a
| | | |
| | | --9.93%-- submit_bio_wait
| | | blkdev_issue_flush
| | | xfs_blkdev_issue_flush
| | | xfs_file_fsync
| | | vfs_fsync_range
| | | do_fsync
| | | sys_fdatasync
| | | entry_SYSCALL_64_fastpath
| | | |
| | | --100.00%-- 0x7f4ade4212ad
| | | syscall_work_queue::work_item_returning<syscall_result<int>, posix_file_impl::flush()::{lambda()#1}>::process
| | | 0x6030000c3ec0
| | |
| | |--32.79%-- io_schedule_timeout
| | | bit_wait_io
| | | __wait_on_bit
| | | |
| | | |--51.67%-- wait_on_page_bit
| | | | |
| | | | |--95.16%-- filemap_fdatawait_range
| | | | | filemap_write_and_wait_range
| | | | | xfs_file_fsync
| | | | | vfs_fsync_range
| | | | | do_fsync
| | | | | sys_fdatasync
| | | | | entry_SYSCALL_64_fastpath
| | | | | 0x7f4ade4212ad
| | | | | syscall_work_queue::work_item_returning<syscall_result<int>, posix_file_impl::flush()::{lambda()#1}>::process
| | | | | 0x60b0000c3ec0
| | | | |
| | | | --4.84%-- __migration_entry_wait
| | | | migration_entry_wait
| | | | handle_mm_fault
| | | | __do_page_fault
| | | | do_page_fault
| | | | page_fault
| | | | std::_Function_handler<void (), httpd::http_server::_date_format_timer::{lambda()#1}>::_M_invoke
| | | | |
| | | | --100.00%-- service::storage_proxy::mutate_prepare<std::vector<mutation, std::allocator<mutation> >, service::storage_proxy::mutate_prepare(std::vector<mutation, std::allocator<mutation> >&, db::consistency_level, db::write_type)::{lambda(mutation const&, db::consistency_level, db::write_type)#1}>
| | | | service::storage_proxy::mutate
| | | | service::storage_proxy::mutate_with_triggers
| | | | cql3::statements::modification_statement::execute_without_condition
| | | | cql3::statements::modification_statement::execute
| | | | cql3::query_processor::process_statement
| | | | transport::cql_server::connection::process_execute
| | | | transport::cql_server::connection::process_request_one
| | | | futurize<future<std::pair<foreign_ptr<shared_ptr<transport::cql_server::response> >, service::client_state> > >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}::operator()(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&) const::{lambda(temporary_buffer<char>)#1}::operator()(temporary_buffer) const::{lambda()#1}::operator()()::{lambda()#1}&>
| | | | futurize<future<> >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}::operator()(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&) const::{lambda(temporary_buffer<char>)#1}, temporary_buffer>
| | | | futurize<future<> >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}, future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> > >
| | | | transport::cql_server::connection::process_request
| | | | do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>
| | | | do_void_futurize_apply<void do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>(transport::cql_server::connection::process()::{lambda()#1}&, transport::cql_server::connection::process()::{lambda()#2}&&, promise<>)::{lambda(future<>)#1}, promise<> >
| | | | do_void_futurize_apply<void do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>(transport::cql_server::connection::process()::{lambda()#1}&, transport::cql_server::connection::process()::{lambda()#2}&&, promise<>)::{lambda(future<>)#1}, promise<> >
| | | | 0x6140000c3000
| | | |
| | | --48.33%-- out_of_line_wait_on_bit
| | | block_truncate_page
| | | xfs_setattr_size
| | | xfs_vn_setattr
| | | notify_change
| | | do_truncate
| | | do_sys_ftruncate.constprop.15
| | | sys_ftruncate
| | | entry_SYSCALL_64_fastpath
| | | __GI___ftruncate64
| | | syscall_work_queue::work_item_returning<syscall_result_extra<stat>, posix_file_impl::stat()::{lambda()#1}>::process
| | | |
| | | |--13.79%-- 0x7f4ad29ff700
| | | |
| | | |--13.79%-- 0x7f4acdbff700
| | | |
| | | |--12.07%-- 0x7f4ad05ff700
| | | |
| | | |--12.07%-- 0x7f4acedff700
| | | |
| | | |--10.34%-- 0x7f4ad0bff700
| | | |
| | | |--6.90%-- 0x7f4ad2fff700
| | | |
| | | |--6.90%-- 0x7f4ad11ff700
| | | |
| | | |--6.90%-- 0x7f4acf9ff700
| | | |
| | | |--6.90%-- 0x7f4acf3ff700
| | | |
| | | |--6.90%-- 0x7f4ace7ff700
| | | |
| | | |--1.72%-- 0x7f4ad17ff700
| | | |
| | | --1.72%-- 0x7f4aca5ff700
| | |
| | --28.69%-- __down
| | down
| | xfs_buf_lock
| | _xfs_buf_find
| | xfs_buf_get_map
| | |
| | |--97.14%-- xfs_buf_read_map
| | | xfs_trans_read_buf_map
| | | |
| | | |--98.04%-- xfs_read_agf
| | | | xfs_alloc_read_agf
| | | | xfs_alloc_fix_freelist
| | | | |
| | | | |--93.00%-- xfs_free_extent
| | | | | xfs_bmap_finish
| | | | | xfs_itruncate_extents
| | | | | |
| | | | | |--87.10%-- xfs_inactive_truncate
| | | | | | xfs_inactive
| | | | | | xfs_fs_evict_inode
| | | | | | evict
| | | | | | iput
| | | | | | __dentry_kill
| | | | | | dput
| | | | | | __fput
| | | | | | ____fput
| | | | | | task_work_run
| | | | | | do_notify_resume
| | | | | | int_signal
| | | | | | __libc_close
| | | | | | std::experimental::fundamentals_v1::bad_optional_access::~bad_optional_access
| | | | | |
| | | | | --12.90%-- xfs_setattr_size
| | | | | xfs_vn_setattr
| | | | | notify_change
| | | | | do_truncate
| | | | | do_sys_ftruncate.constprop.15
| | | | | sys_ftruncate
| | | | | entry_SYSCALL_64_fastpath
| | | | | |
| | | | | --100.00%-- __GI___ftruncate64
| | | | | syscall_work_queue::work_item_returning<syscall_result_extra<stat>, posix_file_impl::stat()::{lambda()#1}>::process
| | | | | |
| | | | | |--20.00%-- 0x7f4ad0bff700
| | | | | |
| | | | | |--20.00%-- 0x7f4acedff700
| | | | | |
| | | | | |--10.00%-- 0x7f4ad2fff700
| | | | | |
| | | | | |--10.00%-- 0x7f4ad17ff700
| | | | | |
| | | | | |--10.00%-- 0x7f4ad11ff700
| | | | | |
| | | | | |--10.00%-- 0x7f4ad05ff700
| | | | | |
| | | | | |--10.00%-- 0x7f4acf3ff700
| | | | | |
| | | | | --10.00%-- 0x7f4acdbff700
| | | | |
| | | | --7.00%-- xfs_alloc_vextent
| | | | xfs_bmap_btalloc
| | | | xfs_bmap_alloc
| | | | xfs_bmapi_write
| | | | xfs_iomap_write_direct
| | | | __xfs_get_blocks
| | | | xfs_get_blocks_direct
| | | | do_blockdev_direct_IO
| | | | __blockdev_direct_IO
| | | | xfs_vm_direct_IO
| | | | xfs_file_dio_aio_write
| | | | xfs_file_write_iter
| | | | aio_run_iocb
| | | | do_io_submit
| | | | sys_io_submit
| | | | entry_SYSCALL_64_fastpath
| | | | io_submit
| | | | 0x46d98a
| | | |
| | | --1.96%-- xfs_read_agi
| | | xfs_iunlink_remove
| | | xfs_ifree
| | | xfs_inactive_ifree
| | | xfs_inactive
| | | xfs_fs_evict_inode
| | | evict
| | | iput
| | | __dentry_kill
| | | dput
| | | __fput
| | | ____fput
| | | task_work_run
| | | do_notify_resume
| | | int_signal
| | | __libc_close
| | | std::experimental::fundamentals_v1::bad_optional_access::~bad_optional_access
| | |
| | --2.86%-- xfs_trans_get_buf_map
| | xfs_btree_get_bufl
| | xfs_bmap_extents_to_btree
| | xfs_bmap_add_extent_hole_real
| | xfs_bmapi_write
| | xfs_iomap_write_direct
| | __xfs_get_blocks
| | xfs_get_blocks_direct
| | do_blockdev_direct_IO
| | __blockdev_direct_IO
| | xfs_vm_direct_IO
| | xfs_file_dio_aio_write
| | xfs_file_write_iter
| | aio_run_iocb
| | do_io_submit
| | sys_io_submit
| | entry_SYSCALL_64_fastpath
| | io_submit
| | 0x46d98a
| |
| |--13.48%-- eventfd_ctx_read
| | eventfd_read
| | __vfs_read
| | vfs_read
| | sys_read
| | entry_SYSCALL_64_fastpath
| | 0x7f4ade6f754d
| | smp_message_queue::respond
| | 0xffffffffffffffff
| |
| |--7.83%-- md_flush_request
| | raid0_make_request
| | md_make_request
| | generic_make_request
| | submit_bio
| | |
| | |--92.54%-- submit_bio_wait
| | | blkdev_issue_flush
| | | xfs_blkdev_issue_flush
| | | xfs_file_fsync
| | | vfs_fsync_range
| | | do_fsync
| | | sys_fdatasync
| | | entry_SYSCALL_64_fastpath
| | | |
| | | --100.00%-- 0x7f4ade4212ad
| | | syscall_work_queue::work_item_returning<syscall_result<int>, posix_file_impl::flush()::{lambda()#1}>::process
| | | 0x6010000c3ec0
| | |
| | --7.46%-- _xfs_buf_ioapply
| | xfs_buf_submit
| | xlog_bdstrat
| | xlog_sync
| | xlog_state_release_iclog
| | |
| | |--73.33%-- _xfs_log_force_lsn
| | | xfs_file_fsync
| | | vfs_fsync_range
| | | do_fsync
| | | sys_fdatasync
| | | entry_SYSCALL_64_fastpath
| | | |
| | | --100.00%-- 0x7f4ade4212ad
| | | syscall_work_queue::work_item_returning<syscall_result<int>, posix_file_impl::flush()::{lambda()#1}>::process
| | | 0x6080000c3ec0
| | |
| | --26.67%-- _xfs_log_force
| | xfs_log_force
| | xfs_buf_lock
| | _xfs_buf_find
| | xfs_buf_get_map
| | xfs_trans_get_buf_map
| | xfs_btree_get_bufl
| | xfs_bmap_extents_to_btree
| | xfs_bmap_add_extent_hole_real
| | xfs_bmapi_write
| | xfs_iomap_write_direct
| | __xfs_get_blocks
| | xfs_get_blocks_direct
| | do_blockdev_direct_IO
| | __blockdev_direct_IO
| | xfs_vm_direct_IO
| | xfs_file_dio_aio_write
| | xfs_file_write_iter
| | aio_run_iocb
| | do_io_submit
| | sys_io_submit
| | entry_SYSCALL_64_fastpath
| | io_submit
| | 0x46d98a
| |
| |--5.53%-- _xfs_log_force_lsn
| | |
| | |--80.28%-- xfs_file_fsync
| | | vfs_fsync_range
| | | do_fsync
| | | sys_fdatasync
| | | entry_SYSCALL_64_fastpath
| | | |
| | | --100.00%-- 0x7f4ade4212ad
| | | syscall_work_queue::work_item_returning<syscall_result<int>, posix_file_impl::flush()::{lambda()#1}>::process
| | | |
| | | |--97.92%-- 0x60d0000c3ec0
| | | |
| | | |--1.04%-- 0x6020000c3ec0
| | | |
| | | --1.04%-- 0x600000557ec0
| | |
| | --19.72%-- xfs_dir_fsync
| | vfs_fsync_range
| | do_fsync
| | sys_fdatasync
| | entry_SYSCALL_64_fastpath
| | |
| | --100.00%-- 0x7f4ade4212ad
| | syscall_work_queue::work_item_returning<syscall_result<int>, posix_file_impl::flush()::{lambda()#1}>::process
| | 0x6040000c3ec0
| |
| |--1.25%-- rwsem_down_read_failed
| | call_rwsem_down_read_failed
| | |
| | |--90.62%-- xfs_ilock
| | | |
| | | |--86.21%-- xfs_ilock_data_map_shared
| | | | __xfs_get_blocks
| | | | xfs_get_blocks_direct
| | | | do_blockdev_direct_IO
| | | | __blockdev_direct_IO
| | | | xfs_vm_direct_IO
| | | | xfs_file_dio_aio_write
| | | | xfs_file_write_iter
| | | | aio_run_iocb
| | | | do_io_submit
| | | | sys_io_submit
| | | | entry_SYSCALL_64_fastpath
| | | | |
| | | | --100.00%-- io_submit
| | | | 0x46d98a
| | | |
| | | |--6.90%-- xfs_file_fsync
| | | | vfs_fsync_range
| | | | do_fsync
| | | | sys_fdatasync
| | | | entry_SYSCALL_64_fastpath
| | | | 0x7f4ade4212ad
| | | | syscall_work_queue::work_item_returning<syscall_result<int>, posix_file_impl::flush()::{lambda()#1}>::process
| | | | 0x6090000c3ec0
| | | |
| | | --6.90%-- xfs_dir_fsync
| | | vfs_fsync_range
| | | do_fsync
| | | sys_fdatasync
| | | entry_SYSCALL_64_fastpath
| | | 0x7f4ade4212ad
| | | syscall_work_queue::work_item_returning<syscall_result<int>, posix_file_impl::flush()::{lambda()#1}>::process
| | | 0x6070000c3ec0
| | |
| | --9.38%-- xfs_log_commit_cil
| | __xfs_trans_commit
| | xfs_trans_commit
| | |
| | |--33.33%-- xfs_setattr_size
| | | xfs_vn_setattr
| | | notify_change
| | | do_truncate
| | | do_sys_ftruncate.constprop.15
| | | sys_ftruncate
| | | entry_SYSCALL_64_fastpath
| | | __GI___ftruncate64
| | | syscall_work_queue::work_item_returning<syscall_result_extra<stat>, posix_file_impl::stat()::{lambda()#1}>::process
| | | 0x7f4acedff700
| | |
| | |--33.33%-- xfs_vn_update_time
| | | file_update_time
| | | xfs_file_aio_write_checks
| | | xfs_file_dio_aio_write
| | | xfs_file_write_iter
| | | aio_run_iocb
| | | do_io_submit
| | | sys_io_submit
| | | entry_SYSCALL_64_fastpath
| | | io_submit
| | | 0x46d98a
| | |
| | --33.33%-- xfs_bmap_add_attrfork
| | xfs_attr_set
| | xfs_initxattrs
| | security_inode_init_security
| | xfs_init_security
| | xfs_generic_create
| | xfs_vn_mknod
| | xfs_vn_create
| | vfs_create
| | path_openat
| | do_filp_open
| | do_sys_open
| | sys_open
| | entry_SYSCALL_64_fastpath
| | 0x7f4ade6f7cdd
| | syscall_work_queue::work_item_returning<syscall_result<int>, reactor::open_file_dma(basic_sstring<char, unsigned int, 15u>, open_flags, file_open_options)::{lambda()#1}>::process
| | 0xffffffffffffffff
| |
| |--0.97%-- rwsem_down_write_failed
| | call_rwsem_down_write_failed
| | xfs_ilock
| | xfs_vn_update_time
| | file_update_time
| | xfs_file_aio_write_checks
| | xfs_file_dio_aio_write
| | xfs_file_write_iter
| | aio_run_iocb
| | do_io_submit
| | sys_io_submit
| | entry_SYSCALL_64_fastpath
| | io_submit
| | 0x46d98a
| |
| |--0.51%-- xlog_cil_force_lsn
| | |
| | |--92.31%-- _xfs_log_force_lsn
| | | |
| | | |--91.67%-- xfs_file_fsync
| | | | vfs_fsync_range
| | | | do_fsync
| | | | sys_fdatasync
| | | | entry_SYSCALL_64_fastpath
| | | | 0x7f4ade4212ad
| | | | syscall_work_queue::work_item_returning<syscall_result<int>, posix_file_impl::flush()::{lambda()#1}>::process
| | | | 0x60b0000c3ec0
| | | |
| | | --8.33%-- xfs_dir_fsync
| | | vfs_fsync_range
| | | do_fsync
| | | sys_fdatasync
| | | entry_SYSCALL_64_fastpath
| | | 0x7f4ade4212ad
| | | syscall_work_queue::work_item_returning<syscall_result<int>, posix_file_impl::flush()::{lambda()#1}>::process
| | | 0x60d0000c3ec0
| | |
| | --7.69%-- _xfs_log_force
| | xfs_log_force
| | xfs_buf_lock
| | _xfs_buf_find
| | xfs_buf_get_map
| | xfs_trans_get_buf_map
| | xfs_btree_get_bufl
| | xfs_bmap_extents_to_btree
| | xfs_bmap_add_extent_hole_real
| | xfs_bmapi_write
| | xfs_iomap_write_direct
| | __xfs_get_blocks
| | xfs_get_blocks_direct
| | do_blockdev_direct_IO
| | __blockdev_direct_IO
| | xfs_vm_direct_IO
| | xfs_file_dio_aio_write
| | xfs_file_write_iter
| | aio_run_iocb
| | do_io_submit
| | sys_io_submit
| | entry_SYSCALL_64_fastpath
| | io_submit
| | 0x46d98a
| --0.04%-- [...]
|
--3.82%-- preempt_schedule_common
|
|--99.02%-- _cond_resched
| |
| |--41.58%-- wait_for_completion
| | |
| | |--66.67%-- flush_work
| | | xlog_cil_force_lsn
| | | |
| | | |--96.43%-- _xfs_log_force_lsn
| | | | |
| | | | |--77.78%-- xfs_file_fsync
| | | | | vfs_fsync_range
| | | | | do_fsync
| | | | | sys_fdatasync
| | | | | entry_SYSCALL_64_fastpath
| | | | | 0x7f4ade4212ad
| | | | | syscall_work_queue::work_item_returning<syscall_result<int>, posix_file_impl::flush()::{lambda()#1}>::process
| | | | | 0x6030000c3ec0
| | | | |
| | | | --22.22%-- xfs_dir_fsync
| | | | vfs_fsync_range
| | | | do_fsync
| | | | sys_fdatasync
| | | | entry_SYSCALL_64_fastpath
| | | | |
| | | | --100.00%-- 0x7f4ade4212ad
| | | | syscall_work_queue::work_item_returning<syscall_result<int>, posix_file_impl::flush()::{lambda()#1}>::process
| | | | 0x6030000c3ec0
| | | |
| | | --3.57%-- _xfs_log_force
| | | xfs_log_force
| | | xfs_buf_lock
| | | _xfs_buf_find
| | | xfs_buf_get_map
| | | xfs_trans_get_buf_map
| | | xfs_btree_get_bufl
| | | xfs_bmap_extents_to_btree
| | | xfs_bmap_add_extent_hole_real
| | | xfs_bmapi_write
| | | xfs_iomap_write_direct
| | | __xfs_get_blocks
| | | xfs_get_blocks_direct
| | | do_blockdev_direct_IO
| | | __blockdev_direct_IO
| | | xfs_vm_direct_IO
| | | xfs_file_dio_aio_write
| | | xfs_file_write_iter
| | | aio_run_iocb
| | | do_io_submit
| | | sys_io_submit
| | | entry_SYSCALL_64_fastpath
| | | io_submit
| | | 0x46d98a
| | |
| | --33.33%-- submit_bio_wait
| | blkdev_issue_flush
| | xfs_blkdev_issue_flush
| | xfs_file_fsync
| | vfs_fsync_range
| | do_fsync
| | sys_fdatasync
| | entry_SYSCALL_64_fastpath
| | |
| | --100.00%-- 0x7f4ade4212ad
| | syscall_work_queue::work_item_returning<syscall_result<int>, posix_file_impl::flush()::{lambda()#1}>::process
| | 0x6030000c3ec0
| |
| |--33.66%-- flush_work
| | xlog_cil_force_lsn
| | |
| | |--97.06%-- _xfs_log_force_lsn
| | | |
| | | |--78.79%-- xfs_file_fsync
| | | | vfs_fsync_range
| | | | do_fsync
| | | | sys_fdatasync
| | | | entry_SYSCALL_64_fastpath
| | | | 0x7f4ade4212ad
| | | | syscall_work_queue::work_item_returning<syscall_result<int>, posix_file_impl::flush()::{lambda()#1}>::process
| | | | 0x6030000c3ec0
| | | |
| | | --21.21%-- xfs_dir_fsync
| | | vfs_fsync_range
| | | do_fsync
| | | sys_fdatasync
| | | entry_SYSCALL_64_fastpath
| | | |
| | | --100.00%-- 0x7f4ade4212ad
| | | syscall_work_queue::work_item_returning<syscall_result<int>, posix_file_impl::flush()::{lambda()#1}>::process
| | | 0x6030000c3ec0
| | |
| | --2.94%-- _xfs_log_force
| | xfs_log_force
| | xfs_buf_lock
| | _xfs_buf_find
| | xfs_buf_get_map
| | xfs_trans_get_buf_map
| | xfs_btree_get_bufl
| | xfs_bmap_extents_to_btree
| | xfs_bmap_add_extent_hole_real
| | xfs_bmapi_write
| | xfs_iomap_write_direct
| | __xfs_get_blocks
| | xfs_get_blocks_direct
| | do_blockdev_direct_IO
| | __blockdev_direct_IO
| | xfs_vm_direct_IO
| | xfs_file_dio_aio_write
| | xfs_file_write_iter
| | aio_run_iocb
| | do_io_submit
| | sys_io_submit
| | entry_SYSCALL_64_fastpath
| | io_submit
| | 0x46d98a
| |
| |--13.86%-- lock_sock_nested
| | |
| | |--78.57%-- tcp_sendmsg
| | | inet_sendmsg
| | | sock_sendmsg
| | | SYSC_sendto
| | | sys_sendto
| | | entry_SYSCALL_64_fastpath
| | | __libc_send
| | | _ZN12continuationIZN6futureIJmEE4thenIZN7reactor14write_all_partER17pollable_fd_statePKvmmEUlmE_S0_IJEEEET0_OT_EUlSC_E_JmEE3runEv
| | | |
| | | |--36.36%-- 0x7f4ad6bf8de0
| | | |
| | | |--9.09%-- 0x4
| | | |
| | | |--9.09%-- 0x7f4adadf8de0
| | | |
| | | |--9.09%-- 0x7f4ada1f8de0
| | | |
| | | |--9.09%-- 0x7f4ad89f8de0
| | | |
| | | |--9.09%-- 0x7f4ad83f8de0
| | | |
| | | |--9.09%-- 0x7f4ad4df8de0
| | | |
| | | --9.09%-- 0x7f4ad35f8de0
| | |
| | --21.43%-- tcp_recvmsg
| | inet_recvmsg
| | sock_recvmsg
| | sock_read_iter
| | __vfs_read
| | vfs_read
| | sys_read
| | entry_SYSCALL_64_fastpath
| | 0x7f4ade6f754d
| | reactor::read_some
| | |
| | |--66.67%-- _ZN12continuationIZN6futureIJEE4thenIZZN7service13storage_proxy22send_to_live_endpointsEmENKUlRSt4pairIK13basic_sstringIcjLj15EESt6vectorIN3gms12inet_addressESaISB_EEEE_clESF_EUlvE_S1_EET0_OT_EUlSK_E_JEE3runEv
| | | reactor::del_timer
| | | 0x6160000e2040
| | |
| | --33.33%-- continuation<future<> future<>::then_wrapped<future<> future<>::finally<auto seastar::with_gate<transport::cql_server::connection::process()::{lambda()#2}::operator()() const::{lambda()#1}>(seastar::gate&, transport::cql_server::connection::process()::{lambda()#2}::operator()() const::{lambda()#1}&&)::{lambda()#1}>(seastar::gate&)::{lambda(future<>)#1}::operator()(future<>)::{lambda(seastar::gate)#1}, future<> >(seastar::gate&)::{lambda(seastar::gate&)#1}>::run
| | reactor::del_timer
| | 0x6030000e2040
| |
| |--3.96%-- generic_make_request_checks
| | generic_make_request
| | submit_bio
| | do_blockdev_direct_IO
| | __blockdev_direct_IO
| | xfs_vm_direct_IO
| | xfs_file_dio_aio_write
| | xfs_file_write_iter
| | aio_run_iocb
| | do_io_submit
| | sys_io_submit
| | entry_SYSCALL_64_fastpath
| | io_submit
| | 0x46d98a
| |
| |--3.96%-- kmem_cache_alloc_node
| | __alloc_skb
| | sk_stream_alloc_skb
| | tcp_sendmsg
| | inet_sendmsg
| | sock_sendmsg
| | SYSC_sendto
| | sys_sendto
| | entry_SYSCALL_64_fastpath
| | __libc_send
| | _ZN12continuationIZN6futureIJmEE4thenIZN7reactor14write_all_partER17pollable_fd_statePKvmmEUlmE_S0_IJEEEET0_OT_EUlSC_E_JmEE3runEv
| | |
| | |--25.00%-- 0x7f4ad9bf8de0
| | |
| | |--25.00%-- 0x7f4ad7df8de0
| | |
| | |--25.00%-- 0x7f4ad77f8de0
| | |
| | --25.00%-- 0x7f4ad59f8de0
| |
| |--0.99%-- unmap_underlying_metadata
| | do_blockdev_direct_IO
| | __blockdev_direct_IO
| | xfs_vm_direct_IO
| | xfs_file_dio_aio_write
| | xfs_file_write_iter
| | aio_run_iocb
| | do_io_submit
| | sys_io_submit
| | entry_SYSCALL_64_fastpath
| | io_submit
| | 0x46d98a
| |
| |--0.99%-- __kmalloc_node_track_caller
| | __kmalloc_reserve.isra.32
| | __alloc_skb
| | sk_stream_alloc_skb
| | tcp_sendmsg
| | inet_sendmsg
| | sock_sendmsg
| | SYSC_sendto
| | sys_sendto
| | entry_SYSCALL_64_fastpath
| | __libc_send
| | _ZN12continuationIZN6futureIJmEE4thenIZN7reactor14write_all_partER17pollable_fd_statePKvmmEUlmE_S0_IJEEEET0_OT_EUlSC_E_JmEE3runEv
| | 0x7f4ad6bf8de0
| |
| --0.99%-- task_work_run
| do_notify_resume
| int_signal
| __libc_close
| std::experimental::fundamentals_v1::bad_optional_access::~bad_optional_access
|
--0.98%-- __cond_resched_softirq
release_sock
tcp_sendmsg
inet_sendmsg
sock_sendmsg
SYSC_sendto
sys_sendto
entry_SYSCALL_64_fastpath
__libc_send
_ZN12continuationIZN6futureIJmEE4thenIZN7reactor14write_all_partER17pollable_fd_statePKvmmEUlmE_S0_IJEEEET0_OT_EUlSC_E_JmEE3runEv
0x7f4ada1f8de0
#
# (For a higher level overview, try: perf report --sort comm,dso)
#
[-- Attachment #3: example-stale.txt --]
[-- Type: text/plain, Size: 1700 bytes --]
[164814.835933] CPU: 22 PID: 48042 Comm: scylla Tainted: G E 4.2.6-200.fc22.x86_64 #1
[164814.835936] Hardware name: Xen HVM domU, BIOS 4.2.amazon 05/06/2015
[164814.835937] 0000000000000000 00000000a8713b7a ffff8802fb977ab8 ffffffff817729ea
[164814.835941] 0000000000000000 ffff88076a69f780 ffff8802fb977ad8 ffffffffa03217a6
[164814.835946] ffff88077119bcb0 0000000000000000 ffff8802fb977b08 ffffffffa034e749
[164814.835951] Call Trace:
[164814.835954] [<ffffffff817729ea>] dump_stack+0x45/0x57
[164814.835971] [<ffffffffa03217a6>] xfs_buf_stale+0x26/0x80 [xfs]
[164814.835989] [<ffffffffa034e749>] xfs_trans_binval+0x79/0x100 [xfs]
[164814.836001] [<ffffffffa02f479b>] xfs_bmap_btree_to_extents+0x12b/0x1a0 [xfs]
[164814.836012] [<ffffffffa02f8977>] xfs_bunmapi+0x967/0x9f0 [xfs]
[164814.836027] [<ffffffffa0334b9e>] xfs_itruncate_extents+0x10e/0x220 [xfs]
[164814.836044] [<ffffffffa033f75a>] ? kmem_zone_alloc+0x5a/0xe0 [xfs]
[164814.836084] [<ffffffffa0334d49>] xfs_inactive_truncate+0x99/0x110 [xfs]
[164814.836120] [<ffffffffa0335aa2>] xfs_inactive+0x102/0x120 [xfs]
[164814.836135] [<ffffffffa033a6cf>] xfs_fs_evict_inode+0x6f/0xa0 [xfs]
[164814.836138] [<ffffffff81238d76>] evict+0xa6/0x170
[164814.836140] [<ffffffff81239026>] iput+0x196/0x220
[164814.836147] [<ffffffff81234fe4>] __dentry_kill+0x174/0x1c0
[164814.836150] [<ffffffff8123514b>] dput+0x11b/0x200
[164814.836155] [<ffffffff8121fe02>] __fput+0x172/0x1e0
[164814.836158] [<ffffffff8121febe>] ____fput+0xe/0x10
[164814.836161] [<ffffffff810bab75>] task_work_run+0x85/0xb0
[164814.836164] [<ffffffff81014a4d>] do_notify_resume+0x8d/0x90
[164814.836167] [<ffffffff817795bc>] int_signal+0x12/0x17
[-- Attachment #4: Type: text/plain, Size: 121 bytes --]
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-11-28 2:43 sleeps and waits during io_submit Glauber Costa
@ 2015-11-30 14:10 ` Brian Foster
2015-11-30 14:29 ` Avi Kivity
2015-11-30 15:49 ` Glauber Costa
2015-11-30 23:10 ` Dave Chinner
1 sibling, 2 replies; 58+ messages in thread
From: Brian Foster @ 2015-11-30 14:10 UTC (permalink / raw)
To: Glauber Costa; +Cc: Avi Kivity, xfs
On Fri, Nov 27, 2015 at 09:43:50PM -0500, Glauber Costa wrote:
> Hello my dear XFSers,
>
> For those of you who don't know, we at ScyllaDB produce a modern NoSQL
> data store that, at the moment, runs on top of XFS only. We deal
> exclusively with asynchronous and direct IO, due to our
> thread-per-core architecture. Due to that, we avoid issuing any
> operation that will sleep.
>
> While debugging an extreme case of bad performance (most likely
> related to a not-so-great disk), I have found a variety of cases in
> which XFS blocks. To find those, I have used perf record -e
> sched:sched_switch -p <pid_of_db>, and I am attaching the perf report
> as xfs-sched_switch.log. Please note that this doesn't tell me for how
> long we block, but as mentioned before, blocking operations outside
> our control are detrimental to us regardless of the elapsed time.
>
> For those who are not acquainted to our internals, please ignore
> everything in that file but the xfs functions. For the xfs symbols,
> there are two kinds of events: the ones that are a children of
> io_submit, where we don't tolerate blocking, and the ones that are
> children of our helper IO thread, to where we push big operations that
> we know will block until we can get rid of them all. We care about the
> former and ignore the latter.
>
> Please allow me to ask you a couple of questions about those findings.
> If we are doing anything wrong, advise on best practices is truly
> welcome.
>
> 1) xfs_buf_lock -> xfs_log_force.
>
> I've started wondering what would make xfs_log_force sleep. But then I
> have noticed that xfs_log_force will only be called when a buffer is
> marked stale. Most of the times a buffer is marked stale seems to be
> due to errors. Although that is not my case (more on that), it got me
> thinking that maybe the right thing to do would be to avoid hitting
> this case altogether?
>
I'm not following where you get the "only if marked stale" part..? It
certainly looks like that's one potential purpose for the call, but this
is called in a variety of other places as well. E.g., forcing the log
via pushing on the ail when it has pinned items is another case. The ail
push itself can originate from transaction reservation, etc., when log
space is needed. In other words, I'm not sure this is something that's
easily controlled from userspace, if at all. Rather, it's a significant
part of the wider state machine the fs uses to manage logging.
> The file example-stale.txt contains a backtrace of the case where we
> are being marked as stale. It seems to be happening when we convert
> the the inode's extents from unwritten to real. Can this case be
> avoided? I won't pretend I know the intricacies of this, but couldn't
> we be keeping extents from the very beginning to avoid creating stale
> buffers?
>
This is down in xfs_fs_evict_inode()->xfs_inactive(), which is generally
when an inode is evicted from cache. In this case, it looks like the
inode is unlinked (permanently removed), the extents are being removed
and a bmap btree block is being invalidated as part of that overall
process. I don't think this has anything to do with unwritten extents.
> 2) xfs_buf_lock -> down
> This is one I truly don't understand. What can be causing contention
> in this lock? We never have two different cores writing to the same
> buffer, nor should we have the same core doingCAP_FOWNER so.
>
This is not one single lock. An XFS buffer is the data structure used to
modify/log/read-write metadata on-disk and each buffer has its own lock
to prevent corruption. Buffer lock contention is possible because the
filesystem has bits of "global" metadata that has to be updated via
buffers.
For example, usually one has multiple allocation groups to maximize
parallelism, but we still have per-ag metadata that has to be tracked
globally with respect to each AG (e.g., free space trees, inode
allocation trees, etc.). Any operation that affects this metadata (e.g.,
block/inode allocation) has to lock the agi/agf buffers along with any
buffers associated with the modified btree leaf/node blocks, etc.
One example in your attached perf traces has several threads looking to
acquire the AGF, which is a per-AG data structure for tracking free
space in the AG. One thread looks like the inode eviction case noted
above (freeing blocks), another looks like a file truncate (also freeing
blocks), and yet another is a block allocation due to a direct I/O
write. Were any of these operations directed to an inode in a separate
AG, they would be able to proceed in parallel (but I believe they would
still hit the same codepaths as far as perf can tell).
> 3) xfs_file_aio_write_checks -> file_update_time -> xfs_vn_update_time
>
> You guys seem to have an interface to avoid that, by setting the
> FMODE_NOCMTIME flag. This is done by issuing the open by handle ioctl,
> which will set this flag for all regular files. That's great, but that
> ioctl required CAP_SYS_ADMIN, which is a big no for us, since we run
> our server as an unprivileged user. I don't understand, however, why
> such an strict check is needed. If we have full rights on the
> filesystem, why can't we issue this operation? In my view, CAP_FOWNER
> should already be enough.I do understand the handles have to be stable
> and a file can have its ownership changed, in which case the previous
> owner would keep the handle valid. Is that the reason you went with
> the most restrictive capability ?
I'm not familiar enough with the open-by-handle stuff to comment on the
permission constraints. Perhaps Dave or others can comment further on
this bit...
Brian
> # To display the perf.data header info, please use --header/--header-only options.
> #
> #
> # Total Lost Samples: 0
> #
> # Samples: 2K of event 'sched:sched_switch'
> # Event count (approx.): 2669
> #
> # Overhead Command Shared Object Symbol
> # ........ ....... ................. ..............
> #
> 100.00% scylla [kernel.kallsyms] [k] __schedule
> |
> ---__schedule
> |
> |--96.18%-- schedule
> | |
> | |--56.14%-- schedule_user
> | | |
> | | |--53.30%-- int_careful
> | | | |
> | | | |--45.05%-- 0x7f4ade6f74ed
> | | | | reactor_backend_epoll::make_reactor_notifier
> | | | | |
> | | | | |--67.63%-- syscall_work_queue::submit_item
> | | | | | |
> | | | | | |--32.05%-- posix_file_impl::truncate
> | | | | | | |
> | | | | | | |--65.33%-- _ZN12continuationIZN6futureIJEE4thenIZN19file_data_sink_impl5flushEvEUlvE_S1_EET0_OT_EUlS7_E_JEE3runEv
> | | | | | | | reactor::del_timer
> | | | | | | | 0x60b0000e2040
> | | | | | | |
> | | | | | | |--20.00%-- db::commitlog::segment::flush(unsigned long)::{lambda()#1}::operator()
> | | | | | | | |
> | | | | | | | |--73.33%-- future<>::then<db::commitlog::segment::flush(unsigned long)::{lambda()#1}, future<lw_shared_ptr<db::commitlog::segment> > >
> | | | | | | | | _ZN12continuationIZN6futureIJ13lw_shared_ptrIN2db9commitlog7segmentEEEE4thenIZNS4_4syncEvEUlT_E_S6_EET0_OS8_EUlSB_E_JS5_EE3runEv
> | | | | | | | | reactor::del_timer
> | | | | | | | | 0x60e0000e2040
> | | | | | | | |
> | | | | | | | --26.67%-- _ZN12continuationIZN6futureIJEE4thenIZN2db9commitlog7segment5flushEmEUlvE_S0_IJ13lw_shared_ptrIS5_EEEEET0_OT_EUlSC_E_JEE3runEv
> | | | | | | | reactor::del_timer
> | | | | | | | 0x6090000e2040
> | | | | | | |
> | | | | | | |--10.67%-- sstables::sstable::seal_sstable
> | | | | | | | std::_Function_handler<void (), futurize<std::result_of<std::decay<sstables::sstable::write_components(mutation_reader, unsigned long, lw_shared_ptr<schema const>, unsigned long)::{lambda()#1}>::type ()>::type>::type seastar::async<sstables::sstable::write_components(mutation_reader, unsigned long, lw_shared_ptr<schema const>, unsigned long)::{lambda()#1}>(std::decay&&, (std::decay<sstables::sstable::write_components(mutation_reader, unsigned long, lw_shared_ptr<schema const>, unsigned long)::{lambda()#1}>::type&&)...)::{lambda(futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type> seastar::async<{lambda()#1}>(futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type>::type, std::decay<{lambda()#1}>::type&&)::work&)#1}::operator()(futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type> seastar::async<{lambda()#1}>(futurize<std::result_of<std::decay<{lambd
a()#1}>::type ()>::type>::type, std::decay<{lambda()#1}>::type&&)::work)::{lambda()#1}>::_M_invoke
> | | | | | | | _GLOBAL__sub_I__ZN12app_templateC2Ev
> | | | | | | |
> | | | | | | --4.00%-- sstables::sstable::write_toc
> | | | | | | sstables::sstable::prepare_write_components
> | | | | | | |
> | | | | | | |--50.00%-- 0x4d3a4f6ec4e8cd75
> | | | | | | |
> | | | | | | --50.00%-- 0x3ebf3dd80e3b174d
> | | | | | |
> | | | | | |--23.93%-- posix_file_impl::discard
> | | | | | | |
> | | | | | | |--82.14%-- _ZN12continuationIZN6futureIImEE4thenIZN19file_data_sink_impl6do_putEm16temporary_bufferIcEEUlmE_S0_IIEEEET0_OT_EUlSA_E_ImEE3runEv
> | | | | | | | reactor::del_timer
> | | | | | | | 0x6080000e2040
> | | | | | | |
> | | | | | | --17.86%-- futurize<future<lw_shared_ptr<db::commitlog::segment> > >::apply<db::commitlog::segment_manager::allocate_segment(bool)::{lambda(file)#1}, file>
> | | | | | | _ZN12continuationIZN6futureIJ4fileEE4thenIZN2db9commitlog15segment_manager16allocate_segmentEbEUlS1_E_S0_IJ13lw_shared_ptrINS5_7segmentEEEEEET0_OT_EUlSE_E_JS1_EE3runEv
> | | | | | |
> | | | | | |--20.94%-- reactor::open_file_dma
> | | | | | | |
> | | | | | | |--20.41%-- db::commitlog::segment_manager::allocate_segment
> | | | | | | | db::commitlog::segment_manager::on_timer()::{lambda()#1}::operator()
> | | | | | | | 0xb8c264
> | | | | | | |
> | | | | | | |--14.29%-- sstables::sstable::write_simple<(sstables::sstable::component_type)8, sstables::statistics>
> | | | | | | | std::_Function_handler<void (), futurize<std::result_of<std::decay<sstables::sstable::write_components(mutation_reader, unsigned long, lw_shared_ptr<schema const>, unsigned long)::{lambda()#1}>::type ()>::type>::type seastar::async<sstables::sstable::write_components(mutation_reader, unsigned long, lw_shared_ptr<schema const>, unsigned long)::{lambda()#1}>(std::decay&&, (std::decay<sstables::sstable::write_components(mutation_reader, unsigned long, lw_shared_ptr<schema const>, unsigned long)::{lambda()#1}>::type&&)...)::{lambda(futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type> seastar::async<{lambda()#1}>(futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type>::type, std::decay<{lambda()#1}>::type&&)::work&)#1}::operator()(futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type> seastar::async<{lambda()#1}>(futurize<std::result_of<std::decay<{lambd
a()#1}>::type ()>::type>::type, std::decay<{lambda()#1}>::type&&)::work)::{lambda()#1}>::_M_invoke
> | | | | | | | _GLOBAL__sub_I__ZN12app_templateC2Ev
> | | | | | | |
> | | | | | | |--12.24%-- sstables::write_crc
> | | | | | | | |
> | | | | | | | |--16.67%-- 0x313532343536002f
> | | | | | | | |
> | | | | | | | |--16.67%-- 0x373633323533002f
> | | | | | | | |
> | | | | | | | |--16.67%-- 0x363139333232002f
> | | | | | | | |
> | | | | | | | |--16.67%-- 0x353933303330002f
> | | | | | | | |
> | | | | | | | |--16.67%-- 0x383930383133002f
> | | | | | | | |
> | | | | | | | --16.67%-- 0x323338303037002f
> | | | | | | |
> | | | | | | |--12.24%-- sstables::write_digest
> | | | | | | |
> | | | | | | |--10.20%-- sstables::sstable::write_simple<(sstables::sstable::component_type)7, sstables::filter>
> | | | | | | | sstables::sstable::write_filter
> | | | | | | | std::_Function_handler<void (), futurize<std::result_of<std::decay<sstables::sstable::write_components(mutation_reader, unsigned long, lw_shared_ptr<schema const>, unsigned long)::{lambda()#1}>::type ()>::type>::type seastar::async<sstables::sstable::write_components(mutation_reader, unsigned long, lw_shared_ptr<schema const>, unsigned long)::{lambda()#1}>(std::decay&&, (std::decay<sstables::sstable::write_components(mutation_reader, unsigned long, lw_shared_ptr<schema const>, unsigned long)::{lambda()#1}>::type&&)...)::{lambda(futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type> seastar::async<{lambda()#1}>(futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type>::type, std::decay<{lambda()#1}>::type&&)::work&)#1}::operator()(futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type> seastar::async<{lambda()#1}>(futurize<std::result_of<std::decay<{lambd
a()#1}>::type ()>::type>::type, std::decay<{lambda()#1}>::type&&)::work)::{lambda()#1}>::_M_invoke
> | | | | | | | _GLOBAL__sub_I__ZN12app_templateC2Ev
> | | | | | | |
> | | | | | | |--10.20%-- sstables::sstable::write_simple<(sstables::sstable::component_type)4, sstables::summary_ka>
> | | | | | | | std::_Function_handler<void (), futurize<std::result_of<std::decay<sstables::sstable::write_components(mutation_reader, unsigned long, lw_shared_ptr<schema const>, unsigned long)::{lambda()#1}>::type ()>::type>::type seastar::async<sstables::sstable::write_components(mutation_reader, unsigned long, lw_shared_ptr<schema const>, unsigned long)::{lambda()#1}>(std::decay&&, (std::decay<sstables::sstable::write_components(mutation_reader, unsigned long, lw_shared_ptr<schema const>, unsigned long)::{lambda()#1}>::type&&)...)::{lambda(futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type> seastar::async<{lambda()#1}>(futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type>::type, std::decay<{lambda()#1}>::type&&)::work&)#1}::operator()(futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type> seastar::async<{lambda()#1}>(futurize<std::result_of<std::decay<{lambd
a()#1}>::type ()>::type>::type, std::decay<{lambda()#1}>::type&&)::work)::{lambda()#1}>::_M_invoke
> | | | | | | | _GLOBAL__sub_I__ZN12app_templateC2Ev
> | | | | | | |
> | | | | | | |--10.20%-- 0x78d93b
> | | | | | | |
> | | | | | | |--6.12%-- sstables::sstable::open_data
> | | | | | | | |
> | | | | | | | --100.00%-- 0x8000000004000000
> | | | | | | |
> | | | | | | --4.08%-- sstables::sstable::write_toc
> | | | | | | sstables::sstable::prepare_write_components
> | | | | | | |
> | | | | | | --100.00%-- 0x6100206690ef
> | | | | | |
> | | | | | |--18.38%-- syscall_work_queue::submit_item
> | | | | | | |
> | | | | | | |--10.00%-- 0x7f4ad89f8fe0
> | | | | | | |
> | | | | | | |--7.50%-- 0x7f4ad83f8fe0
> | | | | | | |
> | | | | | | |--7.50%-- 0x7f4ad6bf8fe0
> | | | | | | |
> | | | | | | |--7.50%-- 0x7f4ad65f8fe0
> | | | | | | |
> | | | | | | |--5.00%-- 0x60b015e8cd90
> | | | | | | |
> | | | | | | |--5.00%-- 0x60100acaed90
> | | | | | | |
> | | | | | | |--5.00%-- 0x607006f04d90
> | | | | | | |
> | | | | | | |--5.00%-- 0xffffffffffffa5d0
> | | | | | | |
> | | | | | | |--2.50%-- 0x60e01acbed90
> | | | | | | |
> | | | | | | |--2.50%-- 0x60e01acbec60
> | | | | | | |
> | | | | | | |--2.50%-- 0x60a018d7ad90
> | | | | | | |
> | | | | | | |--2.50%-- 0x60a018d7ac60
> | | | | | | |
> | | | | | | |--2.50%-- 0x60b015e8cc60
> | | | | | | |
> | | | | | | |--2.50%-- 0x60900bb8ad60
> | | | | | | |
> | | | | | | |--2.50%-- 0x60100acaec60
> | | | | | | |
> | | | | | | |--2.50%-- 0x60800951dd90
> | | | | | | |
> | | | | | | |--2.50%-- 0x60800951dc60
> | | | | | | |
> | | | | | | |--2.50%-- 0x60d009089d90
> | | | | | | |
> | | | | | | |--2.50%-- 0x60d009089c60
> | | | | | | |
> | | | | | | |--2.50%-- 0x607006f04c60
> | | | | | | |
> | | | | | | |--2.50%-- 0x60f005984d60
> | | | | | | |
> | | | | | | |--2.50%-- 0x7f4ad77f8fe0
> | | | | | | |
> | | | | | | |--2.50%-- 0x7f4adb9f8fe0
> | | | | | | |
> | | | | | | |--2.50%-- 0x7f4ad9bf8fe0
> | | | | | | |
> | | | | | | |--2.50%-- 0x7f4ad7df8fe0
> | | | | | | |
> | | | | | | |--2.50%-- 0x7f4ad77f8fe0
> | | | | | | |
> | | | | | | --2.50%-- 0x7f4ad5ff8fe0
> | | | | | |
> | | | | | |--2.99%-- reactor::open_directory
> | | | | | | |
> | | | | | | |--57.14%-- sstables::sstable::filename
> | | | | | | |
> | | | | | | --42.86%-- sstables::sstable::write_toc
> | | | | | | sstables::sstable::prepare_write_components
> | | | | | | |
> | | | | | | |--50.00%-- 0x4d3a4f6ec4e8cd75
> | | | | | | |
> | | | | | | --50.00%-- 0x3ebf3dd80e3b174d
> | | | | | |
> | | | | | --1.71%-- reactor::rename_file
> | | | | | sstables::sstable::seal_sstable
> | | | | | std::_Function_handler<void (), futurize<std::result_of<std::decay<sstables::sstable::write_components(mutation_reader, unsigned long, lw_shared_ptr<schema const>, unsigned long)::{lambda()#1}>::type ()>::type>::type seastar::async<sstables::sstable::write_components(mutation_reader, unsigned long, lw_shared_ptr<schema const>, unsigned long)::{lambda()#1}>(std::decay&&, (std::decay<sstables::sstable::write_components(mutation_reader, unsigned long, lw_shared_ptr<schema const>, unsigned long)::{lambda()#1}>::type&&)...)::{lambda(futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type> seastar::async<{lambda()#1}>(futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type>::type, std::decay<{lambda()#1}>::type&&)::work&)#1}::operator()(futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type> seastar::async<{lambda()#1}>(futurize<std::result_of<std::decay<{lambda()#1}>::ty
pe ()>::type>::type, std::decay<{lambda()#1}>::type&&)::work)::{lambda()#1}>::_M_invoke
> | | | | | _GLOBAL__sub_I__ZN12app_templateC2Ev
> | | | | |
> | | | | --32.37%-- _ZN12continuationIZN6futureIJEE4thenIZN18syscall_work_queue11submit_itemEPNS3_9work_itemEEUlvE_S1_EET0_OT_EUlS9_E_JEE3runEv
> | | | | reactor::del_timer
> | | | | 0x60d0000e2040
> | | | |
> | | | |--29.04%-- __vdso_clock_gettime
> | | | |
> | | | |--19.66%-- 0x7f4ade42b193
> | | | | reactor_backend_epoll::complete_epoll_event
> | | | | |
> | | | | |--41.61%-- smp_message_queue::async_work_item<future<> seastar::sharded<database>::invoke_on<service::storage_proxy::mutate_locally(frozen_mutation const&)::{lambda(database&)#1}, future<> >(unsigned int, service::storage_proxy::mutate_locally(frozen_mutation const&)::{lambda(database&)#1}&&)::{lambda()#1}>::process
> | | | | | |
> | | | | | |--79.03%-- smp_message_queue::process_queue<2ul, smp_message_queue::process_incoming()::{lambda(smp_message_queue::work_item*)#1}>
> | | | | | | |
> | | | | | | |--95.92%-- 0x6070000c3000
> | | | | | | |
> | | | | | | |--2.04%-- 0x61d0000c1000
> | | | | | | |
> | | | | | | --2.04%-- 0x61d0000c1000
> | | | | | |
> | | | | | |--3.23%-- 0x14dd51
> | | | | | |
> | | | | | |--1.61%-- 0x162a54
> | | | | | |
> | | | | | |--1.61%-- 0x161dca
> | | | | | |
> | | | | | |--1.61%-- 0x159c8b
> | | | | | |
> | | | | | |--1.61%-- 0x1598b5
> | | | | | |
> | | | | | |--1.61%-- 0x14dd3e
> | | | | | |
> | | | | | |--1.61%-- 0x14bad8
> | | | | | |
> | | | | | |--1.61%-- 0x14a880
> | | | | | |
> | | | | | |--1.61%-- 0x127105
> | | | | | |
> | | | | | |--1.61%-- 0x6070000e2040
> | | | | | |
> | | | | | |--1.61%-- smp_message_queue::process_queue<2ul, smp_message_queue::process_incoming()::{lambda(smp_message_queue::work_item*)#1}>
> | | | | | | 0x60d0000c3000
> | | | | | |
> | | | | | --1.61%-- __vdso_clock_gettime
> | | | | | 0x7f4ad77f9160
> | | | | |
> | | | | |--30.20%-- __restore_rt
> | | | | | |
> | | | | | |--57.14%-- __vdso_clock_gettime
> | | | | | | 0x1d
> | | | | | |
> | | | | | |--9.52%-- smp_message_queue::smp_message_queue
> | | | | | | 0x6070000c3000
> | | | | | |
> | | | | | |--4.76%-- 0x600000357240
> | | | | | |
> | | | | | |--4.76%-- 0x60000031a640
> | | | | | |
> | | | | | |--2.38%-- posix_file_impl::list_directory
> | | | | | | 0x609000044730
> | | | | | |
> | | | | | |--2.38%-- 0x46efbf
> | | | | | |
> | | | | | |--2.38%-- 0x600000442e40
> | | | | | |
> | | | | | |--2.38%-- 0x600000376440
> | | | | | |
> | | | | | |--2.38%-- 0x6000002bac40
> | | | | | |
> | | | | | |--2.38%-- 0x600000295640
> | | | | | |
> | | | | | |--2.38%-- 0x600000289e40
> | | | | | |
> | | | | | |--2.38%-- 0x60000031a640
> | | | | | |
> | | | | | |--2.38%-- 0x7f4ade6f74ed
> | | | | | | __libc_siglongjmp
> | | | | | | 0x60000047be40
> | | | | | |
> | | | | | --2.38%-- 0x7f4adb3f7fd0
> | | | | |
> | | | | |--14.09%-- 0x33
> | | | | |
> | | | | |--12.08%-- promise<temporary_buffer<char> >::promise
> | | | | | _ZN6futureIJ16temporary_bufferIcEEE4thenIZN12input_streamIcE12read_exactlyEmEUlT_E_S2_EET0_OS6_
> | | | | | |
> | | | | | |--44.44%-- input_stream<char>::read_exactly
> | | | | | | 0x8
> | | | | | |
> | | | | | |--11.11%-- 0x7f4adb3f8ea0
> | | | | | |
> | | | | | |--11.11%-- 0x7f4ad9bf8ea0
> | | | | | |
> | | | | | |--11.11%-- 0x7f4ad89f8ea0
> | | | | | |
> | | | | | |--11.11%-- 0x7f4ad83f8ea0
> | | | | | |
> | | | | | |--5.56%-- 0x7f4ad77f8ea0
> | | | | | |
> | | | | | --5.56%-- 0x7f4ad7df8ea0
> | | | | |
> | | | | |--1.34%-- 0x7f4ad6bf8d80
> | | | | |
> | | | | --0.67%-- 0x7f4adadf8d80
> | | | |
> | | | |--4.43%-- __libc_send
> | | | | _ZN12continuationIZN6futureIJmEE4thenIZN7reactor14write_all_partER17pollable_fd_statePKvmmEUlmE_S0_IJEEEET0_OT_EUlSC_E_JmEE3runEv
> | | | | |
> | | | | |--14.71%-- 0x4
> | | | | |
> | | | | |--11.76%-- 0x7f4ad89f8de0
> | | | | |
> | | | | |--8.82%-- 0x7f4adb3f8de0
> | | | | |
> | | | | |--8.82%-- 0x7f4ad9bf8de0
> | | | | |
> | | | | |--8.82%-- 0x7f4ad77f8de0
> | | | | |
> | | | | |--8.82%-- 0x7f4ad6bf8de0
> | | | | |
> | | | | |--5.88%-- 0x7f4ad83f8de0
> | | | | |
> | | | | |--5.88%-- 0x7f4ad7df8de0
> | | | | |
> | | | | |--5.88%-- 0x7f4ad53f8de0
> | | | | |
> | | | | |--2.94%-- 0x7f4acc9f8de0
> | | | | |
> | | | | |--2.94%-- continuation<future<file>::wait()::{lambda(future_state<file>&&)#1}, file>::~continuation
> | | | | | 0x611003c8e9b8
> | | | | |
> | | | | |--2.94%-- 0x7f4adb9f8de0
> | | | | |
> | | | | |--2.94%-- 0x7f4ad71f8de0
> | | | | |
> | | | | |--2.94%-- 0x7f4ad65f8de0
> | | | | |
> | | | | |--2.94%-- 0x7f4ad59f8de0
> | | | | |
> | | | | --2.94%-- 0x7f4ad35f8de0
> | | | |
> | | | |--1.56%-- 0x7f4ade6f754d
> | | | | reactor::read_some
> | | | | |
> | | | | |--66.67%-- _ZN12continuationIZN6futureIJEE4thenIZZN7service13storage_proxy22send_to_live_endpointsEmENKUlRSt4pairIK13basic_sstringIcjLj15EESt6vectorIN3gms12inet_addressESaISB_EEEE_clESF_EUlvE_S1_EET0_OT_EUlSK_E_JEE3runEv
> | | | | | reactor::del_timer
> | | | | | 0x6070000e2040
> | | | | |
> | | | | |--8.33%-- _ZN12continuationIZN6futureIIEE4thenIZ5sleepINSt6chrono3_V212system_clockEmSt5ratioILl1ELl1000000EEES1_NS4_8durationIT0_T1_EEEUlvE_S1_EESA_OT_EUlSF_E_IEE3runEv
> | | | | | reactor::del_timer
> | | | | | 0x6080000e2040
> | | | | |
> | | | | |--8.33%-- 0x600000483640
> | | | | |
> | | | | |--8.33%-- 0x600000480440
> | | | | |
> | | | | --8.33%-- 0x36
> | | | --0.26%-- [...]
> | | |
> | | --46.70%-- retint_careful
> | | |
> | | |--6.24%-- posix_file_impl::list_directory
> | | | |
> | | | |--80.00%-- 0x60f0000e2020
> | | | |
> | | | |--5.00%-- 0x601000044730
> | | | |
> | | | |--5.00%-- 0x60e000044720
> | | | |
> | | | |--2.50%-- 0x60f000135500
> | | | |
> | | | |--2.50%-- 0x6190000e2098
> | | | |
> | | | |--2.50%-- 0x60d0000c3000
> | | | |
> | | | --2.50%-- 0x1
> | | |
> | | |--3.42%-- boost::lockfree::detail::ringbuffer_base<smp_message_queue::work_item*>::pop
> | | | |
> | | | |--95.65%-- boost::program_options::variables_map::get
> | | | |
> | | | --4.35%-- 0x618000044680
> | | |
> | | |--3.12%-- memory::small_pool::add_more_objects
> | | | |
> | | | |--10.53%-- managed_vector<atomic_cell_or_collection, 5u, unsigned int>::clear_and_release
> | | | | mutation_partition::clustered_row
> | | | | mutation::set_clustered_cell
> | | | | cql3::constants::setter::execute
> | | | | cql3::statements::update_statement::add_update_for_key
> | | | | _ZN8futurizeI6futureIJSt6vectorI8mutationSaIS2_EEEEE5applyIZN4cql310statements22modification_statement13get_mutationsERN7seastar7shardedIN7service13storage_proxyEEERKNS8_13query_optionsEblEUlT_E_JSt10unique_ptrINS8_17update_parametersESt14default_deleteISN_EEEEES5_OSK_OSt5tupleIJDpT0_EE
> | | | | cql3::statements::modification_statement::get_mutations
> | | | | cql3::statements::modification_statement::execute_without_condition
> | | | | cql3::query_options::query_options
> | | | | |
> | | | | |--50.00%-- 0x7f4ad77f80e0
> | | | | |
> | | | | --50.00%-- 0x7f4ad6bf80e0
> | | | |
> | | | |--10.53%-- memory::small_pool::add_more_objects
> | | | | |
> | | | | |--50.00%-- 0x60e00015d000
> | | | | |
> | | | | --50.00%-- 0x60b00af6c758
> | | | |
> | | | |--5.26%-- 0x60a018ee3867
> | | | |
> | | | |--5.26%-- 0x60d00d41f680
> | | | |
> | | | |--5.26%-- 0x61400c6bb4d0
> | | | |
> | | | |--5.26%-- 0x60e007c918d6
> | | | |
> | | | |--5.26%-- 0x60e0078294ce
> | | | |
> | | | |--5.26%-- 0x607006ee4da0
> | | | |
> | | | |--5.26%-- _ZN12continuationIZN6futureIJEE12then_wrappedIZNS1_16handle_exceptionIZN7service13storage_proxy22send_to_live_endpointsEmEUlNSt15__exception_ptr13exception_ptrEE0_EES1_OT_EUlSA_E_S1_EET0_SA_EUlSA_E_JEE3runEv
> | | | | reactor::del_timer
> | | | | 0x6030000e2040
> | | | |
> | | | |--5.26%-- service::storage_proxy::mutate_locally
> | | | | service::storage_proxy::send_to_live_endpoints
> | | | | parallel_for_each<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, service::storage_proxy::mutate_begin(std::vector<unsigned long, std::allocator<unsigned long> >, db::consistency_level)::{lambda(unsigned long)#1}>
> | | | | 0x601000136d00
> | | | |
> | | | |--5.26%-- 0x60a0001900e0
> | | | |
> | | | |--5.26%-- 0x60e00015d040
> | | | |
> | | | |--5.26%-- 0x61300015d000
> | | | |
> | | | |--5.26%-- 0x60e00013bde0
> | | | |
> | | | |--5.26%-- 0x60b00010f308
> | | | |
> | | | |--5.26%-- 0x6010000e4808
> | | | |
> | | | --5.26%-- 0x7f4ad65f7f50
> | | |
> | | |--2.82%-- std::unique_ptr<reactor::pollfn, std::default_delete<std::unique_ptr> > reactor::make_pollfn<reactor::run()::{lambda()#3}>(reactor::run()::{lambda()#3}&&)::the_pollfn::poll_and_check_more_work
> | | | |
> | | | |--25.00%-- boost::lockfree::detail::ringbuffer_base<smp_message_queue::work_item*>::pop
> | | | | boost::program_options::variables_map::get
> | | | |
> | | | |--25.00%-- 0x1
> | | | |
> | | | |--12.50%-- 0x53
> | | | |
> | | | |--12.50%-- 0x3e
> | | | |
> | | | |--12.50%-- 0x24
> | | | |
> | | | --12.50%-- 0xb958000000000000
> | | |
> | | |--2.67%-- std::_Function_handler<partition_presence_checker_result (partition_key const&), column_family::make_partition_presence_checker(lw_shared_ptr<std::map<long, lw_shared_ptr<sstables::sstable>, std::less<long>, std::allocator<std::pair<long const, lw_shared_ptr<sstables::sstable> > > > >)::{lambda(partition_key const&)#1}>::_M_invoke
> | | | |
> | | | |--66.67%-- 0x1b5c280
> | | | |
> | | | |--27.78%-- managed_vector<atomic_cell_or_collection, 5u, unsigned int>::resize
> | | | | row::apply
> | | | | mutation_partition_applier::accept_row_cell
> | | | | mutation_partition_view::accept
> | | | |
> | | | --5.56%-- 0x2a4399
> | | |
> | | |--2.08%-- smp_message_queue::smp_message_queue
> | | | |
> | | | |--60.00%-- 0x60f0000c3000
> | | | |
> | | | |--10.00%-- 0x6000002d7240
> | | | |
> | | | |--10.00%-- 0x19
> | | | |
> | | | |--10.00%-- 0xb
> | | | |
> | | | --10.00%-- 0x7
> | | |
> | | |--1.93%-- smp_message_queue::process_queue<4ul, smp_message_queue::process_completions()::{lambda(smp_message_queue::work_item*)#1}>
> | | |
> | | |--1.63%-- __vdso_clock_gettime
> | | | |
> | | | --100.00%-- __clock_gettime
> | | | std::chrono::_V2::system_clock::now
> | | | 0xa63209
> | | |
> | | |--1.49%-- memory::small_pool::deallocate
> | | | |
> | | | |--40.00%-- managed_vector<atomic_cell_or_collection, 5u, unsigned int>::emplace_back<atomic_cell_or_collection>
> | | | |
> | | | |--20.00%-- std::_Hashtable<unsigned long, std::pair<unsigned long const, service::storage_proxy::rh_entry>, std::allocator<std::pair<unsigned long const, service::storage_proxy::rh_entry> >, std::__detail::_Select1st, std::equal_to<unsigned long>, std::hash<unsigned long>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_erase
> | | | | service::storage_proxy::got_response
> | | | | _ZN12continuationIZN6futureIJEE4thenIZZN7service13storage_proxy22send_to_live_endpointsEmENKUlRSt4pairIK13basic_sstringIcjLj15EESt6vectorIN3gms12inet_addressESaISB_EEEE_clESF_EUlvE_S1_EET0_OT_EUlSK_E_JEE3runEv
> | | | | reactor::del_timer
> | | | | 0x6100000e2040
> | | | |
> | | | |--10.00%-- cql3::statements::modification_statement::get_mutations
> | | | |
> | | | |--10.00%-- cql3::statements::modification_statement::build_partition_keys
> | | | | cql3::statements::modification_statement::create_exploded_clustering_prefix
> | | | | 0x60c014be0b00
> | | | |
> | | | |--10.00%-- mutation_partition::~mutation_partition
> | | | | std::vector<mutation, std::allocator<mutation> >::~vector
> | | | | service::storage_proxy::mutate_with_triggers
> | | | | cql3::statements::modification_statement::execute_without_condition
> | | | | cql3::statements::modification_statement::execute
> | | | | cql3::query_processor::process_statement
> | | | | transport::cql_server::connection::process_execute
> | | | | transport::cql_server::connection::process_request_one
> | | | | futurize<future<std::pair<foreign_ptr<shared_ptr<transport::cql_server::response> >, service::client_state> > >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}::operator()(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&) const::{lambda(temporary_buffer<char>)#1}::operator()(temporary_buffer) const::{lambda()#1}::operator()()::{lambda()#1}&>
> | | | | futurize<future<> >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}::operator()(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&) const::{lambda(temporary_buffer<char>)#1}, temporary_buffer>
> | | | | futurize<future<> >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}, future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> > >
> | | | | 0x8961de
> | | | |
> | | | --10.00%-- object_deleter_impl<deleter>::~object_deleter_impl
> | | | _ZN12continuationIZN6futureIJEE12then_wrappedIZZNS1_7finallyIZ7do_withI11foreign_ptrI10shared_ptrIN9transport10cql_server8responseEEEZZNS8_10connection14write_responseEOSB_ENUlvE_clEvEUlRT_E_EDaOSF_OT0_EUlvE_EES1_SI_ENUlS1_E_clES1_EUlSF_E_S1_EESJ_SI_EUlSI_E_JEED0Ev
> | | | 0x61a0000c3db0
> | | |
> | | |--1.34%-- dht::decorated_key::equal
> | | | |
> | | | |--83.33%-- 0x607000138f00
> | | | |
> | | | --16.67%-- 0x60a0000e0f40
> | | |
> | | |--1.34%-- service::storage_proxy::send_to_live_endpoints
> | | |
> | | |--1.19%-- transport::cql_server::connection::process_execute
> | | | transport::cql_server::connection::process_request_one
> | | | futurize<future<std::pair<foreign_ptr<shared_ptr<transport::cql_server::response> >, service::client_state> > >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}::operator()(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&) const::{lambda(temporary_buffer<char>)#1}::operator()(temporary_buffer) const::{lambda()#1}::operator()()::{lambda()#1}&>
> | | | futurize<future<> >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}::operator()(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&) const::{lambda(temporary_buffer<char>)#1}, temporary_buffer>
> | | | futurize<future<> >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}, future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> > >
> | | | |
> | | | |--87.50%-- transport::cql_server::connection::process_request
> | | | | do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>
> | | | | do_void_futurize_apply<void do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>(transport::cql_server::connection::process()::{lambda()#1}&, transport::cql_server::connection::process()::{lambda()#2}&&, promise<>)::{lambda(future<>)#1}, promise<> >
> | | | | do_void_futurize_apply<void do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>(transport::cql_server::connection::process()::{lambda()#1}&, transport::cql_server::connection::process()::{lambda()#2}&&, promise<>)::{lambda(future<>)#1}, promise<> >
> | | | | 0x60e0000c3000
> | | | |
> | | | --12.50%-- 0x8961de
> | | |
> | | |--1.19%-- reactor::run
> | | | |
> | | | |--87.50%-- smp::configure(boost::program_options::variables_map)::{lambda()#1}::operator()
> | | | | continuation<future<temporary_buffer<char> > future<>::then<future<temporary_buffer<char> > file::dma_read_bulk<char>(unsigned long, unsigned long)::{lambda(unsigned long)#1}::operator()(unsigned long)::{lambda()#3}, future<temporary_buffer<char> > >(future<temporary_buffer<char> > file::dma_read_bulk<char>(unsigned long, unsigned long)::{lambda(unsigned long)#1}::operator()(unsigned long)::{lambda()#3}&&)::{lambda(future<temporary_buffer<char> >)#1}>::run
> | | | | 0x600000043d00
> | | | |
> | | | --12.50%-- app_template::run_deprecated
> | | | main
> | | | __libc_start_main
> | | | _GLOBAL__sub_I__ZN3org6apache9cassandra21g_cassandra_constantsE
> | | | 0x7f4ae20c9fa0
> | | |
> | | |--1.04%-- __clock_gettime
> | | | std::chrono::_V2::system_clock::now
> | | | |
> | | | |--42.86%-- reactor::run
> | | | | smp::configure(boost::program_options::variables_map)::{lambda()#1}::operator()
> | | | | continuation<future<temporary_buffer<char> > future<>::then<future<temporary_buffer<char> > file::dma_read_bulk<char>(unsigned long, unsigned long)::{lambda(unsigned long)#1}::operator()(unsigned long)::{lambda()#3}, future<temporary_buffer<char> > >(future<temporary_buffer<char> > file::dma_read_bulk<char>(unsigned long, unsigned long)::{lambda(unsigned long)#1}::operator()(unsigned long)::{lambda()#3}&&)::{lambda(future<temporary_buffer<char> >)#1}>::run
> | | | | 0x600000043d00
> | | | |
> | | | |--14.29%-- 0xa63209
> | | | |
> | | | |--14.29%-- continuation<future<> future<>::finally<auto do_with<std::vector<frozen_mutation, std::allocator<frozen_mutation> >, shared_ptr<service::storage_proxy>, service::storage_proxy::init_messaging_service()::{lambda(std::vector<frozen_mutation, std::allocator<frozen_mutation> >)#1}::operator()(std::vector<frozen_mutation, std::allocator<frozen_mutation> >) const::{lambda(std::vector<frozen_mutation, std::allocator<frozen_mutation> > const&, shared_ptr<service::storage_proxy>&)#1}>(std::vector<frozen_mutation, std::allocator<frozen_mutation> >&&, shared_ptr<service::storage_proxy>&&, service::storage_proxy::init_messaging_service()::{lambda(std::vector<frozen_mutation, std::allocator<frozen_mutation> >)#1}::operator()(std::vector<frozen_mutation, std::allocator<frozen_mutation> >) const::{lambda(std::vector<frozen_mutation, std::allocator<frozen_mutation> > const&, shared_ptr<service::storage_proxy>&)#1}&&)::
{lambda()#1}>(service::storage_proxy::init_messaging_service()::{lambda(std::vector<frozen_mutation, std::a
> | | | | 0x2b7434
> | | | |
> | | | |--14.29%-- _ZN8futurizeI6futureIJSt10unique_ptrIN4cql317update_parametersESt14default_deleteIS3_EEEEE5applyIZNS2_10statements22modification_statement22make_update_parametersERN7seastar7shardedIN7service13storage_proxyEEE13lw_shared_ptrISt6vectorI13partition_keySaISK_EEESI_I26exploded_clustering_prefixERKNS2_13query_optionsEblEUlT_E_JNSt12experimental15fundamentals_v18optionalINS3_13prefetch_dataEEEEEES7_OST_OSt5tupleIJDpT0_EE
> | | | | cql3::statements::modification_statement::make_update_parameters
> | | | | cql3::statements::modification_statement::get_mutations
> | | | | cql3::statements::modification_statement::execute_without_condition
> | | | | cql3::query_options::query_options
> | | | | 0x7f4ad6bf80e0
> | | | |
> | | | --14.29%-- database::apply_in_memory
> | | | database::do_apply
> | | | _ZN12continuationIZN6futureIJEE4thenIZN8database5applyERK15frozen_mutationEUlvE_S1_EET0_OT_EUlSA_E_JEE3runEv
> | | | reactor::del_timer
> | | | 0x6090000e2040
> | | |
> | | |--1.04%-- memory::small_pool::allocate
> | | | |
> | | | |--14.29%-- 0x5257c379469d9
> | | | |
> | | | |--14.29%-- 0x609002b9fe98
> | | | |
> | | | |--14.29%-- 0x13c8b90
> | | | |
> | | | |--14.29%-- 0x60f000190710
> | | | |
> | | | |--14.29%-- 0x25
> | | | |
> | | | |--14.29%-- 0x7f4ad6bf84c0
> | | | |
> | | | --14.29%-- 0x7f4ad53f81f0
> | | |
> | | |--0.89%-- db::serializer<atomic_cell_view>::serializer
> | | | mutation_partition_serializer::write_without_framing
> | | | frozen_mutation::frozen_mutation
> | | | frozen_mutation::frozen_mutation
> | | |
> | | |--0.89%-- do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>
> | | | do_void_futurize_apply<void do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>(transport::cql_server::connection::process()::{lambda()#1}&, transport::cql_server::connection::process()::{lambda()#2}&&, promise<>)::{lambda(future<>)#1}, promise<> >
> | | | do_void_futurize_apply<void do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>(transport::cql_server::connection::process()::{lambda()#1}&, transport::cql_server::connection::process()::{lambda()#2}&&, promise<>)::{lambda(future<>)#1}, promise<> >
> | | | 0x60f0000c3000
> | | |
> | | |--0.89%-- futurize<future<> >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}::operator()(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&) const::{lambda(temporary_buffer<char>)#1}, temporary_buffer>
> | | | futurize<future<> >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}, future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> > >
> | | | transport::cql_server::connection::process_request
> | | | do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>
> | | | do_void_futurize_apply<void do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>(transport::cql_server::connection::process()::{lambda()#1}&, transport::cql_server::connection::process()::{lambda()#2}&&, promise<>)::{lambda(future<>)#1}, promise<> >
> | | | do_void_futurize_apply<void do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>(transport::cql_server::connection::process()::{lambda()#1}&, transport::cql_server::connection::process()::{lambda()#2}&&, promise<>)::{lambda(future<>)#1}, promise<> >
> | | | |
> | | | |--83.33%-- 0x6090000c3000
> | | | |
> | | | --16.67%-- 0x600000044400
> | | |
> | | |--0.89%-- std::_Function_handler<void (), reactor::run()::{lambda()#8}>::_M_invoke
> | | | |
> | | | |--50.00%-- reactor::run
> | | | | smp::configure(boost::program_options::variables_map)::{lambda()#1}::operator()
> | | | | continuation<future<temporary_buffer<char> > future<>::then<future<temporary_buffer<char> > file::dma_read_bulk<char>(unsigned long, unsigned long)::{lambda(unsigned long)#1}::operator()(unsigned long)::{lambda()#3}, future<temporary_buffer<char> > >(future<temporary_buffer<char> > file::dma_read_bulk<char>(unsigned long, unsigned long)::{lambda(unsigned long)#1}::operator()(unsigned long)::{lambda()#3}&&)::{lambda(future<temporary_buffer<char> >)#1}>::run
> | | | | 0x600000043d00
> | | | |
> | | | --50.00%-- reactor::signals::signal_handler::signal_handler
> | | | 0x3e8
> | | |
> | | |--0.74%-- db::commitlog::segment::allocate
> | | | |
> | | | --100.00%-- db::commitlog::add
> | | | database::do_apply
> | | | |
> | | | |--75.00%-- database::apply
> | | | | smp_message_queue::async_work_item<future<> seastar::sharded<database>::invoke_on<service::storage_proxy::mutate_locally(mutation const&)::{lambda(database&)#1}, future<> >(unsigned int, service::storage_proxy::mutate_locally(mutation const&)::{lambda(database&)#1}&&)::{lambda()#1}>::process
> | | | | smp_message_queue::process_queue<2ul, smp_message_queue::process_incoming()::{lambda(smp_message_queue::work_item*)#1}>
> | | | | boost::lockfree::detail::ringbuffer_base<smp_message_queue::work_item*>::pop
> | | | | boost::program_options::variables_map::get
> | | | |
> | | | --25.00%-- _ZN12continuationIZN6futureIJEE4thenIZN8database5applyERK15frozen_mutationEUlvE_S1_EET0_OT_EUlSA_E_JEE3runEv
> | | | reactor::del_timer
> | | | 0x60b0000e2040
> | | |
> | | |--0.74%-- service::storage_proxy::create_write_response_handler
> | | |
> | | |--0.74%-- transport::cql_server::connection::process_request_one
> | | | futurize<future<std::pair<foreign_ptr<shared_ptr<transport::cql_server::response> >, service::client_state> > >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}::operator()(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&) const::{lambda(temporary_buffer<char>)#1}::operator()(temporary_buffer) const::{lambda()#1}::operator()()::{lambda()#1}&>
> | | | futurize<future<> >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}::operator()(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&) const::{lambda(temporary_buffer<char>)#1}, temporary_buffer>
> | | | futurize<future<> >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}, future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> > >
> | | | |
> | | | |--80.00%-- transport::cql_server::connection::process_request
> | | | | do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>
> | | | | do_void_futurize_apply<void do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>(transport::cql_server::connection::process()::{lambda()#1}&, transport::cql_server::connection::process()::{lambda()#2}&&, promise<>)::{lambda(future<>)#1}, promise<> >
> | | | | do_void_futurize_apply<void do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>(transport::cql_server::connection::process()::{lambda()#1}&, transport::cql_server::connection::process()::{lambda()#2}&&, promise<>)::{lambda(future<>)#1}, promise<> >
> | | | | 0x60a0000c3000
> | | | |
> | | | --20.00%-- 0x8961de
> | | |
> | | |--0.74%-- compound_type<(allow_prefixes)0>::compare
> | | | |
> | | | |--20.00%-- 0x6030056c0f20
> | | | |
> | | | |--20.00%-- boost::intrusive::bstbase2<boost::intrusive::mhtraits<rows_entry, boost::intrusive::set_member_hook<void, void, void, void>, &rows_entry::_link>, rows_entry::compare, (boost::intrusive::algo_types)5, boost::intrusive::detail::default_header_holder<boost::intrusive::rbtree_node_traits<void*, false> > >::find
> | | | | mutation_partition::clustered_row
> | | | | mutation::set_clustered_cell
> | | | | cql3::constants::setter::execute
> | | | | cql3::statements::update_statement::add_update_for_key
> | | | | _ZN8futurizeI6futureIJSt6vectorI8mutationSaIS2_EEEEE5applyIZN4cql310statements22modification_statement13get_mutationsERN7seastar7shardedIN7service13storage_proxyEEERKNS8_13query_optionsEblEUlT_E_JSt10unique_ptrINS8_17update_parametersESt14default_deleteISN_EEEEES5_OSK_OSt5tupleIJDpT0_EE
> | | | | cql3::statements::modification_statement::get_mutations
> | | | | cql3::statements::modification_statement::execute_without_condition
> | | | | cql3::query_options::query_options
> | | | | 0x7f4adb3f80e0
> | | | |
> | | | |--20.00%-- compound_type<(allow_prefixes)0>::compare
> | | | |
> | | | |--20.00%-- mutation_partition::clustered_row
> | | | | boost::intrusive::bstree_impl<boost::intrusive::mhtraits<rows_entry, boost::intrusive::set_member_hook<void, void, void, void>, &rows_entry::_link>, rows_entry::compare, unsigned long, true, (boost::intrusive::algo_types)5, boost::intrusive::detail::default_header_holder<boost::intrusive::rbtree_node_traits<void*, false> > >::insert_unique
> | | | | boost::intrusive::bstree_algorithms<boost::intrusive::rbtree_node_traits<void*, false> >::prev_node
> | | | | 0x12d
> | | | |
> | | | --20.00%-- 0x60f00052daf0
> | | |
> | | |--0.74%-- __memmove_ssse3_back
> | | | |
> | | | |--40.00%-- output_stream<char>::write
> | | | | |
> | | | | |--50.00%-- transport::cql_server::response::output
> | | | | | futurize<future<> >::apply<transport::cql_server::connection::write_response(foreign_ptr<shared_ptr<transport::cql_server::response> >&&)::{lambda()#1}>
> | | | | |
> | | | | --50.00%-- 0x7c7fb2
> | | | | 0x5257c37847fa0
> | | | |
> | | | |--20.00%-- transport::cql_server::connection::read_short_bytes
> | | | | transport::cql_server::connection::process_query
> | | | | 0x7f4ada7f86f0
> | | | |
> | | | |--20.00%-- transport::cql_server::response::output
> | | | | futurize<future<> >::apply<transport::cql_server::connection::write_response(foreign_ptr<shared_ptr<transport::cql_server::response> >&&)::{lambda()#1}>
> | | | | 0x2
> | | | |
> | | | --20.00%-- smp_message_queue::flush_response_batch
> | | | boost::lockfree::detail::ringbuffer_base<smp_message_queue::work_item*>::pop
> | | | boost::program_options::variables_map::get
> | | |
> | | |--0.74%-- syscall_work_queue::work_item_returning<syscall_result_extra<stat>, reactor::file_size(basic_sstring<char, unsigned int, 15u>)::{lambda()#1}>::~work_item_returning
> | | | |
> | | | |--60.00%-- 0x6130000c3000
> | | | |
> | | | |--20.00%-- 0x608001fe59a0
> | | | |
> | | | --20.00%-- 0x16
> | | |
> | | |--0.74%-- __memset_sse2
> | | | |
> | | | |--40.00%-- std::_Hashtable<range<dht::token>, std::pair<range<dht::token> const, std::unordered_set<gms::inet_address, std::hash<gms::inet_address>, std::equal_to<gms::inet_address>, std::allocator<gms::inet_address> > >, std::allocator<std::pair<range<dht::token> const, std::unordered_set<gms::inet_address, std::hash<gms::inet_address>, std::equal_to<gms::inet_address>, std::allocator<gms::inet_address> > > >, std::__detail::_Select1st, std::equal_to<range<dht::token> >, std::hash<range<dht::token> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::~_Hashtable
> | | | | locator::token_metadata::pending_endpoints_for
> | | | | service::storage_proxy::create_write_response_handler
> | | | | service::storage_proxy::mutate_prepare<std::vector<mutation, std::allocator<mutation> >, service::storage_proxy::mutate_prepare(std::vector<mutation, std::allocator<mutation> >&, db::consistency_level, db::write_type)::{lambda(mutation const&, db::consistency_level, db::write_type)#1}>
> | | | | service::storage_proxy::mutate
> | | | | service::storage_proxy::mutate_with_triggers
> | | | | cql3::statements::modification_statement::execute_without_condition
> | | | | cql3::statements::modification_statement::execute
> | | | | cql3::query_processor::process_statement
> | | | | transport::cql_server::connection::process_execute
> | | | | transport::cql_server::connection::process_request_one
> | | | | futurize<future<std::pair<foreign_ptr<shared_ptr<transport::cql_server::response> >, service::client_state> > >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}::operator()(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&) const::{lambda(temporary_buffer<char>)#1}::operator()(temporary_buffer) const::{lambda()#1}::operator()()::{lambda()#1}&>
> | | | | futurize<future<> >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}::operator()(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&) const::{lambda(temporary_buffer<char>)#1}, temporary_buffer>
> | | | | futurize<future<> >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}, future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> > >
> | | | | transport::cql_server::connection::process_request
> | | | | do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>
> | | | | do_void_futurize_apply<void do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>(transport::cql_server::connection::process()::{lambda()#1}&, transport::cql_server::connection::process()::{lambda()#2}&&, promise<>)::{lambda(future<>)#1}, promise<> >
> | | | | do_void_futurize_apply<void do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>(transport::cql_server::connection::process()::{lambda()#1}&, transport::cql_server::connection::process()::{lambda()#2}&&, promise<>)::{lambda(future<>)#1}, promise<> >
> | | | | 0x6020000c3000
> | | | |
> | | | |--40.00%-- service::digest_read_resolver::~digest_read_resolver
> | | | | |
> | | | | --100.00%-- 0x610002612b50
> | | | |
> | | | --20.00%-- std::_Hashtable<basic_sstring<char, unsigned int, 15u>, std::pair<basic_sstring<char, unsigned int, 15u> const, std::vector<gms::inet_address, std::allocator<gms::inet_address> > >, std::allocator<std::pair<basic_sstring<char, unsigned int, 15u> const, std::vector<gms::inet_address, std::allocator<gms::inet_address> > > >, std::__detail::_Select1st, std::equal_to<basic_sstring<char, unsigned int, 15u> >, std::hash<basic_sstring<char, unsigned int, 15u> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::~_Hashtable
> | | | service::storage_proxy::send_to_live_endpoints
> | | | parallel_for_each<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, service::storage_proxy::mutate_begin(std::vector<unsigned long, std::allocator<unsigned long> >, db::consistency_level)::{lambda(unsigned long)#1}>
> | | | service::storage_proxy::mutate
> | | | service::storage_proxy::mutate_with_triggers
> | | | cql3::statements::modification_statement::execute_without_condition
> | | | cql3::statements::modification_statement::execute
> | | | cql3::query_processor::process_statement
> | | | transport::cql_server::connection::process_execute
> | | | transport::cql_server::connection::process_request_one
> | | | futurize<future<std::pair<foreign_ptr<shared_ptr<transport::cql_server::response> >, service::client_state> > >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}::operator()(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&) const::{lambda(temporary_buffer<char>)#1}::operator()(temporary_buffer) const::{lambda()#1}::operator()()::{lambda()#1}&>
> | | | futurize<future<> >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}::operator()(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&) const::{lambda(temporary_buffer<char>)#1}, temporary_buffer>
> | | | futurize<future<> >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}, future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> > >
> | | | transport::cql_server::connection::process_request
> | | | do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>
> | | | do_void_futurize_apply<void do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>(transport::cql_server::connection::process()::{lambda()#1}&, transport::cql_server::connection::process()::{lambda()#2}&&, promise<>)::{lambda(future<>)#1}, promise<> >
> | | | do_void_futurize_apply<void do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>(transport::cql_server::connection::process()::{lambda()#1}&, transport::cql_server::connection::process()::{lambda()#2}&&, promise<>)::{lambda(future<>)#1}, promise<> >
> | | | 0x6070000c3000
> | | |
> | | |--0.74%-- reactor::del_timer
> | | | |
> | | | |--80.00%-- 0x60a0000e2040
> | | | |
> | | | --20.00%-- 0x6080000c3db0
> | | |
> | | |--0.59%-- unimplemented::operator<<
> | | | |
> | | | |--25.00%-- _ZN12continuationIZN6futureIJ10shared_ptrIN9transport8messages14result_messageEEEE4thenIZN4cql315query_processor17process_statementES1_INS8_13cql_statementEERN7service11query_stateERKNS8_13query_optionsEEUlT_E_S6_EET0_OSI_EUlSL_E_JS5_EED2Ev
> | | | | 0x600100000008
> | | | |
> | | | |--25.00%-- floating_type_impl<float>::from_string
> | | | |
> | | | |--25.00%-- 0x60e0000e4c10
> | | | |
> | | | --25.00%-- _ZN12continuationIZN6futureIJ10shared_ptrIN9transport8messages14result_messageEEEE4thenIZN4cql315query_processor17process_statementES1_INS8_13cql_statementEERN7service11query_stateERKNS8_13query_optionsEEUlT_E_S6_EET0_OSI_EUlSL_E_JS5_EED2Ev
> | | | 0x600100000008
> | | |
> | | |--0.59%-- std::_Hashtable<unsigned long, std::pair<unsigned long const, service::storage_proxy::rh_entry>, std::allocator<std::pair<unsigned long const, service::storage_proxy::rh_entry> >, std::__detail::_Select1st, std::equal_to<unsigned long>, std::hash<unsigned long>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_insert_unique_node
> | | | service::storage_proxy::register_response_handler
> | | | service::storage_proxy::create_write_response_handler
> | | | service::storage_proxy::create_write_response_handler
> | | | service::storage_proxy::mutate_prepare<std::vector<mutation, std::allocator<mutation> >, service::storage_proxy::mutate_prepare(std::vector<mutation, std::allocator<mutation> >&, db::consistency_level, db::write_type)::{lambda(mutation const&, db::consistency_level, db::write_type)#1}>
> | | | service::storage_proxy::mutate
> | | | service::storage_proxy::mutate_with_triggers
> | | | cql3::statements::modification_statement::execute_without_condition
> | | | cql3::statements::modification_statement::execute
> | | | cql3::query_processor::process_statement
> | | | transport::cql_server::connection::process_execute
> | | | transport::cql_server::connection::process_request_one
> | | | futurize<future<std::pair<foreign_ptr<shared_ptr<transport::cql_server::response> >, service::client_state> > >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}::operator()(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&) const::{lambda(temporary_buffer<char>)#1}::operator()(temporary_buffer) const::{lambda()#1}::operator()()::{lambda()#1}&>
> | | | futurize<future<> >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}::operator()(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&) const::{lambda(temporary_buffer<char>)#1}, temporary_buffer>
> | | | futurize<future<> >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}, future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> > >
> | | | transport::cql_server::connection::process_request
> | | | do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>
> | | | do_void_futurize_apply<void do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>(transport::cql_server::connection::process()::{lambda()#1}&, transport::cql_server::connection::process()::{lambda()#2}&&, promise<>)::{lambda(future<>)#1}, promise<> >
> | | | do_void_futurize_apply<void do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>(transport::cql_server::connection::process()::{lambda()#1}&, transport::cql_server::connection::process()::{lambda()#2}&&, promise<>)::{lambda(future<>)#1}, promise<> >
> | | | 0x60b0000c3000
> | | |
> | | |--0.59%-- mutation::set_clustered_cell
> | | | |
> | | | |--75.00%-- 0xa
> | | | |
> | | | --25.00%-- cql3::constants::setter::execute
> | | | cql3::statements::update_statement::add_update_for_key
> | | | _ZN8futurizeI6futureIJSt6vectorI8mutationSaIS2_EEEEE5applyIZN4cql310statements22modification_statement13get_mutationsERN7seastar7shardedIN7service13storage_proxyEEERKNS8_13query_optionsEblEUlT_E_JSt10unique_ptrINS8_17update_parametersESt14default_deleteISN_EEEEES5_OSK_OSt5tupleIJDpT0_EE
> | | | cql3::statements::modification_statement::get_mutations
> | | | cql3::statements::modification_statement::execute_without_condition
> | | | cql3::query_options::query_options
> | | | 0x7f4ad89f80e0
> | | |
> | | |--0.59%-- memory::small_pool::small_pool
> | | | |
> | | | |--25.00%-- memory::stats
> | | | | boost::program_options::variables_map::get
> | | | |
> | | | |--25.00%-- memory::reclaimer::~reclaimer
> | | | | 0x1e
> | | | |
> | | | |--25.00%-- memory::allocate_aligned
> | | | |
> | | | --25.00%-- memory::small_pool::add_more_objects
> | | | memory::small_pool::add_more_objects
> | | | 0x6100000e0310
> | | |
> | | |--0.59%-- __memcpy_sse2_unaligned
> | | | |
> | | | |--50.00%-- mutation_partition_applier::accept_row_cell
> | | | | mutation_partition_view::accept
> | | | | boost::intrusive::bstree_algorithms<boost::intrusive::rbtree_node_traits<void*, false> >::prev_node
> | | | | 0x12d
> | | | |
> | | | |--25.00%-- scanning_reader::operator()
> | | | | sstables::sstable::do_write_components
> | | | | sstables::sstable::prepare_write_components
> | | | | std::_Function_handler<void (), futurize<std::result_of<std::decay<sstables::sstable::write_components(mutation_reader, unsigned long, lw_shared_ptr<schema const>, unsigned long)::{lambda()#1}>::type ()>::type>::type seastar::async<sstables::sstable::write_components(mutation_reader, unsigned long, lw_shared_ptr<schema const>, unsigned long)::{lambda()#1}>(std::decay&&, (std::decay<sstables::sstable::write_components(mutation_reader, unsigned long, lw_shared_ptr<schema const>, unsigned long)::{lambda()#1}>::type&&)...)::{lambda(futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type> seastar::async<{lambda()#1}>(futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type>::type, std::decay<{lambda()#1}>::type&&)::work&)#1}::operator()(futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type> seastar::async<{lambda()#1}>(futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::typ
e>::type, std::decay<{lambda()#1}>::type&&)::work)::{lambda()#1}>::_M_invoke
> | | | | _GLOBAL__sub_I__ZN12app_templateC2Ev
> | | | |
> | | | --25.00%-- memtable::find_or_create_partition_slow
> | | | memtable::apply
> | | | database::apply_in_memory
> | | | database::do_apply
> | | | database::apply
> | | | smp_message_queue::async_work_item<future<> seastar::sharded<database>::invoke_on<service::storage_proxy::mutate_locally(mutation const&)::{lambda(database&)#1}, future<> >(unsigned int, service::storage_proxy::mutate_locally(mutation const&)::{lambda(database&)#1}&&)::{lambda()#1}>::process
> | | | smp_message_queue::process_queue<2ul, smp_message_queue::process_incoming()::{lambda(smp_message_queue::work_item*)#1}>
> | | | boost::lockfree::detail::ringbuffer_base<smp_message_queue::work_item*>::pop
> | | | boost::program_options::variables_map::get
> | | |
> | | |--0.59%-- smp_message_queue::flush_response_batch
> | | | |
> | | | |--25.00%-- boost::lockfree::detail::ringbuffer_base<smp_message_queue::work_item*>::pop
> | | | | boost::program_options::variables_map::get
> | | | |
> | | | |--25.00%-- 0x13
> | | | |
> | | | |--25.00%-- 0x7f4ad5ff8f40
> | | | |
> | | | --25.00%-- reactor::run
> | | | smp::configure(boost::program_options::variables_map)::{lambda()#1}::operator()
> | | | continuation<future<temporary_buffer<char> > future<>::then<future<temporary_buffer<char> > file::dma_read_bulk<char>(unsigned long, unsigned long)::{lambda(unsigned long)#1}::operator()(unsigned long)::{lambda()#3}, future<temporary_buffer<char> > >(future<temporary_buffer<char> > file::dma_read_bulk<char>(unsigned long, unsigned long)::{lambda(unsigned long)#1}::operator()(unsigned long)::{lambda()#3}&&)::{lambda(future<temporary_buffer<char> >)#1}>::run
> | | | 0x600000043d00
> | | --54.38%-- [...]
> | |
> | |--14.26%-- schedule_timeout
> | | |
> | | |--38.52%-- wait_for_completion
> | | | |
> | | | |--90.07%-- flush_work
> | | | | xlog_cil_force_lsn
> | | | | |
> | | | | |--96.85%-- _xfs_log_force_lsn
> | | | | | |
> | | | | | |--79.67%-- xfs_file_fsync
> | | | | | | vfs_fsync_range
> | | | | | | do_fsync
> | | | | | | sys_fdatasync
> | | | | | | entry_SYSCALL_64_fastpath
> | | | | | | |
> | | | | | | --100.00%-- 0x7f4ade4212ad
> | | | | | | syscall_work_queue::work_item_returning<syscall_result<int>, posix_file_impl::flush()::{lambda()#1}>::process
> | | | | | | 0x6030000c3ec0
> | | | | | |
> | | | | | --20.33%-- xfs_dir_fsync
> | | | | | vfs_fsync_range
> | | | | | do_fsync
> | | | | | sys_fdatasync
> | | | | | entry_SYSCALL_64_fastpath
> | | | | | |
> | | | | | --100.00%-- 0x7f4ade4212ad
> | | | | | syscall_work_queue::work_item_returning<syscall_result<int>, posix_file_impl::flush()::{lambda()#1}>::process
> | | | | | 0x6040000c3ec0
> | | | | |
> | | | | --3.15%-- _xfs_log_force
> | | | | xfs_log_force
> | | | | xfs_buf_lock
> | | | | _xfs_buf_find
> | | | | xfs_buf_get_map
> | | | | xfs_trans_get_buf_map
> | | | | xfs_btree_get_bufl
> | | | | xfs_bmap_extents_to_btree
> | | | | xfs_bmap_add_extent_hole_real
> | | | | xfs_bmapi_write
> | | | | xfs_iomap_write_direct
> | | | | __xfs_get_blocks
> | | | | xfs_get_blocks_direct
> | | | | do_blockdev_direct_IO
> | | | | __blockdev_direct_IO
> | | | | xfs_vm_direct_IO
> | | | | xfs_file_dio_aio_write
> | | | | xfs_file_write_iter
> | | | | aio_run_iocb
> | | | | do_io_submit
> | | | | sys_io_submit
> | | | | entry_SYSCALL_64_fastpath
> | | | | io_submit
> | | | | 0x46d98a
> | | | |
> | | | --9.93%-- submit_bio_wait
> | | | blkdev_issue_flush
> | | | xfs_blkdev_issue_flush
> | | | xfs_file_fsync
> | | | vfs_fsync_range
> | | | do_fsync
> | | | sys_fdatasync
> | | | entry_SYSCALL_64_fastpath
> | | | |
> | | | --100.00%-- 0x7f4ade4212ad
> | | | syscall_work_queue::work_item_returning<syscall_result<int>, posix_file_impl::flush()::{lambda()#1}>::process
> | | | 0x6030000c3ec0
> | | |
> | | |--32.79%-- io_schedule_timeout
> | | | bit_wait_io
> | | | __wait_on_bit
> | | | |
> | | | |--51.67%-- wait_on_page_bit
> | | | | |
> | | | | |--95.16%-- filemap_fdatawait_range
> | | | | | filemap_write_and_wait_range
> | | | | | xfs_file_fsync
> | | | | | vfs_fsync_range
> | | | | | do_fsync
> | | | | | sys_fdatasync
> | | | | | entry_SYSCALL_64_fastpath
> | | | | | 0x7f4ade4212ad
> | | | | | syscall_work_queue::work_item_returning<syscall_result<int>, posix_file_impl::flush()::{lambda()#1}>::process
> | | | | | 0x60b0000c3ec0
> | | | | |
> | | | | --4.84%-- __migration_entry_wait
> | | | | migration_entry_wait
> | | | | handle_mm_fault
> | | | | __do_page_fault
> | | | | do_page_fault
> | | | | page_fault
> | | | | std::_Function_handler<void (), httpd::http_server::_date_format_timer::{lambda()#1}>::_M_invoke
> | | | | |
> | | | | --100.00%-- service::storage_proxy::mutate_prepare<std::vector<mutation, std::allocator<mutation> >, service::storage_proxy::mutate_prepare(std::vector<mutation, std::allocator<mutation> >&, db::consistency_level, db::write_type)::{lambda(mutation const&, db::consistency_level, db::write_type)#1}>
> | | | | service::storage_proxy::mutate
> | | | | service::storage_proxy::mutate_with_triggers
> | | | | cql3::statements::modification_statement::execute_without_condition
> | | | | cql3::statements::modification_statement::execute
> | | | | cql3::query_processor::process_statement
> | | | | transport::cql_server::connection::process_execute
> | | | | transport::cql_server::connection::process_request_one
> | | | | futurize<future<std::pair<foreign_ptr<shared_ptr<transport::cql_server::response> >, service::client_state> > >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}::operator()(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&) const::{lambda(temporary_buffer<char>)#1}::operator()(temporary_buffer) const::{lambda()#1}::operator()()::{lambda()#1}&>
> | | | | futurize<future<> >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}::operator()(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&) const::{lambda(temporary_buffer<char>)#1}, temporary_buffer>
> | | | | futurize<future<> >::apply<transport::cql_server::connection::process_request()::{lambda(future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> >&&)#1}, future<std::experimental::fundamentals_v1::optional<transport::cql_binary_frame_v3> > >
> | | | | transport::cql_server::connection::process_request
> | | | | do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>
> | | | | do_void_futurize_apply<void do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>(transport::cql_server::connection::process()::{lambda()#1}&, transport::cql_server::connection::process()::{lambda()#2}&&, promise<>)::{lambda(future<>)#1}, promise<> >
> | | | | do_void_futurize_apply<void do_until_continued<transport::cql_server::connection::process()::{lambda()#2}, transport::cql_server::connection::process()::{lambda()#1}&>(transport::cql_server::connection::process()::{lambda()#1}&, transport::cql_server::connection::process()::{lambda()#2}&&, promise<>)::{lambda(future<>)#1}, promise<> >
> | | | | 0x6140000c3000
> | | | |
> | | | --48.33%-- out_of_line_wait_on_bit
> | | | block_truncate_page
> | | | xfs_setattr_size
> | | | xfs_vn_setattr
> | | | notify_change
> | | | do_truncate
> | | | do_sys_ftruncate.constprop.15
> | | | sys_ftruncate
> | | | entry_SYSCALL_64_fastpath
> | | | __GI___ftruncate64
> | | | syscall_work_queue::work_item_returning<syscall_result_extra<stat>, posix_file_impl::stat()::{lambda()#1}>::process
> | | | |
> | | | |--13.79%-- 0x7f4ad29ff700
> | | | |
> | | | |--13.79%-- 0x7f4acdbff700
> | | | |
> | | | |--12.07%-- 0x7f4ad05ff700
> | | | |
> | | | |--12.07%-- 0x7f4acedff700
> | | | |
> | | | |--10.34%-- 0x7f4ad0bff700
> | | | |
> | | | |--6.90%-- 0x7f4ad2fff700
> | | | |
> | | | |--6.90%-- 0x7f4ad11ff700
> | | | |
> | | | |--6.90%-- 0x7f4acf9ff700
> | | | |
> | | | |--6.90%-- 0x7f4acf3ff700
> | | | |
> | | | |--6.90%-- 0x7f4ace7ff700
> | | | |
> | | | |--1.72%-- 0x7f4ad17ff700
> | | | |
> | | | --1.72%-- 0x7f4aca5ff700
> | | |
> | | --28.69%-- __down
> | | down
> | | xfs_buf_lock
> | | _xfs_buf_find
> | | xfs_buf_get_map
> | | |
> | | |--97.14%-- xfs_buf_read_map
> | | | xfs_trans_read_buf_map
> | | | |
> | | | |--98.04%-- xfs_read_agf
> | | | | xfs_alloc_read_agf
> | | | | xfs_alloc_fix_freelist
> | | | | |
> | | | | |--93.00%-- xfs_free_extent
> | | | | | xfs_bmap_finish
> | | | | | xfs_itruncate_extents
> | | | | | |
> | | | | | |--87.10%-- xfs_inactive_truncate
> | | | | | | xfs_inactive
> | | | | | | xfs_fs_evict_inode
> | | | | | | evict
> | | | | | | iput
> | | | | | | __dentry_kill
> | | | | | | dput
> | | | | | | __fput
> | | | | | | ____fput
> | | | | | | task_work_run
> | | | | | | do_notify_resume
> | | | | | | int_signal
> | | | | | | __libc_close
> | | | | | | std::experimental::fundamentals_v1::bad_optional_access::~bad_optional_access
> | | | | | |
> | | | | | --12.90%-- xfs_setattr_size
> | | | | | xfs_vn_setattr
> | | | | | notify_change
> | | | | | do_truncate
> | | | | | do_sys_ftruncate.constprop.15
> | | | | | sys_ftruncate
> | | | | | entry_SYSCALL_64_fastpath
> | | | | | |
> | | | | | --100.00%-- __GI___ftruncate64
> | | | | | syscall_work_queue::work_item_returning<syscall_result_extra<stat>, posix_file_impl::stat()::{lambda()#1}>::process
> | | | | | |
> | | | | | |--20.00%-- 0x7f4ad0bff700
> | | | | | |
> | | | | | |--20.00%-- 0x7f4acedff700
> | | | | | |
> | | | | | |--10.00%-- 0x7f4ad2fff700
> | | | | | |
> | | | | | |--10.00%-- 0x7f4ad17ff700
> | | | | | |
> | | | | | |--10.00%-- 0x7f4ad11ff700
> | | | | | |
> | | | | | |--10.00%-- 0x7f4ad05ff700
> | | | | | |
> | | | | | |--10.00%-- 0x7f4acf3ff700
> | | | | | |
> | | | | | --10.00%-- 0x7f4acdbff700
> | | | | |
> | | | | --7.00%-- xfs_alloc_vextent
> | | | | xfs_bmap_btalloc
> | | | | xfs_bmap_alloc
> | | | | xfs_bmapi_write
> | | | | xfs_iomap_write_direct
> | | | | __xfs_get_blocks
> | | | | xfs_get_blocks_direct
> | | | | do_blockdev_direct_IO
> | | | | __blockdev_direct_IO
> | | | | xfs_vm_direct_IO
> | | | | xfs_file_dio_aio_write
> | | | | xfs_file_write_iter
> | | | | aio_run_iocb
> | | | | do_io_submit
> | | | | sys_io_submit
> | | | | entry_SYSCALL_64_fastpath
> | | | | io_submit
> | | | | 0x46d98a
> | | | |
> | | | --1.96%-- xfs_read_agi
> | | | xfs_iunlink_remove
> | | | xfs_ifree
> | | | xfs_inactive_ifree
> | | | xfs_inactive
> | | | xfs_fs_evict_inode
> | | | evict
> | | | iput
> | | | __dentry_kill
> | | | dput
> | | | __fput
> | | | ____fput
> | | | task_work_run
> | | | do_notify_resume
> | | | int_signal
> | | | __libc_close
> | | | std::experimental::fundamentals_v1::bad_optional_access::~bad_optional_access
> | | |
> | | --2.86%-- xfs_trans_get_buf_map
> | | xfs_btree_get_bufl
> | | xfs_bmap_extents_to_btree
> | | xfs_bmap_add_extent_hole_real
> | | xfs_bmapi_write
> | | xfs_iomap_write_direct
> | | __xfs_get_blocks
> | | xfs_get_blocks_direct
> | | do_blockdev_direct_IO
> | | __blockdev_direct_IO
> | | xfs_vm_direct_IO
> | | xfs_file_dio_aio_write
> | | xfs_file_write_iter
> | | aio_run_iocb
> | | do_io_submit
> | | sys_io_submit
> | | entry_SYSCALL_64_fastpath
> | | io_submit
> | | 0x46d98a
> | |
> | |--13.48%-- eventfd_ctx_read
> | | eventfd_read
> | | __vfs_read
> | | vfs_read
> | | sys_read
> | | entry_SYSCALL_64_fastpath
> | | 0x7f4ade6f754d
> | | smp_message_queue::respond
> | | 0xffffffffffffffff
> | |
> | |--7.83%-- md_flush_request
> | | raid0_make_request
> | | md_make_request
> | | generic_make_request
> | | submit_bio
> | | |
> | | |--92.54%-- submit_bio_wait
> | | | blkdev_issue_flush
> | | | xfs_blkdev_issue_flush
> | | | xfs_file_fsync
> | | | vfs_fsync_range
> | | | do_fsync
> | | | sys_fdatasync
> | | | entry_SYSCALL_64_fastpath
> | | | |
> | | | --100.00%-- 0x7f4ade4212ad
> | | | syscall_work_queue::work_item_returning<syscall_result<int>, posix_file_impl::flush()::{lambda()#1}>::process
> | | | 0x6010000c3ec0
> | | |
> | | --7.46%-- _xfs_buf_ioapply
> | | xfs_buf_submit
> | | xlog_bdstrat
> | | xlog_sync
> | | xlog_state_release_iclog
> | | |
> | | |--73.33%-- _xfs_log_force_lsn
> | | | xfs_file_fsync
> | | | vfs_fsync_range
> | | | do_fsync
> | | | sys_fdatasync
> | | | entry_SYSCALL_64_fastpath
> | | | |
> | | | --100.00%-- 0x7f4ade4212ad
> | | | syscall_work_queue::work_item_returning<syscall_result<int>, posix_file_impl::flush()::{lambda()#1}>::process
> | | | 0x6080000c3ec0
> | | |
> | | --26.67%-- _xfs_log_force
> | | xfs_log_force
> | | xfs_buf_lock
> | | _xfs_buf_find
> | | xfs_buf_get_map
> | | xfs_trans_get_buf_map
> | | xfs_btree_get_bufl
> | | xfs_bmap_extents_to_btree
> | | xfs_bmap_add_extent_hole_real
> | | xfs_bmapi_write
> | | xfs_iomap_write_direct
> | | __xfs_get_blocks
> | | xfs_get_blocks_direct
> | | do_blockdev_direct_IO
> | | __blockdev_direct_IO
> | | xfs_vm_direct_IO
> | | xfs_file_dio_aio_write
> | | xfs_file_write_iter
> | | aio_run_iocb
> | | do_io_submit
> | | sys_io_submit
> | | entry_SYSCALL_64_fastpath
> | | io_submit
> | | 0x46d98a
> | |
> | |--5.53%-- _xfs_log_force_lsn
> | | |
> | | |--80.28%-- xfs_file_fsync
> | | | vfs_fsync_range
> | | | do_fsync
> | | | sys_fdatasync
> | | | entry_SYSCALL_64_fastpath
> | | | |
> | | | --100.00%-- 0x7f4ade4212ad
> | | | syscall_work_queue::work_item_returning<syscall_result<int>, posix_file_impl::flush()::{lambda()#1}>::process
> | | | |
> | | | |--97.92%-- 0x60d0000c3ec0
> | | | |
> | | | |--1.04%-- 0x6020000c3ec0
> | | | |
> | | | --1.04%-- 0x600000557ec0
> | | |
> | | --19.72%-- xfs_dir_fsync
> | | vfs_fsync_range
> | | do_fsync
> | | sys_fdatasync
> | | entry_SYSCALL_64_fastpath
> | | |
> | | --100.00%-- 0x7f4ade4212ad
> | | syscall_work_queue::work_item_returning<syscall_result<int>, posix_file_impl::flush()::{lambda()#1}>::process
> | | 0x6040000c3ec0
> | |
> | |--1.25%-- rwsem_down_read_failed
> | | call_rwsem_down_read_failed
> | | |
> | | |--90.62%-- xfs_ilock
> | | | |
> | | | |--86.21%-- xfs_ilock_data_map_shared
> | | | | __xfs_get_blocks
> | | | | xfs_get_blocks_direct
> | | | | do_blockdev_direct_IO
> | | | | __blockdev_direct_IO
> | | | | xfs_vm_direct_IO
> | | | | xfs_file_dio_aio_write
> | | | | xfs_file_write_iter
> | | | | aio_run_iocb
> | | | | do_io_submit
> | | | | sys_io_submit
> | | | | entry_SYSCALL_64_fastpath
> | | | | |
> | | | | --100.00%-- io_submit
> | | | | 0x46d98a
> | | | |
> | | | |--6.90%-- xfs_file_fsync
> | | | | vfs_fsync_range
> | | | | do_fsync
> | | | | sys_fdatasync
> | | | | entry_SYSCALL_64_fastpath
> | | | | 0x7f4ade4212ad
> | | | | syscall_work_queue::work_item_returning<syscall_result<int>, posix_file_impl::flush()::{lambda()#1}>::process
> | | | | 0x6090000c3ec0
> | | | |
> | | | --6.90%-- xfs_dir_fsync
> | | | vfs_fsync_range
> | | | do_fsync
> | | | sys_fdatasync
> | | | entry_SYSCALL_64_fastpath
> | | | 0x7f4ade4212ad
> | | | syscall_work_queue::work_item_returning<syscall_result<int>, posix_file_impl::flush()::{lambda()#1}>::process
> | | | 0x6070000c3ec0
> | | |
> | | --9.38%-- xfs_log_commit_cil
> | | __xfs_trans_commit
> | | xfs_trans_commit
> | | |
> | | |--33.33%-- xfs_setattr_size
> | | | xfs_vn_setattr
> | | | notify_change
> | | | do_truncate
> | | | do_sys_ftruncate.constprop.15
> | | | sys_ftruncate
> | | | entry_SYSCALL_64_fastpath
> | | | __GI___ftruncate64
> | | | syscall_work_queue::work_item_returning<syscall_result_extra<stat>, posix_file_impl::stat()::{lambda()#1}>::process
> | | | 0x7f4acedff700
> | | |
> | | |--33.33%-- xfs_vn_update_time
> | | | file_update_time
> | | | xfs_file_aio_write_checks
> | | | xfs_file_dio_aio_write
> | | | xfs_file_write_iter
> | | | aio_run_iocb
> | | | do_io_submit
> | | | sys_io_submit
> | | | entry_SYSCALL_64_fastpath
> | | | io_submit
> | | | 0x46d98a
> | | |
> | | --33.33%-- xfs_bmap_add_attrfork
> | | xfs_attr_set
> | | xfs_initxattrs
> | | security_inode_init_security
> | | xfs_init_security
> | | xfs_generic_create
> | | xfs_vn_mknod
> | | xfs_vn_create
> | | vfs_create
> | | path_openat
> | | do_filp_open
> | | do_sys_open
> | | sys_open
> | | entry_SYSCALL_64_fastpath
> | | 0x7f4ade6f7cdd
> | | syscall_work_queue::work_item_returning<syscall_result<int>, reactor::open_file_dma(basic_sstring<char, unsigned int, 15u>, open_flags, file_open_options)::{lambda()#1}>::process
> | | 0xffffffffffffffff
> | |
> | |--0.97%-- rwsem_down_write_failed
> | | call_rwsem_down_write_failed
> | | xfs_ilock
> | | xfs_vn_update_time
> | | file_update_time
> | | xfs_file_aio_write_checks
> | | xfs_file_dio_aio_write
> | | xfs_file_write_iter
> | | aio_run_iocb
> | | do_io_submit
> | | sys_io_submit
> | | entry_SYSCALL_64_fastpath
> | | io_submit
> | | 0x46d98a
> | |
> | |--0.51%-- xlog_cil_force_lsn
> | | |
> | | |--92.31%-- _xfs_log_force_lsn
> | | | |
> | | | |--91.67%-- xfs_file_fsync
> | | | | vfs_fsync_range
> | | | | do_fsync
> | | | | sys_fdatasync
> | | | | entry_SYSCALL_64_fastpath
> | | | | 0x7f4ade4212ad
> | | | | syscall_work_queue::work_item_returning<syscall_result<int>, posix_file_impl::flush()::{lambda()#1}>::process
> | | | | 0x60b0000c3ec0
> | | | |
> | | | --8.33%-- xfs_dir_fsync
> | | | vfs_fsync_range
> | | | do_fsync
> | | | sys_fdatasync
> | | | entry_SYSCALL_64_fastpath
> | | | 0x7f4ade4212ad
> | | | syscall_work_queue::work_item_returning<syscall_result<int>, posix_file_impl::flush()::{lambda()#1}>::process
> | | | 0x60d0000c3ec0
> | | |
> | | --7.69%-- _xfs_log_force
> | | xfs_log_force
> | | xfs_buf_lock
> | | _xfs_buf_find
> | | xfs_buf_get_map
> | | xfs_trans_get_buf_map
> | | xfs_btree_get_bufl
> | | xfs_bmap_extents_to_btree
> | | xfs_bmap_add_extent_hole_real
> | | xfs_bmapi_write
> | | xfs_iomap_write_direct
> | | __xfs_get_blocks
> | | xfs_get_blocks_direct
> | | do_blockdev_direct_IO
> | | __blockdev_direct_IO
> | | xfs_vm_direct_IO
> | | xfs_file_dio_aio_write
> | | xfs_file_write_iter
> | | aio_run_iocb
> | | do_io_submit
> | | sys_io_submit
> | | entry_SYSCALL_64_fastpath
> | | io_submit
> | | 0x46d98a
> | --0.04%-- [...]
> |
> --3.82%-- preempt_schedule_common
> |
> |--99.02%-- _cond_resched
> | |
> | |--41.58%-- wait_for_completion
> | | |
> | | |--66.67%-- flush_work
> | | | xlog_cil_force_lsn
> | | | |
> | | | |--96.43%-- _xfs_log_force_lsn
> | | | | |
> | | | | |--77.78%-- xfs_file_fsync
> | | | | | vfs_fsync_range
> | | | | | do_fsync
> | | | | | sys_fdatasync
> | | | | | entry_SYSCALL_64_fastpath
> | | | | | 0x7f4ade4212ad
> | | | | | syscall_work_queue::work_item_returning<syscall_result<int>, posix_file_impl::flush()::{lambda()#1}>::process
> | | | | | 0x6030000c3ec0
> | | | | |
> | | | | --22.22%-- xfs_dir_fsync
> | | | | vfs_fsync_range
> | | | | do_fsync
> | | | | sys_fdatasync
> | | | | entry_SYSCALL_64_fastpath
> | | | | |
> | | | | --100.00%-- 0x7f4ade4212ad
> | | | | syscall_work_queue::work_item_returning<syscall_result<int>, posix_file_impl::flush()::{lambda()#1}>::process
> | | | | 0x6030000c3ec0
> | | | |
> | | | --3.57%-- _xfs_log_force
> | | | xfs_log_force
> | | | xfs_buf_lock
> | | | _xfs_buf_find
> | | | xfs_buf_get_map
> | | | xfs_trans_get_buf_map
> | | | xfs_btree_get_bufl
> | | | xfs_bmap_extents_to_btree
> | | | xfs_bmap_add_extent_hole_real
> | | | xfs_bmapi_write
> | | | xfs_iomap_write_direct
> | | | __xfs_get_blocks
> | | | xfs_get_blocks_direct
> | | | do_blockdev_direct_IO
> | | | __blockdev_direct_IO
> | | | xfs_vm_direct_IO
> | | | xfs_file_dio_aio_write
> | | | xfs_file_write_iter
> | | | aio_run_iocb
> | | | do_io_submit
> | | | sys_io_submit
> | | | entry_SYSCALL_64_fastpath
> | | | io_submit
> | | | 0x46d98a
> | | |
> | | --33.33%-- submit_bio_wait
> | | blkdev_issue_flush
> | | xfs_blkdev_issue_flush
> | | xfs_file_fsync
> | | vfs_fsync_range
> | | do_fsync
> | | sys_fdatasync
> | | entry_SYSCALL_64_fastpath
> | | |
> | | --100.00%-- 0x7f4ade4212ad
> | | syscall_work_queue::work_item_returning<syscall_result<int>, posix_file_impl::flush()::{lambda()#1}>::process
> | | 0x6030000c3ec0
> | |
> | |--33.66%-- flush_work
> | | xlog_cil_force_lsn
> | | |
> | | |--97.06%-- _xfs_log_force_lsn
> | | | |
> | | | |--78.79%-- xfs_file_fsync
> | | | | vfs_fsync_range
> | | | | do_fsync
> | | | | sys_fdatasync
> | | | | entry_SYSCALL_64_fastpath
> | | | | 0x7f4ade4212ad
> | | | | syscall_work_queue::work_item_returning<syscall_result<int>, posix_file_impl::flush()::{lambda()#1}>::process
> | | | | 0x6030000c3ec0
> | | | |
> | | | --21.21%-- xfs_dir_fsync
> | | | vfs_fsync_range
> | | | do_fsync
> | | | sys_fdatasync
> | | | entry_SYSCALL_64_fastpath
> | | | |
> | | | --100.00%-- 0x7f4ade4212ad
> | | | syscall_work_queue::work_item_returning<syscall_result<int>, posix_file_impl::flush()::{lambda()#1}>::process
> | | | 0x6030000c3ec0
> | | |
> | | --2.94%-- _xfs_log_force
> | | xfs_log_force
> | | xfs_buf_lock
> | | _xfs_buf_find
> | | xfs_buf_get_map
> | | xfs_trans_get_buf_map
> | | xfs_btree_get_bufl
> | | xfs_bmap_extents_to_btree
> | | xfs_bmap_add_extent_hole_real
> | | xfs_bmapi_write
> | | xfs_iomap_write_direct
> | | __xfs_get_blocks
> | | xfs_get_blocks_direct
> | | do_blockdev_direct_IO
> | | __blockdev_direct_IO
> | | xfs_vm_direct_IO
> | | xfs_file_dio_aio_write
> | | xfs_file_write_iter
> | | aio_run_iocb
> | | do_io_submit
> | | sys_io_submit
> | | entry_SYSCALL_64_fastpath
> | | io_submit
> | | 0x46d98a
> | |
> | |--13.86%-- lock_sock_nested
> | | |
> | | |--78.57%-- tcp_sendmsg
> | | | inet_sendmsg
> | | | sock_sendmsg
> | | | SYSC_sendto
> | | | sys_sendto
> | | | entry_SYSCALL_64_fastpath
> | | | __libc_send
> | | | _ZN12continuationIZN6futureIJmEE4thenIZN7reactor14write_all_partER17pollable_fd_statePKvmmEUlmE_S0_IJEEEET0_OT_EUlSC_E_JmEE3runEv
> | | | |
> | | | |--36.36%-- 0x7f4ad6bf8de0
> | | | |
> | | | |--9.09%-- 0x4
> | | | |
> | | | |--9.09%-- 0x7f4adadf8de0
> | | | |
> | | | |--9.09%-- 0x7f4ada1f8de0
> | | | |
> | | | |--9.09%-- 0x7f4ad89f8de0
> | | | |
> | | | |--9.09%-- 0x7f4ad83f8de0
> | | | |
> | | | |--9.09%-- 0x7f4ad4df8de0
> | | | |
> | | | --9.09%-- 0x7f4ad35f8de0
> | | |
> | | --21.43%-- tcp_recvmsg
> | | inet_recvmsg
> | | sock_recvmsg
> | | sock_read_iter
> | | __vfs_read
> | | vfs_read
> | | sys_read
> | | entry_SYSCALL_64_fastpath
> | | 0x7f4ade6f754d
> | | reactor::read_some
> | | |
> | | |--66.67%-- _ZN12continuationIZN6futureIJEE4thenIZZN7service13storage_proxy22send_to_live_endpointsEmENKUlRSt4pairIK13basic_sstringIcjLj15EESt6vectorIN3gms12inet_addressESaISB_EEEE_clESF_EUlvE_S1_EET0_OT_EUlSK_E_JEE3runEv
> | | | reactor::del_timer
> | | | 0x6160000e2040
> | | |
> | | --33.33%-- continuation<future<> future<>::then_wrapped<future<> future<>::finally<auto seastar::with_gate<transport::cql_server::connection::process()::{lambda()#2}::operator()() const::{lambda()#1}>(seastar::gate&, transport::cql_server::connection::process()::{lambda()#2}::operator()() const::{lambda()#1}&&)::{lambda()#1}>(seastar::gate&)::{lambda(future<>)#1}::operator()(future<>)::{lambda(seastar::gate)#1}, future<> >(seastar::gate&)::{lambda(seastar::gate&)#1}>::run
> | | reactor::del_timer
> | | 0x6030000e2040
> | |
> | |--3.96%-- generic_make_request_checks
> | | generic_make_request
> | | submit_bio
> | | do_blockdev_direct_IO
> | | __blockdev_direct_IO
> | | xfs_vm_direct_IO
> | | xfs_file_dio_aio_write
> | | xfs_file_write_iter
> | | aio_run_iocb
> | | do_io_submit
> | | sys_io_submit
> | | entry_SYSCALL_64_fastpath
> | | io_submit
> | | 0x46d98a
> | |
> | |--3.96%-- kmem_cache_alloc_node
> | | __alloc_skb
> | | sk_stream_alloc_skb
> | | tcp_sendmsg
> | | inet_sendmsg
> | | sock_sendmsg
> | | SYSC_sendto
> | | sys_sendto
> | | entry_SYSCALL_64_fastpath
> | | __libc_send
> | | _ZN12continuationIZN6futureIJmEE4thenIZN7reactor14write_all_partER17pollable_fd_statePKvmmEUlmE_S0_IJEEEET0_OT_EUlSC_E_JmEE3runEv
> | | |
> | | |--25.00%-- 0x7f4ad9bf8de0
> | | |
> | | |--25.00%-- 0x7f4ad7df8de0
> | | |
> | | |--25.00%-- 0x7f4ad77f8de0
> | | |
> | | --25.00%-- 0x7f4ad59f8de0
> | |
> | |--0.99%-- unmap_underlying_metadata
> | | do_blockdev_direct_IO
> | | __blockdev_direct_IO
> | | xfs_vm_direct_IO
> | | xfs_file_dio_aio_write
> | | xfs_file_write_iter
> | | aio_run_iocb
> | | do_io_submit
> | | sys_io_submit
> | | entry_SYSCALL_64_fastpath
> | | io_submit
> | | 0x46d98a
> | |
> | |--0.99%-- __kmalloc_node_track_caller
> | | __kmalloc_reserve.isra.32
> | | __alloc_skb
> | | sk_stream_alloc_skb
> | | tcp_sendmsg
> | | inet_sendmsg
> | | sock_sendmsg
> | | SYSC_sendto
> | | sys_sendto
> | | entry_SYSCALL_64_fastpath
> | | __libc_send
> | | _ZN12continuationIZN6futureIJmEE4thenIZN7reactor14write_all_partER17pollable_fd_statePKvmmEUlmE_S0_IJEEEET0_OT_EUlSC_E_JmEE3runEv
> | | 0x7f4ad6bf8de0
> | |
> | --0.99%-- task_work_run
> | do_notify_resume
> | int_signal
> | __libc_close
> | std::experimental::fundamentals_v1::bad_optional_access::~bad_optional_access
> |
> --0.98%-- __cond_resched_softirq
> release_sock
> tcp_sendmsg
> inet_sendmsg
> sock_sendmsg
> SYSC_sendto
> sys_sendto
> entry_SYSCALL_64_fastpath
> __libc_send
> _ZN12continuationIZN6futureIJmEE4thenIZN7reactor14write_all_partER17pollable_fd_statePKvmmEUlmE_S0_IJEEEET0_OT_EUlSC_E_JmEE3runEv
> 0x7f4ada1f8de0
>
>
>
> #
> # (For a higher level overview, try: perf report --sort comm,dso)
> #
> [164814.835933] CPU: 22 PID: 48042 Comm: scylla Tainted: G E 4.2.6-200.fc22.x86_64 #1
> [164814.835936] Hardware name: Xen HVM domU, BIOS 4.2.amazon 05/06/2015
> [164814.835937] 0000000000000000 00000000a8713b7a ffff8802fb977ab8 ffffffff817729ea
> [164814.835941] 0000000000000000 ffff88076a69f780 ffff8802fb977ad8 ffffffffa03217a6
> [164814.835946] ffff88077119bcb0 0000000000000000 ffff8802fb977b08 ffffffffa034e749
> [164814.835951] Call Trace:
> [164814.835954] [<ffffffff817729ea>] dump_stack+0x45/0x57
> [164814.835971] [<ffffffffa03217a6>] xfs_buf_stale+0x26/0x80 [xfs]
> [164814.835989] [<ffffffffa034e749>] xfs_trans_binval+0x79/0x100 [xfs]
> [164814.836001] [<ffffffffa02f479b>] xfs_bmap_btree_to_extents+0x12b/0x1a0 [xfs]
> [164814.836012] [<ffffffffa02f8977>] xfs_bunmapi+0x967/0x9f0 [xfs]
> [164814.836027] [<ffffffffa0334b9e>] xfs_itruncate_extents+0x10e/0x220 [xfs]
> [164814.836044] [<ffffffffa033f75a>] ? kmem_zone_alloc+0x5a/0xe0 [xfs]
> [164814.836084] [<ffffffffa0334d49>] xfs_inactive_truncate+0x99/0x110 [xfs]
> [164814.836120] [<ffffffffa0335aa2>] xfs_inactive+0x102/0x120 [xfs]
> [164814.836135] [<ffffffffa033a6cf>] xfs_fs_evict_inode+0x6f/0xa0 [xfs]
> [164814.836138] [<ffffffff81238d76>] evict+0xa6/0x170
> [164814.836140] [<ffffffff81239026>] iput+0x196/0x220
> [164814.836147] [<ffffffff81234fe4>] __dentry_kill+0x174/0x1c0
> [164814.836150] [<ffffffff8123514b>] dput+0x11b/0x200
> [164814.836155] [<ffffffff8121fe02>] __fput+0x172/0x1e0
> [164814.836158] [<ffffffff8121febe>] ____fput+0xe/0x10
> [164814.836161] [<ffffffff810bab75>] task_work_run+0x85/0xb0
> [164814.836164] [<ffffffff81014a4d>] do_notify_resume+0x8d/0x90
> [164814.836167] [<ffffffff817795bc>] int_signal+0x12/0x17
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-11-30 14:10 ` Brian Foster
@ 2015-11-30 14:29 ` Avi Kivity
2015-11-30 16:14 ` Brian Foster
2015-11-30 15:49 ` Glauber Costa
1 sibling, 1 reply; 58+ messages in thread
From: Avi Kivity @ 2015-11-30 14:29 UTC (permalink / raw)
To: Brian Foster, Glauber Costa; +Cc: xfs
On 11/30/2015 04:10 PM, Brian Foster wrote:
>> 2) xfs_buf_lock -> down
>> This is one I truly don't understand. What can be causing contention
>> in this lock? We never have two different cores writing to the same
>> buffer, nor should we have the same core doingCAP_FOWNER so.
>>
> This is not one single lock. An XFS buffer is the data structure used to
> modify/log/read-write metadata on-disk and each buffer has its own lock
> to prevent corruption. Buffer lock contention is possible because the
> filesystem has bits of "global" metadata that has to be updated via
> buffers.
>
> For example, usually one has multiple allocation groups to maximize
> parallelism, but we still have per-ag metadata that has to be tracked
> globally with respect to each AG (e.g., free space trees, inode
> allocation trees, etc.). Any operation that affects this metadata (e.g.,
> block/inode allocation) has to lock the agi/agf buffers along with any
> buffers associated with the modified btree leaf/node blocks, etc.
>
> One example in your attached perf traces has several threads looking to
> acquire the AGF, which is a per-AG data structure for tracking free
> space in the AG. One thread looks like the inode eviction case noted
> above (freeing blocks), another looks like a file truncate (also freeing
> blocks), and yet another is a block allocation due to a direct I/O
> write. Were any of these operations directed to an inode in a separate
> AG, they would be able to proceed in parallel (but I believe they would
> still hit the same codepaths as far as perf can tell).
I guess we can mitigate (but not eliminate) this by creating more
allocation groups. What is the default value for agsize? Are there any
downsides to decreasing it, besides consuming more memory?
Are those locks held around I/O, or just CPU operations, or a mix?
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-11-30 14:10 ` Brian Foster
2015-11-30 14:29 ` Avi Kivity
@ 2015-11-30 15:49 ` Glauber Costa
2015-12-01 13:11 ` Brian Foster
1 sibling, 1 reply; 58+ messages in thread
From: Glauber Costa @ 2015-11-30 15:49 UTC (permalink / raw)
To: Brian Foster; +Cc: Avi Kivity, xfs
Hi Brian
>> 1) xfs_buf_lock -> xfs_log_force.
>>
>> I've started wondering what would make xfs_log_force sleep. But then I
>> have noticed that xfs_log_force will only be called when a buffer is
>> marked stale. Most of the times a buffer is marked stale seems to be
>> due to errors. Although that is not my case (more on that), it got me
>> thinking that maybe the right thing to do would be to avoid hitting
>> this case altogether?
>>
>
> I'm not following where you get the "only if marked stale" part..? It
> certainly looks like that's one potential purpose for the call, but this
> is called in a variety of other places as well. E.g., forcing the log
> via pushing on the ail when it has pinned items is another case. The ail
> push itself can originate from transaction reservation, etc., when log
> space is needed. In other words, I'm not sure this is something that's
> easily controlled from userspace, if at all. Rather, it's a significant
> part of the wider state machine the fs uses to manage logging.
I understand that in general xfs_log_force can be called from many
places. But in our traces the ones we see sleeping are coming from
xfs_buf_lock. The code for xfs_buf_lock reads:
if (atomic_read(&bp->b_pin_count) && (bp->b_flags & XBF_STALE))
xfs_log_force(bp->b_target->bt_mount, 0);
which if I read correctly, will be called only for stale buffers. True
thing they happen to be pinned as well, but somehow the stale part
caught my attention. It seemed to me from briefly looking that the
stale condition was a more "avoidable" one. (keep in mind I am not an
awesome XFSer, may be missing something)
>
>> The file example-stale.txt contains a backtrace of the case where we
>> are being marked as stale. It seems to be happening when we convert
>> the the inode's extents from unwritten to real. Can this case be
>> avoided? I won't pretend I know the intricacies of this, but couldn't
>> we be keeping extents from the very beginning to avoid creating stale
>> buffers?
>>
>
> This is down in xfs_fs_evict_inode()->xfs_inactive(), which is generally
> when an inode is evicted from cache. In this case, it looks like the
> inode is unlinked (permanently removed), the extents are being removed
> and a bmap btree block is being invalidated as part of that overall
> process. I don't think this has anything to do with unwritten extents.
>
Cool. If the inode is indeed unliked, could that sill be triggering
that condition in xfs_buf_lock? I am not even close to fully
understanding how XFS manages and/or recycles buffers, but it seems to
me that if an inode is going away, there isn't really any reason to
contend for its buffers.
>> 2) xfs_buf_lock -> down
>> This is one I truly don't understand. What can be causing contention
>> in this lock? We never have two different cores writing to the same
>> buffer, nor should we have the same core doingCAP_FOWNER so.
>>
>
> This is not one single lock. An XFS buffer is the data structure used to
> modify/log/read-write metadata on-disk and each buffer has its own lock
> to prevent corruption. Buffer lock contention is possible because the
> filesystem has bits of "global" metadata that has to be updated via
> buffers.
I see. Since I hate guessing, is there any way you would recommend for
us to probe the system to determine if this contention scenario is
indeed the one we are seeing?
We usually open a file, write to it from a single core only,
sequentially, direct IO only, as well behavedly as we can, with all
the effort in the world to be good kids to the extent Santa will bring
us presents without us even asking.
So we were very puzzled to see contention. Contention for global
metadata updates is the best explanation we've had so far, and would
be great if we could verify it is indeed the case.
>
> For example, usually one has multiple allocation groups to maximize
> parallelism, but we still have per-ag metadata that has to be tracked
> globally with respect to each AG (e.g., free space trees, inode
> allocation trees, etc.). Any operation that affects this metadata (e.g.,
> block/inode allocation) has to lock the agi/agf buffers along with any
> buffers associated with the modified btree leaf/node blocks, etc.
>
> One example in your attached perf traces has several threads looking to
> acquire the AGF, which is a per-AG data structure for tracking free
> space in the AG. One thread looks like the inode eviction case noted
> above (freeing blocks), another looks like a file truncate (also freeing
> blocks), and yet another is a block allocation due to a direct I/O
> write. Were any of these operations directed to an inode in a separate
> AG, they would be able to proceed in parallel (but I believe they would
> still hit the same codepaths as far as perf can tell).
This is great, great, awesome info Brian. Thanks. We are so far
allocating inodes and truncating them when we need a new one, but
maybe there is some allocation pattern that is friendlier to the AG? I
understand that with such a data structure it may very well be
impossible to get rid of all waiting, but we will certainly do all we
can to mitigate it.
>
>> 3) xfs_file_aio_write_checks -> file_update_time -> xfs_vn_update_time
>>
>> You guys seem to have an interface to avoid that, by setting the
>> FMODE_NOCMTIME flag. This is done by issuing the open by handle ioctl,
>> which will set this flag for all regular files. That's great, but that
>> ioctl required CAP_SYS_ADMIN, which is a big no for us, since we run
>> our server as an unprivileged user. I don't understand, however, why
>> such an strict check is needed. If we have full rights on the
>> filesystem, why can't we issue this operation? In my view, CAP_FOWNER
>> should already be enough.I do understand the handles have to be stable
>> and a file can have its ownership changed, in which case the previous
>> owner would keep the handle valid. Is that the reason you went with
>> the most restrictive capability ?
>
> I'm not familiar enough with the open-by-handle stuff to comment on the
> permission constraints. Perhaps Dave or others can comment further on
> this bit...
>
> Brian
Thanks again Brian. The pointer to the AG stuff was really helpful.
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-11-30 14:29 ` Avi Kivity
@ 2015-11-30 16:14 ` Brian Foster
2015-12-01 9:08 ` Avi Kivity
0 siblings, 1 reply; 58+ messages in thread
From: Brian Foster @ 2015-11-30 16:14 UTC (permalink / raw)
To: Avi Kivity; +Cc: Glauber Costa, xfs
On Mon, Nov 30, 2015 at 04:29:13PM +0200, Avi Kivity wrote:
>
>
> On 11/30/2015 04:10 PM, Brian Foster wrote:
> >>2) xfs_buf_lock -> down
> >>This is one I truly don't understand. What can be causing contention
> >>in this lock? We never have two different cores writing to the same
> >>buffer, nor should we have the same core doingCAP_FOWNER so.
> >>
> >This is not one single lock. An XFS buffer is the data structure used to
> >modify/log/read-write metadata on-disk and each buffer has its own lock
> >to prevent corruption. Buffer lock contention is possible because the
> >filesystem has bits of "global" metadata that has to be updated via
> >buffers.
> >
> >For example, usually one has multiple allocation groups to maximize
> >parallelism, but we still have per-ag metadata that has to be tracked
> >globally with respect to each AG (e.g., free space trees, inode
> >allocation trees, etc.). Any operation that affects this metadata (e.g.,
> >block/inode allocation) has to lock the agi/agf buffers along with any
> >buffers associated with the modified btree leaf/node blocks, etc.
> >
> >One example in your attached perf traces has several threads looking to
> >acquire the AGF, which is a per-AG data structure for tracking free
> >space in the AG. One thread looks like the inode eviction case noted
> >above (freeing blocks), another looks like a file truncate (also freeing
> >blocks), and yet another is a block allocation due to a direct I/O
> >write. Were any of these operations directed to an inode in a separate
> >AG, they would be able to proceed in parallel (but I believe they would
> >still hit the same codepaths as far as perf can tell).
>
> I guess we can mitigate (but not eliminate) this by creating more allocation
> groups. What is the default value for agsize? Are there any downsides to
> decreasing it, besides consuming more memory?
>
I suppose so, but I would be careful to check that you actually see
contention and test that increasing agcount actually helps. As
mentioned, I'm not sure off hand if the perf trace alone would look any
different if you have multiple metadata operations in progress on
separate AGs.
My understanding is that there are diminishing returns to high AG counts
and usually 32-64 is sufficient for most storage. Dave might be able to
elaborate more on that... (I think this would make a good FAQ entry,
actually).
The agsize/agcount mkfs-time heuristics change depending on the type of
storage. A single AG can be up to 1TB and if the fs is not considered
"multidisk" (e.g., no stripe unit/width is defined), 4 AGs is the
default up to 4TB. If a stripe unit is set, the agsize/agcount is
adjusted depending on the size of the overall volume (see
xfsprogs-dev/mkfs/xfs_mkfs.c:calc_default_ag_geometry() for details).
> Are those locks held around I/O, or just CPU operations, or a mix?
I believe it's a mix of modifications and I/O, though it looks like some
of the I/O cases don't necessarily wait on the lock. E.g., the AIL
pushing case will trylock and defer to the next list iteration if the
buffer is busy.
Brian
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-11-28 2:43 sleeps and waits during io_submit Glauber Costa
2015-11-30 14:10 ` Brian Foster
@ 2015-11-30 23:10 ` Dave Chinner
2015-11-30 23:51 ` Glauber Costa
1 sibling, 1 reply; 58+ messages in thread
From: Dave Chinner @ 2015-11-30 23:10 UTC (permalink / raw)
To: Glauber Costa; +Cc: Avi Kivity, xfs
On Fri, Nov 27, 2015 at 09:43:50PM -0500, Glauber Costa wrote:
> Hello my dear XFSers,
>
> For those of you who don't know, we at ScyllaDB produce a modern NoSQL
> data store that, at the moment, runs on top of XFS only. We deal
> exclusively with asynchronous and direct IO, due to our
> thread-per-core architecture. Due to that, we avoid issuing any
> operation that will sleep.
>
> While debugging an extreme case of bad performance (most likely
> related to a not-so-great disk), I have found a variety of cases in
> which XFS blocks. To find those, I have used perf record -e
> sched:sched_switch -p <pid_of_db>, and I am attaching the perf report
> as xfs-sched_switch.log. Please note that this doesn't tell me for how
> long we block, but as mentioned before, blocking operations outside
> our control are detrimental to us regardless of the elapsed time.
>
> For those who are not acquainted to our internals, please ignore
> everything in that file but the xfs functions. For the xfs symbols,
> there are two kinds of events: the ones that are a children of
> io_submit, where we don't tolerate blocking, and the ones that are
> children of our helper IO thread, to where we push big operations that
> we know will block until we can get rid of them all. We care about the
> former and ignore the latter.
>
> Please allow me to ask you a couple of questions about those findings.
> If we are doing anything wrong, advise on best practices is truly
> welcome.
>
> 1) xfs_buf_lock -> xfs_log_force.
>
> I've started wondering what would make xfs_log_force sleep. But then I
> have noticed that xfs_log_force will only be called when a buffer is
> marked stale. Most of the times a buffer is marked stale seems to be
> due to errors. Although that is not my case (more on that), it got me
> thinking that maybe the right thing to do would be to avoid hitting
> this case altogether?
The buffer is stale because it has recently been freed, and we
cannot re-use it until the transaction that freed it has been
committed to the journal. e.g. this trace:
--3.15%-- _xfs_log_force
xfs_log_force
xfs_buf_lock
_xfs_buf_find
xfs_buf_get_map
xfs_trans_get_buf_map
xfs_btree_get_bufl
xfs_bmap_extents_to_btree
xfs_bmap_add_extent_hole_real
xfs_bmapi_write
xfs_iomap_write_direct
__xfs_get_blocks
xfs_get_blocks_direct
do_blockdev_direct_IO
__blockdev_direct_IO
xfs_vm_direct_IO
xfs_file_dio_aio_write
xfs_file_write_iter
aio_run_iocb
do_io_submit
sys_io_submit
entry_SYSCALL_64_fastpath
io_submit
0x46d98a
implies something like this has happened:
truncate
free extent
extent list now fits inline in inode
btree-to-extent format change
free last bmbt block X
mark block contents stale
add block to busy extent list
place block on AGFL
AIO write
allocate extent
inline extents full
extent-to-btree conversion
allocate bmbt block
grab block X from free list
get locked buffer for block X
xfs_buf_lock
buffer stale
log force to commit previous free transaction to disk
.....
log force completes
buffer removed from busy extent list
buffer no longer stale
add bmbt record to new block
update btree indexes
....
And, looking at the trace you attached, we see this:
dump_stack
xfs_buf_stale
xfs_trans_binval
xfs_bmap_btree_to_extents <<<<<<<<<
xfs_bunmapi
xfs_itruncate_extents
So I'd say this is a pretty clear indication that we're immediately
recycling freed blocks from the free list here....
> The file example-stale.txt contains a backtrace of the case where we
> are being marked as stale. It seems to be happening when we convert
> the the inode's extents from unwritten to real. Can this case be
> avoided? I won't pretend I know the intricacies of this, but couldn't
> we be keeping extents from the very beginning to avoid creating stale
> buffers?
>
> 2) xfs_buf_lock -> down
> This is one I truly don't understand. What can be causing contention
> in this lock? We never have two different cores writing to the same
> buffer, nor should we have the same core doingCAP_FOWNER so.
As Brian pointed out, this is probably AGF or AGI contention -
attempting to allocate/free extents or inodes in the same AG at the
same time will show this sort of pattern. This trace shows AGF
contention:
down
xfs_buf_lock
_xfs_buf_find
xfs_buf_get_map
xfs_buf_read_map
xfs_trans_read_buf_map
xfs_read_agf
.....
This trace shows AGI contention:
down
xfs_buf_lock
_xfs_buf_find
xfs_buf_get_map
xfs_buf_read_map
xfs_trans_read_buf_map
xfs_read_agi
....
> 3) xfs_file_aio_write_checks -> file_update_time -> xfs_vn_update_time
This trace?:
rwsem_down_write_failed
call_rwsem_down_write_failed
xfs_ilock
xfs_vn_update_time
file_update_time
xfs_file_aio_write_checks
xfs_file_dio_aio_write
xfs_file_write_iter
aio_run_iocb
do_io_submit
sys_io_submit
entry_SYSCALL_64_fastpath
io_submit
0x46d98a
Which is an mtime timestamp update racing with another operation
that takes the internal metadata lock (e.g. block mapping/allocation
for that inode).
> You guys seem to have an interface to avoid that, by setting the
> FMODE_NOCMTIME flag.
We've talked about exposing this through open() for Ceph.
http://www.kernelhub.org/?p=2&msg=744325
https://lkml.org/lkml/2015/5/15/671
Read the first thread for why it's problematic to expose this to
userspace - I won't repeat it all here.
As it is, there was some code recently hacked into ext4 to reduce
mtime overhead - the MS_LAZYTIME superblock option. What it does it
prevent the inode from being marked dirty when timestamps are
updated and hence the timestamps are never journalled or written
until something else marks the inode metadata dirty (e.g. block
allocation). ext4 gets away with this because it doesn't actually
journal timestamp changes - they get captured in the journal by
other modifications that are journalled, but still rely on the inod
being marked dirty for fsync, writeback and inode cache eviction
doing the right thing.
The ext4 implementation looks like the timestamp updates can be
thrown away, as the inodes are not marked dirty and so on memory
pressure they will simply be reclaimed without writing back the
updated timestamps that are held in memory. I suspect fsync will
also have problems on ext4 as the inode is not metadata dirty or
journalled, and hence the timestamp changes will never get written
back.
And, IMO, the worst part of the ext4 implementation is that the
inode buffer writeback code in ext4 now checks to see if any of the
other inodes in the buffer being written back need to have their
inode timestamps updated. IOWs, ext4 now does writeback of
/unjournalled metadata/ to inodes that are purely timestamp dirty.
We /used/ to do shit like this in XFS. We got rid of it in
preference of journalling everything because the corner cases in log
recovery meant that after a crash the inodes were in inconsistent
states, and that meant we had unexpected, unpredictable recovery
behaviour where files weren't the expected size and/or didn't
contain the expected data. Hence going back to the bad old days of
hacking around the journal "for speed" doesn't exactly fill me with
joy.
Let me have a think about how we can implement lazytime in a sane
way, such that fsync() works correctly, we don't throw away
timstamp changes in memory reclaim and we don't write unlogged
changes to the on-disk locations....
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-11-30 23:10 ` Dave Chinner
@ 2015-11-30 23:51 ` Glauber Costa
2015-12-01 20:30 ` Dave Chinner
0 siblings, 1 reply; 58+ messages in thread
From: Glauber Costa @ 2015-11-30 23:51 UTC (permalink / raw)
To: Dave Chinner; +Cc: Avi Kivity, xfs
Hi Dave
On Mon, Nov 30, 2015 at 6:10 PM, Dave Chinner <david@fromorbit.com> wrote:
> On Fri, Nov 27, 2015 at 09:43:50PM -0500, Glauber Costa wrote:
>> Hello my dear XFSers,
>>
>> For those of you who don't know, we at ScyllaDB produce a modern NoSQL
>> data store that, at the moment, runs on top of XFS only. We deal
>> exclusively with asynchronous and direct IO, due to our
>> thread-per-core architecture. Due to that, we avoid issuing any
>> operation that will sleep.
>>
>> While debugging an extreme case of bad performance (most likely
>> related to a not-so-great disk), I have found a variety of cases in
>> which XFS blocks. To find those, I have used perf record -e
>> sched:sched_switch -p <pid_of_db>, and I am attaching the perf report
>> as xfs-sched_switch.log. Please note that this doesn't tell me for how
>> long we block, but as mentioned before, blocking operations outside
>> our control are detrimental to us regardless of the elapsed time.
>>
>> For those who are not acquainted to our internals, please ignore
>> everything in that file but the xfs functions. For the xfs symbols,
>> there are two kinds of events: the ones that are a children of
>> io_submit, where we don't tolerate blocking, and the ones that are
>> children of our helper IO thread, to where we push big operations that
>> we know will block until we can get rid of them all. We care about the
>> former and ignore the latter.
>>
>> Please allow me to ask you a couple of questions about those findings.
>> If we are doing anything wrong, advise on best practices is truly
>> welcome.
>>
>> 1) xfs_buf_lock -> xfs_log_force.
>>
>> I've started wondering what would make xfs_log_force sleep. But then I
>> have noticed that xfs_log_force will only be called when a buffer is
>> marked stale. Most of the times a buffer is marked stale seems to be
>> due to errors. Although that is not my case (more on that), it got me
>> thinking that maybe the right thing to do would be to avoid hitting
>> this case altogether?
>
> The buffer is stale because it has recently been freed, and we
> cannot re-use it until the transaction that freed it has been
> committed to the journal. e.g. this trace:
>
> --3.15%-- _xfs_log_force
> xfs_log_force
> xfs_buf_lock
> _xfs_buf_find
> xfs_buf_get_map
> xfs_trans_get_buf_map
> xfs_btree_get_bufl
> xfs_bmap_extents_to_btree
> xfs_bmap_add_extent_hole_real
> xfs_bmapi_write
> xfs_iomap_write_direct
> __xfs_get_blocks
> xfs_get_blocks_direct
> do_blockdev_direct_IO
> __blockdev_direct_IO
> xfs_vm_direct_IO
> xfs_file_dio_aio_write
> xfs_file_write_iter
> aio_run_iocb
> do_io_submit
> sys_io_submit
> entry_SYSCALL_64_fastpath
> io_submit
> 0x46d98a
>
> implies something like this has happened:
>
> truncate
> free extent
> extent list now fits inline in inode
> btree-to-extent format change
> free last bmbt block X
> mark block contents stale
> add block to busy extent list
> place block on AGFL
>
> AIO write
> allocate extent
> inline extents full
> extent-to-btree conversion
> allocate bmbt block
> grab block X from free list
> get locked buffer for block X
> xfs_buf_lock
> buffer stale
> log force to commit previous free transaction to disk
> .....
> log force completes
> buffer removed from busy extent list
> buffer no longer stale
> add bmbt record to new block
> update btree indexes
> ....
>
>
> And, looking at the trace you attached, we see this:
>
> dump_stack
> xfs_buf_stale
> xfs_trans_binval
> xfs_bmap_btree_to_extents <<<<<<<<<
> xfs_bunmapi
> xfs_itruncate_extents
>
> So I'd say this is a pretty clear indication that we're immediately
> recycling freed blocks from the free list here....
>
>> The file example-stale.txt contains a backtrace of the case where we
>> are being marked as stale. It seems to be happening when we convert
>> the the inode's extents from unwritten to real. Can this case be
>> avoided? I won't pretend I know the intricacies of this, but couldn't
>> we be keeping extents from the very beginning to avoid creating stale
>> buffers?
>>
>> 2) xfs_buf_lock -> down
>> This is one I truly don't understand. What can be causing contention
>> in this lock? We never have two different cores writing to the same
>> buffer, nor should we have the same core doingCAP_FOWNER so.
>
> As Brian pointed out, this is probably AGF or AGI contention -
> attempting to allocate/free extents or inodes in the same AG at the
> same time will show this sort of pattern. This trace shows AGF
> contention:
>
> down
> xfs_buf_lock
> _xfs_buf_find
> xfs_buf_get_map
> xfs_buf_read_map
> xfs_trans_read_buf_map
> xfs_read_agf
> .....
>
> This trace shows AGI contention:
>
> down
> xfs_buf_lock
> _xfs_buf_find
> xfs_buf_get_map
> xfs_buf_read_map
> xfs_trans_read_buf_map
> xfs_read_agi
> ....
>
Great. I will take a look at how can we mitigate those on our side. I
will need some time to understand all of that better, so for now I'd
just leave you guys with a big thank you.
>> 3) xfs_file_aio_write_checks -> file_update_time -> xfs_vn_update_time
>
> This trace?:
>
> rwsem_down_write_failed
> call_rwsem_down_write_failed
> xfs_ilock
> xfs_vn_update_time
> file_update_time
> xfs_file_aio_write_checks
> xfs_file_dio_aio_write
> xfs_file_write_iter
> aio_run_iocb
> do_io_submit
> sys_io_submit
> entry_SYSCALL_64_fastpath
> io_submit
> 0x46d98a
>
> Which is an mtime timestamp update racing with another operation
> that takes the internal metadata lock (e.g. block mapping/allocation
> for that inode).
>
>> You guys seem to have an interface to avoid that, by setting the
>> FMODE_NOCMTIME flag.
>
> We've talked about exposing this through open() for Ceph.
>
> http://www.kernelhub.org/?p=2&msg=744325
> https://lkml.org/lkml/2015/5/15/671
>
> Read the first thread for why it's problematic to expose this to
> userspace - I won't repeat it all here.
>
> As it is, there was some code recently hacked into ext4 to reduce
> mtime overhead - the MS_LAZYTIME superblock option. What it does it
> prevent the inode from being marked dirty when timestamps are
> updated and hence the timestamps are never journalled or written
> until something else marks the inode metadata dirty (e.g. block
> allocation). ext4 gets away with this because it doesn't actually
> journal timestamp changes - they get captured in the journal by
> other modifications that are journalled, but still rely on the inod
> being marked dirty for fsync, writeback and inode cache eviction
> doing the right thing.
>
> The ext4 implementation looks like the timestamp updates can be
> thrown away, as the inodes are not marked dirty and so on memory
> pressure they will simply be reclaimed without writing back the
> updated timestamps that are held in memory. I suspect fsync will
> also have problems on ext4 as the inode is not metadata dirty or
> journalled, and hence the timestamp changes will never get written
> back.
>
> And, IMO, the worst part of the ext4 implementation is that the
> inode buffer writeback code in ext4 now checks to see if any of the
> other inodes in the buffer being written back need to have their
> inode timestamps updated. IOWs, ext4 now does writeback of
> /unjournalled metadata/ to inodes that are purely timestamp dirty.
>
> We /used/ to do shit like this in XFS. We got rid of it in
> preference of journalling everything because the corner cases in log
> recovery meant that after a crash the inodes were in inconsistent
> states, and that meant we had unexpected, unpredictable recovery
> behaviour where files weren't the expected size and/or didn't
> contain the expected data. Hence going back to the bad old days of
> hacking around the journal "for speed" doesn't exactly fill me with
> joy.
>
> Let me have a think about how we can implement lazytime in a sane
> way, such that fsync() works correctly, we don't throw away
> timstamp changes in memory reclaim and we don't write unlogged
> changes to the on-disk locations....
I trust you fully for matters related to speed.
Keep in mind, though, that at least for us the fact that it blocks is
a lot worse than the fact that it is slow. We can work around slow,
but blocking basically means that we won't have any more work to push
- since we don't do threading. The processor that stales just sits
idle until the lock is released. So any non-blocking solution to this
would already be a win for us.
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-11-30 16:14 ` Brian Foster
@ 2015-12-01 9:08 ` Avi Kivity
2015-12-01 13:11 ` Brian Foster
0 siblings, 1 reply; 58+ messages in thread
From: Avi Kivity @ 2015-12-01 9:08 UTC (permalink / raw)
To: Brian Foster; +Cc: Glauber Costa, xfs
On 11/30/2015 06:14 PM, Brian Foster wrote:
> On Mon, Nov 30, 2015 at 04:29:13PM +0200, Avi Kivity wrote:
>>
>> On 11/30/2015 04:10 PM, Brian Foster wrote:
>>>> 2) xfs_buf_lock -> down
>>>> This is one I truly don't understand. What can be causing contention
>>>> in this lock? We never have two different cores writing to the same
>>>> buffer, nor should we have the same core doingCAP_FOWNER so.
>>>>
>>> This is not one single lock. An XFS buffer is the data structure used to
>>> modify/log/read-write metadata on-disk and each buffer has its own lock
>>> to prevent corruption. Buffer lock contention is possible because the
>>> filesystem has bits of "global" metadata that has to be updated via
>>> buffers.
>>>
>>> For example, usually one has multiple allocation groups to maximize
>>> parallelism, but we still have per-ag metadata that has to be tracked
>>> globally with respect to each AG (e.g., free space trees, inode
>>> allocation trees, etc.). Any operation that affects this metadata (e.g.,
>>> block/inode allocation) has to lock the agi/agf buffers along with any
>>> buffers associated with the modified btree leaf/node blocks, etc.
>>>
>>> One example in your attached perf traces has several threads looking to
>>> acquire the AGF, which is a per-AG data structure for tracking free
>>> space in the AG. One thread looks like the inode eviction case noted
>>> above (freeing blocks), another looks like a file truncate (also freeing
>>> blocks), and yet another is a block allocation due to a direct I/O
>>> write. Were any of these operations directed to an inode in a separate
>>> AG, they would be able to proceed in parallel (but I believe they would
>>> still hit the same codepaths as far as perf can tell).
>> I guess we can mitigate (but not eliminate) this by creating more allocation
>> groups. What is the default value for agsize? Are there any downsides to
>> decreasing it, besides consuming more memory?
>>
> I suppose so, but I would be careful to check that you actually see
> contention and test that increasing agcount actually helps. As
> mentioned, I'm not sure off hand if the perf trace alone would look any
> different if you have multiple metadata operations in progress on
> separate AGs.
>
> My understanding is that there are diminishing returns to high AG counts
> and usually 32-64 is sufficient for most storage. Dave might be able to
> elaborate more on that... (I think this would make a good FAQ entry,
> actually).
>
> The agsize/agcount mkfs-time heuristics change depending on the type of
> storage. A single AG can be up to 1TB and if the fs is not considered
> "multidisk" (e.g., no stripe unit/width is defined), 4 AGs is the
> default up to 4TB. If a stripe unit is set, the agsize/agcount is
> adjusted depending on the size of the overall volume (see
> xfsprogs-dev/mkfs/xfs_mkfs.c:calc_default_ag_geometry() for details).
We'll experiment with this. Surely it depends on more than the amount
of storage? If you have a high op rate you'll be more likely to excite
contention, no?
>
>> Are those locks held around I/O, or just CPU operations, or a mix?
> I believe it's a mix of modifications and I/O, though it looks like some
> of the I/O cases don't necessarily wait on the lock. E.g., the AIL
> pushing case will trylock and defer to the next list iteration if the
> buffer is busy.
>
Ok. For us sleeping in io_submit() is death because we have no other
thread on that core to take its place.
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-01 9:08 ` Avi Kivity
@ 2015-12-01 13:11 ` Brian Foster
2015-12-01 13:58 ` Avi Kivity
0 siblings, 1 reply; 58+ messages in thread
From: Brian Foster @ 2015-12-01 13:11 UTC (permalink / raw)
To: Avi Kivity; +Cc: Glauber Costa, xfs
On Tue, Dec 01, 2015 at 11:08:47AM +0200, Avi Kivity wrote:
> On 11/30/2015 06:14 PM, Brian Foster wrote:
> >On Mon, Nov 30, 2015 at 04:29:13PM +0200, Avi Kivity wrote:
> >>
> >>On 11/30/2015 04:10 PM, Brian Foster wrote:
...
> >The agsize/agcount mkfs-time heuristics change depending on the type of
> >storage. A single AG can be up to 1TB and if the fs is not considered
> >"multidisk" (e.g., no stripe unit/width is defined), 4 AGs is the
> >default up to 4TB. If a stripe unit is set, the agsize/agcount is
> >adjusted depending on the size of the overall volume (see
> >xfsprogs-dev/mkfs/xfs_mkfs.c:calc_default_ag_geometry() for details).
>
> We'll experiment with this. Surely it depends on more than the amount of
> storage? If you have a high op rate you'll be more likely to excite
> contention, no?
>
Sure. The absolute optimal configuration for your workload probably
depends on more than storage size, but mkfs doesn't have that
information. In general, it tries to use the most reasonable
configuration based on the storage and expected workload. If you want to
tweak it beyond that, indeed, the best bet is to experiment with what
works.
> >
> >>Are those locks held around I/O, or just CPU operations, or a mix?
> >I believe it's a mix of modifications and I/O, though it looks like some
> >of the I/O cases don't necessarily wait on the lock. E.g., the AIL
> >pushing case will trylock and defer to the next list iteration if the
> >buffer is busy.
> >
>
> Ok. For us sleeping in io_submit() is death because we have no other thread
> on that core to take its place.
>
The above is with regard to metadata I/O, whereas io_submit() is
obviously for user I/O. io_submit() can probably block in a variety of
places afaict... it might have to read in the inode extent map, allocate
blocks, take inode/ag locks, reserve log space for transactions, etc.
It sounds to me that first and foremost you want to make sure you don't
have however many parallel operations you typically have running
contending on the same inodes or AGs. Hint: creating files under
separate subdirectories is a quick and easy way to allocate inodes under
separate AGs (the agno is encoded into the upper bits of the inode
number). Reducing the frequency of block allocation/frees might also be
another help (e.g., preallocate and reuse files, 'mount -o ikeep,'
etc.). Beyond that, you probably want to make sure the log is large
enough to support all concurrent operations. See the xfs_log_grant_*
tracepoints for a window into if/how long transaction reservations might
be waiting on the log.
Brian
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-11-30 15:49 ` Glauber Costa
@ 2015-12-01 13:11 ` Brian Foster
2015-12-01 13:39 ` Glauber Costa
0 siblings, 1 reply; 58+ messages in thread
From: Brian Foster @ 2015-12-01 13:11 UTC (permalink / raw)
To: Glauber Costa; +Cc: Avi Kivity, xfs
On Mon, Nov 30, 2015 at 10:49:27AM -0500, Glauber Costa wrote:
> Hi Brian
>
> >> 1) xfs_buf_lock -> xfs_log_force.
> >>
> >> I've started wondering what would make xfs_log_force sleep. But then I
> >> have noticed that xfs_log_force will only be called when a buffer is
> >> marked stale. Most of the times a buffer is marked stale seems to be
> >> due to errors. Although that is not my case (more on that), it got me
> >> thinking that maybe the right thing to do would be to avoid hitting
> >> this case altogether?
> >>
> >
> > I'm not following where you get the "only if marked stale" part..? It
> > certainly looks like that's one potential purpose for the call, but this
> > is called in a variety of other places as well. E.g., forcing the log
> > via pushing on the ail when it has pinned items is another case. The ail
> > push itself can originate from transaction reservation, etc., when log
> > space is needed. In other words, I'm not sure this is something that's
> > easily controlled from userspace, if at all. Rather, it's a significant
> > part of the wider state machine the fs uses to manage logging.
>
> I understand that in general xfs_log_force can be called from many
> places. But in our traces the ones we see sleeping are coming from
> xfs_buf_lock. The code for xfs_buf_lock reads:
>
> if (atomic_read(&bp->b_pin_count) && (bp->b_flags & XBF_STALE))
> xfs_log_force(bp->b_target->bt_mount, 0);
>
>
> which if I read correctly, will be called only for stale buffers. True
> thing they happen to be pinned as well, but somehow the stale part
> caught my attention. It seemed to me from briefly looking that the
> stale condition was a more "avoidable" one. (keep in mind I am not an
> awesome XFSer, may be missing something)
>
It's not really avoidable. It's an expected buffer state when metadata
blocks/buffers are freed as actions must be taken if they are reused.
Dave's breakdown describes how you might be hitting this based on your
traces.
> >
> >> The file example-stale.txt contains a backtrace of the case where we
> >> are being marked as stale. It seems to be happening when we convert
> >> the the inode's extents from unwritten to real. Can this case be
> >> avoided? I won't pretend I know the intricacies of this, but couldn't
> >> we be keeping extents from the very beginning to avoid creating stale
> >> buffers?
> >>
> >
> > This is down in xfs_fs_evict_inode()->xfs_inactive(), which is generally
> > when an inode is evicted from cache. In this case, it looks like the
> > inode is unlinked (permanently removed), the extents are being removed
> > and a bmap btree block is being invalidated as part of that overall
> > process. I don't think this has anything to do with unwritten extents.
> >
>
> Cool. If the inode is indeed unliked, could that sill be triggering
> that condition in xfs_buf_lock? I am not even close to fully
> understanding how XFS manages and/or recycles buffers, but it seems to
> me that if an inode is going away, there isn't really any reason to
> contend for its buffers.
>
I think so.. the inode removal will free various metadata blocks and
they could still be in the stale state by the time something else comes
along and allocates them (re: Dave's example covers this).
> >> 2) xfs_buf_lock -> down
> >> This is one I truly don't understand. What can be causing contention
> >> in this lock? We never have two different cores writing to the same
> >> buffer, nor should we have the same core doingCAP_FOWNER so.
> >>
> >
> > This is not one single lock. An XFS buffer is the data structure used to
> > modify/log/read-write metadata on-disk and each buffer has its own lock
> > to prevent corruption. Buffer lock contention is possible because the
> > filesystem has bits of "global" metadata that has to be updated via
> > buffers.
>
> I see. Since I hate guessing, is there any way you would recommend for
> us to probe the system to determine if this contention scenario is
> indeed the one we are seeing?
>
I'd probably use perf as you are, I'm just not sure if there's any real
way to tell which threads are contending on which AGs. I'm not terribly
experienced with perf. I suppose that if the AGF/AGI read/lock traces
are high up on the list, the chances are higher you're spending a lot of
time waiting on AGs. It's relatively easy to increase the AG count and
allocate inodes under separate AGs (see my previous mail) as an
experiment to see if such contention is reduced.
> We usually open a file, write to it from a single core only,
> sequentially, direct IO only, as well behavedly as we can, with all
> the effort in the world to be good kids to the extent Santa will bring
> us presents without us even asking.
>
> So we were very puzzled to see contention. Contention for global
> metadata updates is the best explanation we've had so far, and would
> be great if we could verify it is indeed the case.
>
> >
> > For example, usually one has multiple allocation groups to maximize
> > parallelism, but we still have per-ag metadata that has to be tracked
> > globally with respect to each AG (e.g., free space trees, inode
> > allocation trees, etc.). Any operation that affects this metadata (e.g.,
> > block/inode allocation) has to lock the agi/agf buffers along with any
> > buffers associated with the modified btree leaf/node blocks, etc.
> >
> > One example in your attached perf traces has several threads looking to
> > acquire the AGF, which is a per-AG data structure for tracking free
> > space in the AG. One thread looks like the inode eviction case noted
> > above (freeing blocks), another looks like a file truncate (also freeing
> > blocks), and yet another is a block allocation due to a direct I/O
> > write. Were any of these operations directed to an inode in a separate
> > AG, they would be able to proceed in parallel (but I believe they would
> > still hit the same codepaths as far as perf can tell).
>
> This is great, great, awesome info Brian. Thanks. We are so far
> allocating inodes and truncating them when we need a new one, but
> maybe there is some allocation pattern that is friendlier to the AG? I
> understand that with such a data structure it may very well be
> impossible to get rid of all waiting, but we will certainly do all we
> can to mitigate it.
>
The truncate will free blocks and require block allocation on subsequent
writes. That might be something you could look into trying to avoid
(e.g., keeping files around and reusing space), but that depends on your
application design. Inodes chunks are allocated and freed dynamically by
default as well. The 'ikeep' mount option keeps inode chunks around
indefinitely (even if individual inodes are all freed) if you wanted to
avoid inode chunk reallocation and know you have a fairly stable working
set of inodes. Per-inode extent size hints might be another option to
increase the size of allocations and perhaps reduce the number of them.
Brian
> >
> >> 3) xfs_file_aio_write_checks -> file_update_time -> xfs_vn_update_time
> >>
> >> You guys seem to have an interface to avoid that, by setting the
> >> FMODE_NOCMTIME flag. This is done by issuing the open by handle ioctl,
> >> which will set this flag for all regular files. That's great, but that
> >> ioctl required CAP_SYS_ADMIN, which is a big no for us, since we run
> >> our server as an unprivileged user. I don't understand, however, why
> >> such an strict check is needed. If we have full rights on the
> >> filesystem, why can't we issue this operation? In my view, CAP_FOWNER
> >> should already be enough.I do understand the handles have to be stable
> >> and a file can have its ownership changed, in which case the previous
> >> owner would keep the handle valid. Is that the reason you went with
> >> the most restrictive capability ?
> >
> > I'm not familiar enough with the open-by-handle stuff to comment on the
> > permission constraints. Perhaps Dave or others can comment further on
> > this bit...
> >
> > Brian
>
> Thanks again Brian. The pointer to the AG stuff was really helpful.
>
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-01 13:11 ` Brian Foster
@ 2015-12-01 13:39 ` Glauber Costa
2015-12-01 14:02 ` Brian Foster
0 siblings, 1 reply; 58+ messages in thread
From: Glauber Costa @ 2015-12-01 13:39 UTC (permalink / raw)
To: Brian Foster; +Cc: Avi Kivity, xfs
>
> The truncate will free blocks and require block allocation on subsequent
> writes. That might be something you could look into trying to avoid
> (e.g., keeping files around and reusing space), but that depends on your
> application design.
This one is a bit hard. We have a journal-like structure for the
modifications issued to the data store, which dominates most of our
write workloads (including this one that I am discussing here). We
could keep they around by renaming them outside of user visibility and
then renaming them back, but that would mean that we are now using
twice as much space. Perhaps we could use a pool that can at least
guarantee one or two allocations from a pre-existing file. I am
assuming here that renaming the file won't block. If it does, we are
better off not doing so.
> Inodes chunks are allocated and freed dynamically by
> default as well. The 'ikeep' mount option keeps inode chunks around
> indefinitely (even if individual inodes are all freed) if you wanted to
> avoid inode chunk reallocation and know you have a fairly stable working
> set of inodes.
I believe we do have a fairly stable inode working set, even though
that depends a bit on what's considered stable. For our journal-like
structure, we will keep them around until we are sure the information
is safe and them delete them - creating new ones as we receive more
data. But that's always bounded in size.
Am I correct to understand that ikeep being passed, new allocations
would just reuse space from the empty chunks on disk?
> Per-inode extent size hints might be another option to
> increase the size of allocations and perhaps reduce the number of them.
>
That's absolutely greatastic. Our files for that journal are all more
or less the same size. That's a great candidate for a hint.
> Brian
Thanks again, Brian
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-01 13:11 ` Brian Foster
@ 2015-12-01 13:58 ` Avi Kivity
2015-12-01 14:01 ` Glauber Costa
2015-12-01 14:56 ` Brian Foster
0 siblings, 2 replies; 58+ messages in thread
From: Avi Kivity @ 2015-12-01 13:58 UTC (permalink / raw)
To: Brian Foster; +Cc: Glauber Costa, xfs
On 12/01/2015 03:11 PM, Brian Foster wrote:
> On Tue, Dec 01, 2015 at 11:08:47AM +0200, Avi Kivity wrote:
>> On 11/30/2015 06:14 PM, Brian Foster wrote:
>>> On Mon, Nov 30, 2015 at 04:29:13PM +0200, Avi Kivity wrote:
>>>> On 11/30/2015 04:10 PM, Brian Foster wrote:
> ...
>>> The agsize/agcount mkfs-time heuristics change depending on the type of
>>> storage. A single AG can be up to 1TB and if the fs is not considered
>>> "multidisk" (e.g., no stripe unit/width is defined), 4 AGs is the
>>> default up to 4TB. If a stripe unit is set, the agsize/agcount is
>>> adjusted depending on the size of the overall volume (see
>>> xfsprogs-dev/mkfs/xfs_mkfs.c:calc_default_ag_geometry() for details).
>> We'll experiment with this. Surely it depends on more than the amount of
>> storage? If you have a high op rate you'll be more likely to excite
>> contention, no?
>>
> Sure. The absolute optimal configuration for your workload probably
> depends on more than storage size, but mkfs doesn't have that
> information. In general, it tries to use the most reasonable
> configuration based on the storage and expected workload. If you want to
> tweak it beyond that, indeed, the best bet is to experiment with what
> works.
We will do that.
>>>> Are those locks held around I/O, or just CPU operations, or a mix?
>>> I believe it's a mix of modifications and I/O, though it looks like some
>>> of the I/O cases don't necessarily wait on the lock. E.g., the AIL
>>> pushing case will trylock and defer to the next list iteration if the
>>> buffer is busy.
>>>
>> Ok. For us sleeping in io_submit() is death because we have no other thread
>> on that core to take its place.
>>
> The above is with regard to metadata I/O, whereas io_submit() is
> obviously for user I/O.
Won't io_submit() also trigger metadata I/O? Or is that all deferred to
async tasks? I don't mind them blocking each other as long as they let
my io_submit alone.
> io_submit() can probably block in a variety of
> places afaict... it might have to read in the inode extent map, allocate
> blocks, take inode/ag locks, reserve log space for transactions, etc.
Any chance of changing all that to be asynchronous? Doesn't sound too
hard, if somebody else has to do it.
>
> It sounds to me that first and foremost you want to make sure you don't
> have however many parallel operations you typically have running
> contending on the same inodes or AGs. Hint: creating files under
> separate subdirectories is a quick and easy way to allocate inodes under
> separate AGs (the agno is encoded into the upper bits of the inode
> number).
Unfortunately our directory layout cannot be changed. And doesn't this
require having agcount == O(number of active files)? That is easily in
the thousands.
> Reducing the frequency of block allocation/frees might also be
> another help (e.g., preallocate and reuse files,
Isn't that discouraged for SSDs?
We can do that for a subset of our files.
We do use XFS_IOC_FSSETXATTR though.
> 'mount -o ikeep,'
Interesting. Our files are large so we could try this.
> etc.). Beyond that, you probably want to make sure the log is large
> enough to support all concurrent operations. See the xfs_log_grant_*
> tracepoints for a window into if/how long transaction reservations might
> be waiting on the log.
I see that on an 400G fs, the log is 180MB. Seems plenty large for
write operations that are mostly large sequential, though I've no real
feel for the numbers. Will keep an eye on this.
Thanks for all the info.
> Brian
>
>> _______________________________________________
>> xfs mailing list
>> xfs@oss.sgi.com
>> http://oss.sgi.com/mailman/listinfo/xfs
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-01 13:58 ` Avi Kivity
@ 2015-12-01 14:01 ` Glauber Costa
2015-12-01 14:37 ` Avi Kivity
2015-12-01 20:45 ` Dave Chinner
2015-12-01 14:56 ` Brian Foster
1 sibling, 2 replies; 58+ messages in thread
From: Glauber Costa @ 2015-12-01 14:01 UTC (permalink / raw)
To: Avi Kivity; +Cc: Brian Foster, xfs
On Tue, Dec 1, 2015 at 8:58 AM, Avi Kivity <avi@scylladb.com> wrote:
>
>
> On 12/01/2015 03:11 PM, Brian Foster wrote:
>>
>> On Tue, Dec 01, 2015 at 11:08:47AM +0200, Avi Kivity wrote:
>>>
>>> On 11/30/2015 06:14 PM, Brian Foster wrote:
>>>>
>>>> On Mon, Nov 30, 2015 at 04:29:13PM +0200, Avi Kivity wrote:
>>>>>
>>>>> On 11/30/2015 04:10 PM, Brian Foster wrote:
>>
>> ...
>>>>
>>>> The agsize/agcount mkfs-time heuristics change depending on the type of
>>>> storage. A single AG can be up to 1TB and if the fs is not considered
>>>> "multidisk" (e.g., no stripe unit/width is defined), 4 AGs is the
>>>> default up to 4TB. If a stripe unit is set, the agsize/agcount is
>>>> adjusted depending on the size of the overall volume (see
>>>> xfsprogs-dev/mkfs/xfs_mkfs.c:calc_default_ag_geometry() for details).
>>>
>>> We'll experiment with this. Surely it depends on more than the amount of
>>> storage? If you have a high op rate you'll be more likely to excite
>>> contention, no?
>>>
>> Sure. The absolute optimal configuration for your workload probably
>> depends on more than storage size, but mkfs doesn't have that
>> information. In general, it tries to use the most reasonable
>> configuration based on the storage and expected workload. If you want to
>> tweak it beyond that, indeed, the best bet is to experiment with what
>> works.
>
>
> We will do that.
>
>>>>> Are those locks held around I/O, or just CPU operations, or a mix?
>>>>
>>>> I believe it's a mix of modifications and I/O, though it looks like some
>>>> of the I/O cases don't necessarily wait on the lock. E.g., the AIL
>>>> pushing case will trylock and defer to the next list iteration if the
>>>> buffer is busy.
>>>>
>>> Ok. For us sleeping in io_submit() is death because we have no other
>>> thread
>>> on that core to take its place.
>>>
>> The above is with regard to metadata I/O, whereas io_submit() is
>> obviously for user I/O.
>
>
> Won't io_submit() also trigger metadata I/O? Or is that all deferred to
> async tasks? I don't mind them blocking each other as long as they let my
> io_submit alone.
>
>> io_submit() can probably block in a variety of
>> places afaict... it might have to read in the inode extent map, allocate
>> blocks, take inode/ag locks, reserve log space for transactions, etc.
>
>
> Any chance of changing all that to be asynchronous? Doesn't sound too hard,
> if somebody else has to do it.
>
>>
>> It sounds to me that first and foremost you want to make sure you don't
>> have however many parallel operations you typically have running
>> contending on the same inodes or AGs. Hint: creating files under
>> separate subdirectories is a quick and easy way to allocate inodes under
>> separate AGs (the agno is encoded into the upper bits of the inode
>> number).
>
>
> Unfortunately our directory layout cannot be changed. And doesn't this
> require having agcount == O(number of active files)? That is easily in the
> thousands.
Actually, wouldn't agcount == O(nr_cpus) be good enough?
>
>> Reducing the frequency of block allocation/frees might also be
>> another help (e.g., preallocate and reuse files,
>
>
> Isn't that discouraged for SSDs?
>
> We can do that for a subset of our files.
>
> We do use XFS_IOC_FSSETXATTR though.
>
>> 'mount -o ikeep,'
>
>
> Interesting. Our files are large so we could try this.
>
>> etc.). Beyond that, you probably want to make sure the log is large
>> enough to support all concurrent operations. See the xfs_log_grant_*
>> tracepoints for a window into if/how long transaction reservations might
>> be waiting on the log.
>
>
> I see that on an 400G fs, the log is 180MB. Seems plenty large for write
> operations that are mostly large sequential, though I've no real feel for
> the numbers. Will keep an eye on this.
>
> Thanks for all the info.
>
>
>> Brian
>>
>>> _______________________________________________
>>> xfs mailing list
>>> xfs@oss.sgi.com
>>> http://oss.sgi.com/mailman/listinfo/xfs
>
>
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-01 13:39 ` Glauber Costa
@ 2015-12-01 14:02 ` Brian Foster
0 siblings, 0 replies; 58+ messages in thread
From: Brian Foster @ 2015-12-01 14:02 UTC (permalink / raw)
To: Glauber Costa; +Cc: Avi Kivity, xfs
On Tue, Dec 01, 2015 at 08:39:06AM -0500, Glauber Costa wrote:
> >
> > The truncate will free blocks and require block allocation on subsequent
> > writes. That might be something you could look into trying to avoid
> > (e.g., keeping files around and reusing space), but that depends on your
> > application design.
>
>
> This one is a bit hard. We have a journal-like structure for the
> modifications issued to the data store, which dominates most of our
> write workloads (including this one that I am discussing here). We
> could keep they around by renaming them outside of user visibility and
> then renaming them back, but that would mean that we are now using
> twice as much space. Perhaps we could use a pool that can at least
> guarantee one or two allocations from a pre-existing file. I am
> assuming here that renaming the file won't block. If it does, we are
> better off not doing so.
>
> > Inodes chunks are allocated and freed dynamically by
> > default as well. The 'ikeep' mount option keeps inode chunks around
> > indefinitely (even if individual inodes are all freed) if you wanted to
> > avoid inode chunk reallocation and know you have a fairly stable working
> > set of inodes.
>
> I believe we do have a fairly stable inode working set, even though
> that depends a bit on what's considered stable. For our journal-like
> structure, we will keep them around until we are sure the information
> is safe and them delete them - creating new ones as we receive more
> data. But that's always bounded in size.
>
> Am I correct to understand that ikeep being passed, new allocations
> would just reuse space from the empty chunks on disk?
>
Yes.. current behavior is that inodes are allocated and freed in chunks
of 64. When the entire chunk of inodes is freed from the namespace, the
chunk is freed (i.e., it is now free space). With ikeep, inode chunks
are never freed. When an individual inode allocation request is made,
the inode is allocated from one of the existing inode chunks before a
new chunk is allocated.
The tradeoff is that you could consume a significant amount of space
with inodes, free a bunch of them and that space is not freed. So that
is something to be aware of for your use case, particularly if the fs
has other uses from your journaling mechanism described above because it
affects the entire fs.
>
> > Per-inode extent size hints might be another option to
> > increase the size of allocations and perhaps reduce the number of them.
> >
>
> That's absolutely greatastic. Our files for that journal are all more
> or less the same size. That's a great candidate for a hint.
>
You could consider preallocation (fallocate()) as well if you know the
full size in advance.
Brian
> > Brian
>
> Thanks again, Brian
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-01 14:01 ` Glauber Costa
@ 2015-12-01 14:37 ` Avi Kivity
2015-12-01 20:45 ` Dave Chinner
1 sibling, 0 replies; 58+ messages in thread
From: Avi Kivity @ 2015-12-01 14:37 UTC (permalink / raw)
To: Glauber Costa; +Cc: Brian Foster, xfs
On 12/01/2015 04:01 PM, Glauber Costa wrote:
> On Tue, Dec 1, 2015 at 8:58 AM, Avi Kivity <avi@scylladb.com> wrote:
>>
>> On 12/01/2015 03:11 PM, Brian Foster wrote:
>>> On Tue, Dec 01, 2015 at 11:08:47AM +0200, Avi Kivity wrote:
>>>> On 11/30/2015 06:14 PM, Brian Foster wrote:
>>>>> On Mon, Nov 30, 2015 at 04:29:13PM +0200, Avi Kivity wrote:
>>>>>> On 11/30/2015 04:10 PM, Brian Foster wrote:
>>> ...
>>>>> The agsize/agcount mkfs-time heuristics change depending on the type of
>>>>> storage. A single AG can be up to 1TB and if the fs is not considered
>>>>> "multidisk" (e.g., no stripe unit/width is defined), 4 AGs is the
>>>>> default up to 4TB. If a stripe unit is set, the agsize/agcount is
>>>>> adjusted depending on the size of the overall volume (see
>>>>> xfsprogs-dev/mkfs/xfs_mkfs.c:calc_default_ag_geometry() for details).
>>>> We'll experiment with this. Surely it depends on more than the amount of
>>>> storage? If you have a high op rate you'll be more likely to excite
>>>> contention, no?
>>>>
>>> Sure. The absolute optimal configuration for your workload probably
>>> depends on more than storage size, but mkfs doesn't have that
>>> information. In general, it tries to use the most reasonable
>>> configuration based on the storage and expected workload. If you want to
>>> tweak it beyond that, indeed, the best bet is to experiment with what
>>> works.
>>
>> We will do that.
>>
>>>>>> Are those locks held around I/O, or just CPU operations, or a mix?
>>>>> I believe it's a mix of modifications and I/O, though it looks like some
>>>>> of the I/O cases don't necessarily wait on the lock. E.g., the AIL
>>>>> pushing case will trylock and defer to the next list iteration if the
>>>>> buffer is busy.
>>>>>
>>>> Ok. For us sleeping in io_submit() is death because we have no other
>>>> thread
>>>> on that core to take its place.
>>>>
>>> The above is with regard to metadata I/O, whereas io_submit() is
>>> obviously for user I/O.
>>
>> Won't io_submit() also trigger metadata I/O? Or is that all deferred to
>> async tasks? I don't mind them blocking each other as long as they let my
>> io_submit alone.
>>
>>> io_submit() can probably block in a variety of
>>> places afaict... it might have to read in the inode extent map, allocate
>>> blocks, take inode/ag locks, reserve log space for transactions, etc.
>>
>> Any chance of changing all that to be asynchronous? Doesn't sound too hard,
>> if somebody else has to do it.
>>
>>> It sounds to me that first and foremost you want to make sure you don't
>>> have however many parallel operations you typically have running
>>> contending on the same inodes or AGs. Hint: creating files under
>>> separate subdirectories is a quick and easy way to allocate inodes under
>>> separate AGs (the agno is encoded into the upper bits of the inode
>>> number).
>>
>> Unfortunately our directory layout cannot be changed. And doesn't this
>> require having agcount == O(number of active files)? That is easily in the
>> thousands.
> Actually, wouldn't agcount == O(nr_cpus) be good enough?
Depends on whether the locks are around I/O or cpu access only.
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-01 13:58 ` Avi Kivity
2015-12-01 14:01 ` Glauber Costa
@ 2015-12-01 14:56 ` Brian Foster
2015-12-01 15:22 ` Avi Kivity
1 sibling, 1 reply; 58+ messages in thread
From: Brian Foster @ 2015-12-01 14:56 UTC (permalink / raw)
To: Avi Kivity; +Cc: Glauber Costa, xfs
On Tue, Dec 01, 2015 at 03:58:28PM +0200, Avi Kivity wrote:
>
>
> On 12/01/2015 03:11 PM, Brian Foster wrote:
> >On Tue, Dec 01, 2015 at 11:08:47AM +0200, Avi Kivity wrote:
> >>On 11/30/2015 06:14 PM, Brian Foster wrote:
> >>>On Mon, Nov 30, 2015 at 04:29:13PM +0200, Avi Kivity wrote:
> >>>>On 11/30/2015 04:10 PM, Brian Foster wrote:
> >...
> >>>The agsize/agcount mkfs-time heuristics change depending on the type of
> >>>storage. A single AG can be up to 1TB and if the fs is not considered
> >>>"multidisk" (e.g., no stripe unit/width is defined), 4 AGs is the
> >>>default up to 4TB. If a stripe unit is set, the agsize/agcount is
> >>>adjusted depending on the size of the overall volume (see
> >>>xfsprogs-dev/mkfs/xfs_mkfs.c:calc_default_ag_geometry() for details).
> >>We'll experiment with this. Surely it depends on more than the amount of
> >>storage? If you have a high op rate you'll be more likely to excite
> >>contention, no?
> >>
> >Sure. The absolute optimal configuration for your workload probably
> >depends on more than storage size, but mkfs doesn't have that
> >information. In general, it tries to use the most reasonable
> >configuration based on the storage and expected workload. If you want to
> >tweak it beyond that, indeed, the best bet is to experiment with what
> >works.
>
> We will do that.
>
> >>>>Are those locks held around I/O, or just CPU operations, or a mix?
> >>>I believe it's a mix of modifications and I/O, though it looks like some
> >>>of the I/O cases don't necessarily wait on the lock. E.g., the AIL
> >>>pushing case will trylock and defer to the next list iteration if the
> >>>buffer is busy.
> >>>
> >>Ok. For us sleeping in io_submit() is death because we have no other thread
> >>on that core to take its place.
> >>
> >The above is with regard to metadata I/O, whereas io_submit() is
> >obviously for user I/O.
>
> Won't io_submit() also trigger metadata I/O? Or is that all deferred to
> async tasks? I don't mind them blocking each other as long as they let my
> io_submit alone.
>
Yeah, it can trigger metadata reads, force the log (the stale buffer
example) or push the AIL (wait on log space). Metadata changes made
directly via your I/O request are logged/committed via transactions,
which are generally processed asynchronously from that point on.
> > io_submit() can probably block in a variety of
> >places afaict... it might have to read in the inode extent map, allocate
> >blocks, take inode/ag locks, reserve log space for transactions, etc.
>
> Any chance of changing all that to be asynchronous? Doesn't sound too hard,
> if somebody else has to do it.
>
I'm not following... if the fs needs to read in the inode extent map to
prepare for an allocation, what else can the thread do but wait? Are you
suggesting the request kick off whatever the blocking action happens to
be asynchronously and return with an error such that the request can be
retried later?
> >
> >It sounds to me that first and foremost you want to make sure you don't
> >have however many parallel operations you typically have running
> >contending on the same inodes or AGs. Hint: creating files under
> >separate subdirectories is a quick and easy way to allocate inodes under
> >separate AGs (the agno is encoded into the upper bits of the inode
> >number).
>
> Unfortunately our directory layout cannot be changed. And doesn't this
> require having agcount == O(number of active files)? That is easily in the
> thousands.
>
I think Glauber's O(nr_cpus) comment is probably the more likely
ballpark, but really it's something you'll probably just need to test to
see how far you need to go to avoid AG contention.
I'm primarily throwing the subdir thing out there for testing purposes.
It's just an easy way to create inodes in a bunch of separate AGs so you
can determine whether/how much it really helps with modified AG counts.
I don't know enough about your application design to really comment on
that...
> > Reducing the frequency of block allocation/frees might also be
> >another help (e.g., preallocate and reuse files,
>
> Isn't that discouraged for SSDs?
>
Perhaps, if you're referring to the fact that the blocks are never freed
and thus never discarded..? Are you running fstrim?
If so, it would certainly impact that by holding blocks as allocated to
inodes as opposed to putting them in free space trees where they can be
discarded. If not, I don't see how it would make a difference, but
perhaps I misunderstand the point. That said, there's probably others on
the list who can more definitively discuss SSD characteristics than I...
> We can do that for a subset of our files.
>
> We do use XFS_IOC_FSSETXATTR though.
>
> >'mount -o ikeep,'
>
> Interesting. Our files are large so we could try this.
>
Just to be clear... this behavior change is more directly associated
with file count than file size (though indirectly larger files might
mean you have less of them, if that's your point).
To generalize a bit, I'd be more weary of using this option if your
filesystem can be used in an unstructured manner in any way. For
example, if the file count can balloon up and back down temporarily,
that's going to allocate a bunch of metadata space for inodes that won't
ever be reclaimed or reused for anything other than inodes.
> >etc.). Beyond that, you probably want to make sure the log is large
> >enough to support all concurrent operations. See the xfs_log_grant_*
> >tracepoints for a window into if/how long transaction reservations might
> >be waiting on the log.
>
> I see that on an 400G fs, the log is 180MB. Seems plenty large for write
> operations that are mostly large sequential, though I've no real feel for
> the numbers. Will keep an eye on this.
>
FWIW, XFS on recent kernels has grown some sysfs entries that might help
give an idea of log reservation state at runtime. See the entries under
/sys/fs/xfs/<dev>/log for details.
Brian
> Thanks for all the info.
>
> >Brian
> >
> >>_______________________________________________
> >>xfs mailing list
> >>xfs@oss.sgi.com
> >>http://oss.sgi.com/mailman/listinfo/xfs
>
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-01 14:56 ` Brian Foster
@ 2015-12-01 15:22 ` Avi Kivity
2015-12-01 16:01 ` Brian Foster
2015-12-01 21:04 ` Dave Chinner
0 siblings, 2 replies; 58+ messages in thread
From: Avi Kivity @ 2015-12-01 15:22 UTC (permalink / raw)
To: Brian Foster; +Cc: Glauber Costa, xfs
On 12/01/2015 04:56 PM, Brian Foster wrote:
> On Tue, Dec 01, 2015 at 03:58:28PM +0200, Avi Kivity wrote:
>>
>> On 12/01/2015 03:11 PM, Brian Foster wrote:
>>> On Tue, Dec 01, 2015 at 11:08:47AM +0200, Avi Kivity wrote:
>>>> On 11/30/2015 06:14 PM, Brian Foster wrote:
>>>>> On Mon, Nov 30, 2015 at 04:29:13PM +0200, Avi Kivity wrote:
>>>>>> On 11/30/2015 04:10 PM, Brian Foster wrote:
>>> ...
>>>>> The agsize/agcount mkfs-time heuristics change depending on the type of
>>>>> storage. A single AG can be up to 1TB and if the fs is not considered
>>>>> "multidisk" (e.g., no stripe unit/width is defined), 4 AGs is the
>>>>> default up to 4TB. If a stripe unit is set, the agsize/agcount is
>>>>> adjusted depending on the size of the overall volume (see
>>>>> xfsprogs-dev/mkfs/xfs_mkfs.c:calc_default_ag_geometry() for details).
>>>> We'll experiment with this. Surely it depends on more than the amount of
>>>> storage? If you have a high op rate you'll be more likely to excite
>>>> contention, no?
>>>>
>>> Sure. The absolute optimal configuration for your workload probably
>>> depends on more than storage size, but mkfs doesn't have that
>>> information. In general, it tries to use the most reasonable
>>> configuration based on the storage and expected workload. If you want to
>>> tweak it beyond that, indeed, the best bet is to experiment with what
>>> works.
>> We will do that.
>>
>>>>>> Are those locks held around I/O, or just CPU operations, or a mix?
>>>>> I believe it's a mix of modifications and I/O, though it looks like some
>>>>> of the I/O cases don't necessarily wait on the lock. E.g., the AIL
>>>>> pushing case will trylock and defer to the next list iteration if the
>>>>> buffer is busy.
>>>>>
>>>> Ok. For us sleeping in io_submit() is death because we have no other thread
>>>> on that core to take its place.
>>>>
>>> The above is with regard to metadata I/O, whereas io_submit() is
>>> obviously for user I/O.
>> Won't io_submit() also trigger metadata I/O? Or is that all deferred to
>> async tasks? I don't mind them blocking each other as long as they let my
>> io_submit alone.
>>
> Yeah, it can trigger metadata reads, force the log (the stale buffer
> example) or push the AIL (wait on log space). Metadata changes made
> directly via your I/O request are logged/committed via transactions,
> which are generally processed asynchronously from that point on.
>
>>> io_submit() can probably block in a variety of
>>> places afaict... it might have to read in the inode extent map, allocate
>>> blocks, take inode/ag locks, reserve log space for transactions, etc.
>> Any chance of changing all that to be asynchronous? Doesn't sound too hard,
>> if somebody else has to do it.
>>
> I'm not following... if the fs needs to read in the inode extent map to
> prepare for an allocation, what else can the thread do but wait? Are you
> suggesting the request kick off whatever the blocking action happens to
> be asynchronously and return with an error such that the request can be
> retried later?
Not quite, it should be invisible to the caller.
That is, the code called by io_submit() (file_operations::write_iter, it
seems to be called today) can kick off this operation and have it
continue from where it left off.
Seastar (the async user framework which we use to drive xfs) makes
writing code like this easy, using continuations; but of course from
ordinary threaded code it can be quite hard.
btw, there was an attempt to make ext[34] async using this method, but I
think it was ripped out. Yes, the mortal remains can still be seen with
'git grep EIOCBQUEUED'.
>
>>> It sounds to me that first and foremost you want to make sure you don't
>>> have however many parallel operations you typically have running
>>> contending on the same inodes or AGs. Hint: creating files under
>>> separate subdirectories is a quick and easy way to allocate inodes under
>>> separate AGs (the agno is encoded into the upper bits of the inode
>>> number).
>> Unfortunately our directory layout cannot be changed. And doesn't this
>> require having agcount == O(number of active files)? That is easily in the
>> thousands.
>>
> I think Glauber's O(nr_cpus) comment is probably the more likely
> ballpark, but really it's something you'll probably just need to test to
> see how far you need to go to avoid AG contention.
>
> I'm primarily throwing the subdir thing out there for testing purposes.
> It's just an easy way to create inodes in a bunch of separate AGs so you
> can determine whether/how much it really helps with modified AG counts.
> I don't know enough about your application design to really comment on
> that...
We have O(cpus) shards that operate independently. Each shard writes
32MB commitlog files (that are pre-truncated to 32MB to allow concurrent
writes without blocking); the files are then flushed and closed, and
later removed. In parallel there are sequential writes and reads of
large files using 128kB buffers), as well as random reads. Files are
immutable (append-only), and if a file is being written, it is not
concurrently read. In general files are not shared across shards. All
I/O is async and O_DIRECT. open(), truncate(), fdatasync(), and friends
are called from a helper thread.
As far as I can tell it should a very friendly load for XFS and SSDs.
>
>>> Reducing the frequency of block allocation/frees might also be
>>> another help (e.g., preallocate and reuse files,
>> Isn't that discouraged for SSDs?
>>
> Perhaps, if you're referring to the fact that the blocks are never freed
> and thus never discarded..? Are you running fstrim?
mount -o discard. And yes, overwrites are supposedly more expensive
than trim old data + allocate new data, but maybe if you compare it with
the work XFS has to do, perhaps the tradeoff is bad.
>
> If so, it would certainly impact that by holding blocks as allocated to
> inodes as opposed to putting them in free space trees where they can be
> discarded. If not, I don't see how it would make a difference, but
> perhaps I misunderstand the point. That said, there's probably others on
> the list who can more definitively discuss SSD characteristics than I...
>
>> We can do that for a subset of our files.
>>
>> We do use XFS_IOC_FSSETXATTR though.
>>
>>> 'mount -o ikeep,'
>> Interesting. Our files are large so we could try this.
>>
> Just to be clear... this behavior change is more directly associated
> with file count than file size (though indirectly larger files might
> mean you have less of them, if that's your point).
Yes, that's what I meant, and especially that if a lot of files are
removed we'd be losing the inode space allocated to them.
>
> To generalize a bit, I'd be more weary of using this option if your
> filesystem can be used in an unstructured manner in any way. For
> example, if the file count can balloon up and back down temporarily,
> that's going to allocate a bunch of metadata space for inodes that won't
> ever be reclaimed or reused for anything other than inodes.
Exactly. File count can balloon, but files will be large, so even the
worst case waste is very limited.
>
>>> etc.). Beyond that, you probably want to make sure the log is large
>>> enough to support all concurrent operations. See the xfs_log_grant_*
>>> tracepoints for a window into if/how long transaction reservations might
>>> be waiting on the log.
>> I see that on an 400G fs, the log is 180MB. Seems plenty large for write
>> operations that are mostly large sequential, though I've no real feel for
>> the numbers. Will keep an eye on this.
>>
> FWIW, XFS on recent kernels has grown some sysfs entries that might help
> give an idea of log reservation state at runtime. See the entries under
> /sys/fs/xfs/<dev>/log for details.
Great. We will study those with great interest.
>
> Brian
>
>> Thanks for all the info.
>>
>>> Brian
>>>
>>>> _______________________________________________
>>>> xfs mailing list
>>>> xfs@oss.sgi.com
>>>> http://oss.sgi.com/mailman/listinfo/xfs
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-01 15:22 ` Avi Kivity
@ 2015-12-01 16:01 ` Brian Foster
2015-12-01 16:08 ` Avi Kivity
2015-12-01 21:04 ` Dave Chinner
1 sibling, 1 reply; 58+ messages in thread
From: Brian Foster @ 2015-12-01 16:01 UTC (permalink / raw)
To: Avi Kivity; +Cc: Glauber Costa, xfs
On Tue, Dec 01, 2015 at 05:22:38PM +0200, Avi Kivity wrote:
>
>
> On 12/01/2015 04:56 PM, Brian Foster wrote:
> >On Tue, Dec 01, 2015 at 03:58:28PM +0200, Avi Kivity wrote:
> >>
> >>On 12/01/2015 03:11 PM, Brian Foster wrote:
> >>>On Tue, Dec 01, 2015 at 11:08:47AM +0200, Avi Kivity wrote:
> >>>>On 11/30/2015 06:14 PM, Brian Foster wrote:
> >>>>>On Mon, Nov 30, 2015 at 04:29:13PM +0200, Avi Kivity wrote:
> >>>>>>On 11/30/2015 04:10 PM, Brian Foster wrote:
> >>>...
> >>>>>The agsize/agcount mkfs-time heuristics change depending on the type of
> >>>>>storage. A single AG can be up to 1TB and if the fs is not considered
> >>>>>"multidisk" (e.g., no stripe unit/width is defined), 4 AGs is the
> >>>>>default up to 4TB. If a stripe unit is set, the agsize/agcount is
> >>>>>adjusted depending on the size of the overall volume (see
> >>>>>xfsprogs-dev/mkfs/xfs_mkfs.c:calc_default_ag_geometry() for details).
> >>>>We'll experiment with this. Surely it depends on more than the amount of
> >>>>storage? If you have a high op rate you'll be more likely to excite
> >>>>contention, no?
> >>>>
> >>>Sure. The absolute optimal configuration for your workload probably
> >>>depends on more than storage size, but mkfs doesn't have that
> >>>information. In general, it tries to use the most reasonable
> >>>configuration based on the storage and expected workload. If you want to
> >>>tweak it beyond that, indeed, the best bet is to experiment with what
> >>>works.
> >>We will do that.
> >>
> >>>>>>Are those locks held around I/O, or just CPU operations, or a mix?
> >>>>>I believe it's a mix of modifications and I/O, though it looks like some
> >>>>>of the I/O cases don't necessarily wait on the lock. E.g., the AIL
> >>>>>pushing case will trylock and defer to the next list iteration if the
> >>>>>buffer is busy.
> >>>>>
> >>>>Ok. For us sleeping in io_submit() is death because we have no other thread
> >>>>on that core to take its place.
> >>>>
> >>>The above is with regard to metadata I/O, whereas io_submit() is
> >>>obviously for user I/O.
> >>Won't io_submit() also trigger metadata I/O? Or is that all deferred to
> >>async tasks? I don't mind them blocking each other as long as they let my
> >>io_submit alone.
> >>
> >Yeah, it can trigger metadata reads, force the log (the stale buffer
> >example) or push the AIL (wait on log space). Metadata changes made
> >directly via your I/O request are logged/committed via transactions,
> >which are generally processed asynchronously from that point on.
> >
> >>> io_submit() can probably block in a variety of
> >>>places afaict... it might have to read in the inode extent map, allocate
> >>>blocks, take inode/ag locks, reserve log space for transactions, etc.
> >>Any chance of changing all that to be asynchronous? Doesn't sound too hard,
> >>if somebody else has to do it.
> >>
> >I'm not following... if the fs needs to read in the inode extent map to
> >prepare for an allocation, what else can the thread do but wait? Are you
> >suggesting the request kick off whatever the blocking action happens to
> >be asynchronously and return with an error such that the request can be
> >retried later?
>
> Not quite, it should be invisible to the caller.
>
> That is, the code called by io_submit() (file_operations::write_iter, it
> seems to be called today) can kick off this operation and have it continue
> from where it left off.
>
Isn't that generally what happens today? We submit an I/O which is
asynchronous in nature and wait on a completion, which causes the cpu to
schedule and execute another task until the completion is set by I/O
completion (via an async callback). At that point, the issuing thread
continues where it left off. I suspect I'm missing something... can you
elaborate on what you'd do differently here (and how it helps)?
> Seastar (the async user framework which we use to drive xfs) makes writing
> code like this easy, using continuations; but of course from ordinary
> threaded code it can be quite hard.
>
> btw, there was an attempt to make ext[34] async using this method, but I
> think it was ripped out. Yes, the mortal remains can still be seen with
> 'git grep EIOCBQUEUED'.
>
> >
> >>>It sounds to me that first and foremost you want to make sure you don't
> >>>have however many parallel operations you typically have running
> >>>contending on the same inodes or AGs. Hint: creating files under
> >>>separate subdirectories is a quick and easy way to allocate inodes under
> >>>separate AGs (the agno is encoded into the upper bits of the inode
> >>>number).
> >>Unfortunately our directory layout cannot be changed. And doesn't this
> >>require having agcount == O(number of active files)? That is easily in the
> >>thousands.
> >>
> >I think Glauber's O(nr_cpus) comment is probably the more likely
> >ballpark, but really it's something you'll probably just need to test to
> >see how far you need to go to avoid AG contention.
> >
> >I'm primarily throwing the subdir thing out there for testing purposes.
> >It's just an easy way to create inodes in a bunch of separate AGs so you
> >can determine whether/how much it really helps with modified AG counts.
> >I don't know enough about your application design to really comment on
> >that...
>
> We have O(cpus) shards that operate independently. Each shard writes 32MB
> commitlog files (that are pre-truncated to 32MB to allow concurrent writes
> without blocking); the files are then flushed and closed, and later removed.
> In parallel there are sequential writes and reads of large files using 128kB
> buffers), as well as random reads. Files are immutable (append-only), and
> if a file is being written, it is not concurrently read. In general files
> are not shared across shards. All I/O is async and O_DIRECT. open(),
> truncate(), fdatasync(), and friends are called from a helper thread.
>
> As far as I can tell it should a very friendly load for XFS and SSDs.
>
> >
> >>> Reducing the frequency of block allocation/frees might also be
> >>>another help (e.g., preallocate and reuse files,
> >>Isn't that discouraged for SSDs?
> >>
> >Perhaps, if you're referring to the fact that the blocks are never freed
> >and thus never discarded..? Are you running fstrim?
>
> mount -o discard. And yes, overwrites are supposedly more expensive than
> trim old data + allocate new data, but maybe if you compare it with the work
> XFS has to do, perhaps the tradeoff is bad.
>
Ok, my understanding is that '-o discard' is not recommended in favor of
periodic fstrim for performance reasons, but that may or may not still
be the case.
Brian
>
> >
> >If so, it would certainly impact that by holding blocks as allocated to
> >inodes as opposed to putting them in free space trees where they can be
> >discarded. If not, I don't see how it would make a difference, but
> >perhaps I misunderstand the point. That said, there's probably others on
> >the list who can more definitively discuss SSD characteristics than I...
>
>
>
> >
> >>We can do that for a subset of our files.
> >>
> >>We do use XFS_IOC_FSSETXATTR though.
> >>
> >>>'mount -o ikeep,'
> >>Interesting. Our files are large so we could try this.
> >>
> >Just to be clear... this behavior change is more directly associated
> >with file count than file size (though indirectly larger files might
> >mean you have less of them, if that's your point).
>
> Yes, that's what I meant, and especially that if a lot of files are removed
> we'd be losing the inode space allocated to them.
>
> >
> >To generalize a bit, I'd be more weary of using this option if your
> >filesystem can be used in an unstructured manner in any way. For
> >example, if the file count can balloon up and back down temporarily,
> >that's going to allocate a bunch of metadata space for inodes that won't
> >ever be reclaimed or reused for anything other than inodes.
>
> Exactly. File count can balloon, but files will be large, so even the worst
> case waste is very limited.
>
> >
> >>>etc.). Beyond that, you probably want to make sure the log is large
> >>>enough to support all concurrent operations. See the xfs_log_grant_*
> >>>tracepoints for a window into if/how long transaction reservations might
> >>>be waiting on the log.
> >>I see that on an 400G fs, the log is 180MB. Seems plenty large for write
> >>operations that are mostly large sequential, though I've no real feel for
> >>the numbers. Will keep an eye on this.
> >>
> >FWIW, XFS on recent kernels has grown some sysfs entries that might help
> >give an idea of log reservation state at runtime. See the entries under
> >/sys/fs/xfs/<dev>/log for details.
>
> Great. We will study those with great interest.
>
> >
> >Brian
> >
> >>Thanks for all the info.
> >>
> >>>Brian
> >>>
> >>>>_______________________________________________
> >>>>xfs mailing list
> >>>>xfs@oss.sgi.com
> >>>>http://oss.sgi.com/mailman/listinfo/xfs
>
>
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-01 16:01 ` Brian Foster
@ 2015-12-01 16:08 ` Avi Kivity
2015-12-01 16:29 ` Brian Foster
0 siblings, 1 reply; 58+ messages in thread
From: Avi Kivity @ 2015-12-01 16:08 UTC (permalink / raw)
To: Brian Foster; +Cc: Glauber Costa, xfs
On 12/01/2015 06:01 PM, Brian Foster wrote:
> On Tue, Dec 01, 2015 at 05:22:38PM +0200, Avi Kivity wrote:
>>
>> On 12/01/2015 04:56 PM, Brian Foster wrote:
>>> On Tue, Dec 01, 2015 at 03:58:28PM +0200, Avi Kivity wrote:
>>>> On 12/01/2015 03:11 PM, Brian Foster wrote:
>>>>> On Tue, Dec 01, 2015 at 11:08:47AM +0200, Avi Kivity wrote:
>>>>>> On 11/30/2015 06:14 PM, Brian Foster wrote:
>>>>>>> On Mon, Nov 30, 2015 at 04:29:13PM +0200, Avi Kivity wrote:
>>>>>>>> On 11/30/2015 04:10 PM, Brian Foster wrote:
>>>>> ...
>>>>>>> The agsize/agcount mkfs-time heuristics change depending on the type of
>>>>>>> storage. A single AG can be up to 1TB and if the fs is not considered
>>>>>>> "multidisk" (e.g., no stripe unit/width is defined), 4 AGs is the
>>>>>>> default up to 4TB. If a stripe unit is set, the agsize/agcount is
>>>>>>> adjusted depending on the size of the overall volume (see
>>>>>>> xfsprogs-dev/mkfs/xfs_mkfs.c:calc_default_ag_geometry() for details).
>>>>>> We'll experiment with this. Surely it depends on more than the amount of
>>>>>> storage? If you have a high op rate you'll be more likely to excite
>>>>>> contention, no?
>>>>>>
>>>>> Sure. The absolute optimal configuration for your workload probably
>>>>> depends on more than storage size, but mkfs doesn't have that
>>>>> information. In general, it tries to use the most reasonable
>>>>> configuration based on the storage and expected workload. If you want to
>>>>> tweak it beyond that, indeed, the best bet is to experiment with what
>>>>> works.
>>>> We will do that.
>>>>
>>>>>>>> Are those locks held around I/O, or just CPU operations, or a mix?
>>>>>>> I believe it's a mix of modifications and I/O, though it looks like some
>>>>>>> of the I/O cases don't necessarily wait on the lock. E.g., the AIL
>>>>>>> pushing case will trylock and defer to the next list iteration if the
>>>>>>> buffer is busy.
>>>>>>>
>>>>>> Ok. For us sleeping in io_submit() is death because we have no other thread
>>>>>> on that core to take its place.
>>>>>>
>>>>> The above is with regard to metadata I/O, whereas io_submit() is
>>>>> obviously for user I/O.
>>>> Won't io_submit() also trigger metadata I/O? Or is that all deferred to
>>>> async tasks? I don't mind them blocking each other as long as they let my
>>>> io_submit alone.
>>>>
>>> Yeah, it can trigger metadata reads, force the log (the stale buffer
>>> example) or push the AIL (wait on log space). Metadata changes made
>>> directly via your I/O request are logged/committed via transactions,
>>> which are generally processed asynchronously from that point on.
>>>
>>>>> io_submit() can probably block in a variety of
>>>>> places afaict... it might have to read in the inode extent map, allocate
>>>>> blocks, take inode/ag locks, reserve log space for transactions, etc.
>>>> Any chance of changing all that to be asynchronous? Doesn't sound too hard,
>>>> if somebody else has to do it.
>>>>
>>> I'm not following... if the fs needs to read in the inode extent map to
>>> prepare for an allocation, what else can the thread do but wait? Are you
>>> suggesting the request kick off whatever the blocking action happens to
>>> be asynchronously and return with an error such that the request can be
>>> retried later?
>> Not quite, it should be invisible to the caller.
>>
>> That is, the code called by io_submit() (file_operations::write_iter, it
>> seems to be called today) can kick off this operation and have it continue
>> from where it left off.
>>
> Isn't that generally what happens today?
You tell me. According to $subject, apparently not enough. Maybe we're
triggering it more often, or we suffer more when it does trigger (the
latter probably more likely).
> We submit an I/O which is
> asynchronous in nature and wait on a completion, which causes the cpu to
> schedule and execute another task until the completion is set by I/O
> completion (via an async callback). At that point, the issuing thread
> continues where it left off. I suspect I'm missing something... can you
> elaborate on what you'd do differently here (and how it helps)?
Just apply the same technique everywhere: convert locks to trylock +
schedule a continuation on failure.
>
>> Seastar (the async user framework which we use to drive xfs) makes writing
>> code like this easy, using continuations; but of course from ordinary
>> threaded code it can be quite hard.
>>
>> btw, there was an attempt to make ext[34] async using this method, but I
>> think it was ripped out. Yes, the mortal remains can still be seen with
>> 'git grep EIOCBQUEUED'.
>>
>>>>> It sounds to me that first and foremost you want to make sure you don't
>>>>> have however many parallel operations you typically have running
>>>>> contending on the same inodes or AGs. Hint: creating files under
>>>>> separate subdirectories is a quick and easy way to allocate inodes under
>>>>> separate AGs (the agno is encoded into the upper bits of the inode
>>>>> number).
>>>> Unfortunately our directory layout cannot be changed. And doesn't this
>>>> require having agcount == O(number of active files)? That is easily in the
>>>> thousands.
>>>>
>>> I think Glauber's O(nr_cpus) comment is probably the more likely
>>> ballpark, but really it's something you'll probably just need to test to
>>> see how far you need to go to avoid AG contention.
>>>
>>> I'm primarily throwing the subdir thing out there for testing purposes.
>>> It's just an easy way to create inodes in a bunch of separate AGs so you
>>> can determine whether/how much it really helps with modified AG counts.
>>> I don't know enough about your application design to really comment on
>>> that...
>> We have O(cpus) shards that operate independently. Each shard writes 32MB
>> commitlog files (that are pre-truncated to 32MB to allow concurrent writes
>> without blocking); the files are then flushed and closed, and later removed.
>> In parallel there are sequential writes and reads of large files using 128kB
>> buffers), as well as random reads. Files are immutable (append-only), and
>> if a file is being written, it is not concurrently read. In general files
>> are not shared across shards. All I/O is async and O_DIRECT. open(),
>> truncate(), fdatasync(), and friends are called from a helper thread.
>>
>> As far as I can tell it should a very friendly load for XFS and SSDs.
>>
>>>>> Reducing the frequency of block allocation/frees might also be
>>>>> another help (e.g., preallocate and reuse files,
>>>> Isn't that discouraged for SSDs?
>>>>
>>> Perhaps, if you're referring to the fact that the blocks are never freed
>>> and thus never discarded..? Are you running fstrim?
>> mount -o discard. And yes, overwrites are supposedly more expensive than
>> trim old data + allocate new data, but maybe if you compare it with the work
>> XFS has to do, perhaps the tradeoff is bad.
>>
> Ok, my understanding is that '-o discard' is not recommended in favor of
> periodic fstrim for performance reasons, but that may or may not still
> be the case.
I understand that most SSDs have queued trim these days, but maybe I'm
optimistic.
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-01 16:08 ` Avi Kivity
@ 2015-12-01 16:29 ` Brian Foster
2015-12-01 17:09 ` Avi Kivity
0 siblings, 1 reply; 58+ messages in thread
From: Brian Foster @ 2015-12-01 16:29 UTC (permalink / raw)
To: Avi Kivity; +Cc: Glauber Costa, xfs
On Tue, Dec 01, 2015 at 06:08:51PM +0200, Avi Kivity wrote:
>
>
> On 12/01/2015 06:01 PM, Brian Foster wrote:
> >On Tue, Dec 01, 2015 at 05:22:38PM +0200, Avi Kivity wrote:
> >>
> >>On 12/01/2015 04:56 PM, Brian Foster wrote:
> >>>On Tue, Dec 01, 2015 at 03:58:28PM +0200, Avi Kivity wrote:
> >>>>On 12/01/2015 03:11 PM, Brian Foster wrote:
> >>>>>On Tue, Dec 01, 2015 at 11:08:47AM +0200, Avi Kivity wrote:
> >>>>>>On 11/30/2015 06:14 PM, Brian Foster wrote:
> >>>>>>>On Mon, Nov 30, 2015 at 04:29:13PM +0200, Avi Kivity wrote:
> >>>>>>>>On 11/30/2015 04:10 PM, Brian Foster wrote:
> >>>>>...
> >>>>>>>The agsize/agcount mkfs-time heuristics change depending on the type of
> >>>>>>>storage. A single AG can be up to 1TB and if the fs is not considered
> >>>>>>>"multidisk" (e.g., no stripe unit/width is defined), 4 AGs is the
> >>>>>>>default up to 4TB. If a stripe unit is set, the agsize/agcount is
> >>>>>>>adjusted depending on the size of the overall volume (see
> >>>>>>>xfsprogs-dev/mkfs/xfs_mkfs.c:calc_default_ag_geometry() for details).
> >>>>>>We'll experiment with this. Surely it depends on more than the amount of
> >>>>>>storage? If you have a high op rate you'll be more likely to excite
> >>>>>>contention, no?
> >>>>>>
> >>>>>Sure. The absolute optimal configuration for your workload probably
> >>>>>depends on more than storage size, but mkfs doesn't have that
> >>>>>information. In general, it tries to use the most reasonable
> >>>>>configuration based on the storage and expected workload. If you want to
> >>>>>tweak it beyond that, indeed, the best bet is to experiment with what
> >>>>>works.
> >>>>We will do that.
> >>>>
> >>>>>>>>Are those locks held around I/O, or just CPU operations, or a mix?
> >>>>>>>I believe it's a mix of modifications and I/O, though it looks like some
> >>>>>>>of the I/O cases don't necessarily wait on the lock. E.g., the AIL
> >>>>>>>pushing case will trylock and defer to the next list iteration if the
> >>>>>>>buffer is busy.
> >>>>>>>
> >>>>>>Ok. For us sleeping in io_submit() is death because we have no other thread
> >>>>>>on that core to take its place.
> >>>>>>
> >>>>>The above is with regard to metadata I/O, whereas io_submit() is
> >>>>>obviously for user I/O.
> >>>>Won't io_submit() also trigger metadata I/O? Or is that all deferred to
> >>>>async tasks? I don't mind them blocking each other as long as they let my
> >>>>io_submit alone.
> >>>>
> >>>Yeah, it can trigger metadata reads, force the log (the stale buffer
> >>>example) or push the AIL (wait on log space). Metadata changes made
> >>>directly via your I/O request are logged/committed via transactions,
> >>>which are generally processed asynchronously from that point on.
> >>>
> >>>>> io_submit() can probably block in a variety of
> >>>>>places afaict... it might have to read in the inode extent map, allocate
> >>>>>blocks, take inode/ag locks, reserve log space for transactions, etc.
> >>>>Any chance of changing all that to be asynchronous? Doesn't sound too hard,
> >>>>if somebody else has to do it.
> >>>>
> >>>I'm not following... if the fs needs to read in the inode extent map to
> >>>prepare for an allocation, what else can the thread do but wait? Are you
> >>>suggesting the request kick off whatever the blocking action happens to
> >>>be asynchronously and return with an error such that the request can be
> >>>retried later?
> >>Not quite, it should be invisible to the caller.
> >>
> >>That is, the code called by io_submit() (file_operations::write_iter, it
> >>seems to be called today) can kick off this operation and have it continue
> >>from where it left off.
> >>
> >Isn't that generally what happens today?
>
> You tell me. According to $subject, apparently not enough. Maybe we're
> triggering it more often, or we suffer more when it does trigger (the latter
> probably more likely).
>
The original mail describes looking at the sched:sched_switch tracepoint
which on a quick look, appears to fire whenever a cpu context switch
occurs. This likely triggers any time we wait on an I/O or a contended
lock (among other situations I'm sure), and it signifies that something
else is going to execute in our place until this thread can make
progress.
> > We submit an I/O which is
> >asynchronous in nature and wait on a completion, which causes the cpu to
> >schedule and execute another task until the completion is set by I/O
> >completion (via an async callback). At that point, the issuing thread
> >continues where it left off. I suspect I'm missing something... can you
> >elaborate on what you'd do differently here (and how it helps)?
>
> Just apply the same technique everywhere: convert locks to trylock +
> schedule a continuation on failure.
>
I'm certainly not an expert on the kernel scheduling, locking and
serialization mechanisms, but my understanding is that most things
outside of spin locks are reschedule points. For example, the
wait_for_completion() calls XFS uses to wait on I/O boil down to
schedule_timeout() calls. Buffer locks are implemented as semaphores and
down() can end up in the same place.
Brian
> >
> >>Seastar (the async user framework which we use to drive xfs) makes writing
> >>code like this easy, using continuations; but of course from ordinary
> >>threaded code it can be quite hard.
> >>
> >>btw, there was an attempt to make ext[34] async using this method, but I
> >>think it was ripped out. Yes, the mortal remains can still be seen with
> >>'git grep EIOCBQUEUED'.
> >>
> >>>>>It sounds to me that first and foremost you want to make sure you don't
> >>>>>have however many parallel operations you typically have running
> >>>>>contending on the same inodes or AGs. Hint: creating files under
> >>>>>separate subdirectories is a quick and easy way to allocate inodes under
> >>>>>separate AGs (the agno is encoded into the upper bits of the inode
> >>>>>number).
> >>>>Unfortunately our directory layout cannot be changed. And doesn't this
> >>>>require having agcount == O(number of active files)? That is easily in the
> >>>>thousands.
> >>>>
> >>>I think Glauber's O(nr_cpus) comment is probably the more likely
> >>>ballpark, but really it's something you'll probably just need to test to
> >>>see how far you need to go to avoid AG contention.
> >>>
> >>>I'm primarily throwing the subdir thing out there for testing purposes.
> >>>It's just an easy way to create inodes in a bunch of separate AGs so you
> >>>can determine whether/how much it really helps with modified AG counts.
> >>>I don't know enough about your application design to really comment on
> >>>that...
> >>We have O(cpus) shards that operate independently. Each shard writes 32MB
> >>commitlog files (that are pre-truncated to 32MB to allow concurrent writes
> >>without blocking); the files are then flushed and closed, and later removed.
> >>In parallel there are sequential writes and reads of large files using 128kB
> >>buffers), as well as random reads. Files are immutable (append-only), and
> >>if a file is being written, it is not concurrently read. In general files
> >>are not shared across shards. All I/O is async and O_DIRECT. open(),
> >>truncate(), fdatasync(), and friends are called from a helper thread.
> >>
> >>As far as I can tell it should a very friendly load for XFS and SSDs.
> >>
> >>>>> Reducing the frequency of block allocation/frees might also be
> >>>>>another help (e.g., preallocate and reuse files,
> >>>>Isn't that discouraged for SSDs?
> >>>>
> >>>Perhaps, if you're referring to the fact that the blocks are never freed
> >>>and thus never discarded..? Are you running fstrim?
> >>mount -o discard. And yes, overwrites are supposedly more expensive than
> >>trim old data + allocate new data, but maybe if you compare it with the work
> >>XFS has to do, perhaps the tradeoff is bad.
> >>
> >Ok, my understanding is that '-o discard' is not recommended in favor of
> >periodic fstrim for performance reasons, but that may or may not still
> >be the case.
>
> I understand that most SSDs have queued trim these days, but maybe I'm
> optimistic.
>
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-01 16:29 ` Brian Foster
@ 2015-12-01 17:09 ` Avi Kivity
2015-12-01 18:03 ` Carlos Maiolino
2015-12-01 18:51 ` Brian Foster
0 siblings, 2 replies; 58+ messages in thread
From: Avi Kivity @ 2015-12-01 17:09 UTC (permalink / raw)
To: Brian Foster; +Cc: Glauber Costa, xfs
On 12/01/2015 06:29 PM, Brian Foster wrote:
> On Tue, Dec 01, 2015 at 06:08:51PM +0200, Avi Kivity wrote:
>>
>> On 12/01/2015 06:01 PM, Brian Foster wrote:
>>> On Tue, Dec 01, 2015 at 05:22:38PM +0200, Avi Kivity wrote:
>>>> On 12/01/2015 04:56 PM, Brian Foster wrote:
>>>>> On Tue, Dec 01, 2015 at 03:58:28PM +0200, Avi Kivity wrote:
>>>>>> On 12/01/2015 03:11 PM, Brian Foster wrote:
>>>>>>> On Tue, Dec 01, 2015 at 11:08:47AM +0200, Avi Kivity wrote:
>>>>>>>> On 11/30/2015 06:14 PM, Brian Foster wrote:
>>>>>>>>> On Mon, Nov 30, 2015 at 04:29:13PM +0200, Avi Kivity wrote:
>>>>>>>>>> On 11/30/2015 04:10 PM, Brian Foster wrote:
>>>>>>> ...
>>>>>>>>> The agsize/agcount mkfs-time heuristics change depending on the type of
>>>>>>>>> storage. A single AG can be up to 1TB and if the fs is not considered
>>>>>>>>> "multidisk" (e.g., no stripe unit/width is defined), 4 AGs is the
>>>>>>>>> default up to 4TB. If a stripe unit is set, the agsize/agcount is
>>>>>>>>> adjusted depending on the size of the overall volume (see
>>>>>>>>> xfsprogs-dev/mkfs/xfs_mkfs.c:calc_default_ag_geometry() for details).
>>>>>>>> We'll experiment with this. Surely it depends on more than the amount of
>>>>>>>> storage? If you have a high op rate you'll be more likely to excite
>>>>>>>> contention, no?
>>>>>>>>
>>>>>>> Sure. The absolute optimal configuration for your workload probably
>>>>>>> depends on more than storage size, but mkfs doesn't have that
>>>>>>> information. In general, it tries to use the most reasonable
>>>>>>> configuration based on the storage and expected workload. If you want to
>>>>>>> tweak it beyond that, indeed, the best bet is to experiment with what
>>>>>>> works.
>>>>>> We will do that.
>>>>>>
>>>>>>>>>> Are those locks held around I/O, or just CPU operations, or a mix?
>>>>>>>>> I believe it's a mix of modifications and I/O, though it looks like some
>>>>>>>>> of the I/O cases don't necessarily wait on the lock. E.g., the AIL
>>>>>>>>> pushing case will trylock and defer to the next list iteration if the
>>>>>>>>> buffer is busy.
>>>>>>>>>
>>>>>>>> Ok. For us sleeping in io_submit() is death because we have no other thread
>>>>>>>> on that core to take its place.
>>>>>>>>
>>>>>>> The above is with regard to metadata I/O, whereas io_submit() is
>>>>>>> obviously for user I/O.
>>>>>> Won't io_submit() also trigger metadata I/O? Or is that all deferred to
>>>>>> async tasks? I don't mind them blocking each other as long as they let my
>>>>>> io_submit alone.
>>>>>>
>>>>> Yeah, it can trigger metadata reads, force the log (the stale buffer
>>>>> example) or push the AIL (wait on log space). Metadata changes made
>>>>> directly via your I/O request are logged/committed via transactions,
>>>>> which are generally processed asynchronously from that point on.
>>>>>
>>>>>>> io_submit() can probably block in a variety of
>>>>>>> places afaict... it might have to read in the inode extent map, allocate
>>>>>>> blocks, take inode/ag locks, reserve log space for transactions, etc.
>>>>>> Any chance of changing all that to be asynchronous? Doesn't sound too hard,
>>>>>> if somebody else has to do it.
>>>>>>
>>>>> I'm not following... if the fs needs to read in the inode extent map to
>>>>> prepare for an allocation, what else can the thread do but wait? Are you
>>>>> suggesting the request kick off whatever the blocking action happens to
>>>>> be asynchronously and return with an error such that the request can be
>>>>> retried later?
>>>> Not quite, it should be invisible to the caller.
>>>>
>>>> That is, the code called by io_submit() (file_operations::write_iter, it
>>>> seems to be called today) can kick off this operation and have it continue
>>> >from where it left off.
>>> Isn't that generally what happens today?
>> You tell me. According to $subject, apparently not enough. Maybe we're
>> triggering it more often, or we suffer more when it does trigger (the latter
>> probably more likely).
>>
> The original mail describes looking at the sched:sched_switch tracepoint
> which on a quick look, appears to fire whenever a cpu context switch
> occurs. This likely triggers any time we wait on an I/O or a contended
> lock (among other situations I'm sure), and it signifies that something
> else is going to execute in our place until this thread can make
> progress.
For us, nothing else can execute in our place, we usually have exactly
one thread per logical core. So we are heavily dependent on io_submit
not sleeping.
The case of a contended lock is, to me, less worrying. It can be
reduced by using more allocation groups, which is apparently the shared
resource under contention.
The case of waiting for I/O is much more worrying, because I/O latency
are much higher. But it seems like most of the DIO path does not
trigger locking around I/O (and we are careful to avoid the ones that
do, like writing beyond eof).
(sorry for repeating myself, I have the feeling we are talking past each
other and want to be on the same page)
>
>>> We submit an I/O which is
>>> asynchronous in nature and wait on a completion, which causes the cpu to
>>> schedule and execute another task until the completion is set by I/O
>>> completion (via an async callback). At that point, the issuing thread
>>> continues where it left off. I suspect I'm missing something... can you
>>> elaborate on what you'd do differently here (and how it helps)?
>> Just apply the same technique everywhere: convert locks to trylock +
>> schedule a continuation on failure.
>>
> I'm certainly not an expert on the kernel scheduling, locking and
> serialization mechanisms, but my understanding is that most things
> outside of spin locks are reschedule points. For example, the
> wait_for_completion() calls XFS uses to wait on I/O boil down to
> schedule_timeout() calls. Buffer locks are implemented as semaphores and
> down() can end up in the same place.
But, for the most part, XFS seems to be able to avoid sleeping. The
call to __blockdev_direct_IO only launches the I/O, so any locking is
only around cpu operations and, unless there is contention, won't cause
us to sleep in io_submit().
Trying to follow the code, it looks like xfs_get_blocks_direct (and
__blockdev_direct_IO's get_block parameter in general) is synchronous,
so we're just lucky to have everything in cache. If it isn't, we block
right there. I really hope I'm misreading this and some other magic is
happening elsewhere instead of this.
> Brian
>
>>>> Seastar (the async user framework which we use to drive xfs) makes writing
>>>> code like this easy, using continuations; but of course from ordinary
>>>> threaded code it can be quite hard.
>>>>
>>>> btw, there was an attempt to make ext[34] async using this method, but I
>>>> think it was ripped out. Yes, the mortal remains can still be seen with
>>>> 'git grep EIOCBQUEUED'.
>>>>
>>>>>>> It sounds to me that first and foremost you want to make sure you don't
>>>>>>> have however many parallel operations you typically have running
>>>>>>> contending on the same inodes or AGs. Hint: creating files under
>>>>>>> separate subdirectories is a quick and easy way to allocate inodes under
>>>>>>> separate AGs (the agno is encoded into the upper bits of the inode
>>>>>>> number).
>>>>>> Unfortunately our directory layout cannot be changed. And doesn't this
>>>>>> require having agcount == O(number of active files)? That is easily in the
>>>>>> thousands.
>>>>>>
>>>>> I think Glauber's O(nr_cpus) comment is probably the more likely
>>>>> ballpark, but really it's something you'll probably just need to test to
>>>>> see how far you need to go to avoid AG contention.
>>>>>
>>>>> I'm primarily throwing the subdir thing out there for testing purposes.
>>>>> It's just an easy way to create inodes in a bunch of separate AGs so you
>>>>> can determine whether/how much it really helps with modified AG counts.
>>>>> I don't know enough about your application design to really comment on
>>>>> that...
>>>> We have O(cpus) shards that operate independently. Each shard writes 32MB
>>>> commitlog files (that are pre-truncated to 32MB to allow concurrent writes
>>>> without blocking); the files are then flushed and closed, and later removed.
>>>> In parallel there are sequential writes and reads of large files using 128kB
>>>> buffers), as well as random reads. Files are immutable (append-only), and
>>>> if a file is being written, it is not concurrently read. In general files
>>>> are not shared across shards. All I/O is async and O_DIRECT. open(),
>>>> truncate(), fdatasync(), and friends are called from a helper thread.
>>>>
>>>> As far as I can tell it should a very friendly load for XFS and SSDs.
>>>>
>>>>>>> Reducing the frequency of block allocation/frees might also be
>>>>>>> another help (e.g., preallocate and reuse files,
>>>>>> Isn't that discouraged for SSDs?
>>>>>>
>>>>> Perhaps, if you're referring to the fact that the blocks are never freed
>>>>> and thus never discarded..? Are you running fstrim?
>>>> mount -o discard. And yes, overwrites are supposedly more expensive than
>>>> trim old data + allocate new data, but maybe if you compare it with the work
>>>> XFS has to do, perhaps the tradeoff is bad.
>>>>
>>> Ok, my understanding is that '-o discard' is not recommended in favor of
>>> periodic fstrim for performance reasons, but that may or may not still
>>> be the case.
>> I understand that most SSDs have queued trim these days, but maybe I'm
>> optimistic.
>>
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-01 17:09 ` Avi Kivity
@ 2015-12-01 18:03 ` Carlos Maiolino
2015-12-01 19:07 ` Avi Kivity
2015-12-01 18:51 ` Brian Foster
1 sibling, 1 reply; 58+ messages in thread
From: Carlos Maiolino @ 2015-12-01 18:03 UTC (permalink / raw)
To: Avi Kivity; +Cc: Glauber Costa, xfs
Hi Avi,
> >else is going to execute in our place until this thread can make
> >progress.
>
> For us, nothing else can execute in our place, we usually have exactly one
> thread per logical core. So we are heavily dependent on io_submit not
> sleeping.
>
> The case of a contended lock is, to me, less worrying. It can be reduced by
> using more allocation groups, which is apparently the shared resource under
> contention.
>
I apologize if I misread your previous comments, but, IIRC you said you can't
change the directory structure your application is using, and IIRC your
application does not spread files across several directories.
XFS spread files across the allocation groups, based on the directory these
files are created, trying to keep files as close as possible from their
metadata.
Directories are spreaded across the AGs in a 'round-robin' way, each
new directory, will be created in the next allocation group, and, xfs will try
to allocate the files in the same AG as its parent directory. (Take a look at
the 'rotorstep' sysctl option for xfs).
So, unless you have the files distributed across enough directories, increasing
the number of allocation groups may not change the lock contention you're
facing in this case.
I really don't remember if it has been mentioned already, but if not, it might
be worth to take this point in consideration.
anyway, just my 0.02
> The case of waiting for I/O is much more worrying, because I/O latency are
> much higher. But it seems like most of the DIO path does not trigger
> locking around I/O (and we are careful to avoid the ones that do, like
> writing beyond eof).
>
> (sorry for repeating myself, I have the feeling we are talking past each
> other and want to be on the same page)
>
> >
> >>> We submit an I/O which is
> >>>asynchronous in nature and wait on a completion, which causes the cpu to
> >>>schedule and execute another task until the completion is set by I/O
> >>>completion (via an async callback). At that point, the issuing thread
> >>>continues where it left off. I suspect I'm missing something... can you
> >>>elaborate on what you'd do differently here (and how it helps)?
> >>Just apply the same technique everywhere: convert locks to trylock +
> >>schedule a continuation on failure.
> >>
> >I'm certainly not an expert on the kernel scheduling, locking and
> >serialization mechanisms, but my understanding is that most things
> >outside of spin locks are reschedule points. For example, the
> >wait_for_completion() calls XFS uses to wait on I/O boil down to
> >schedule_timeout() calls. Buffer locks are implemented as semaphores and
> >down() can end up in the same place.
>
> But, for the most part, XFS seems to be able to avoid sleeping. The call to
> __blockdev_direct_IO only launches the I/O, so any locking is only around
> cpu operations and, unless there is contention, won't cause us to sleep in
> io_submit().
>
> Trying to follow the code, it looks like xfs_get_blocks_direct (and
> __blockdev_direct_IO's get_block parameter in general) is synchronous, so
> we're just lucky to have everything in cache. If it isn't, we block right
> there. I really hope I'm misreading this and some other magic is happening
> elsewhere instead of this.
>
> >Brian
> >
> >>>>Seastar (the async user framework which we use to drive xfs) makes writing
> >>>>code like this easy, using continuations; but of course from ordinary
> >>>>threaded code it can be quite hard.
> >>>>
> >>>>btw, there was an attempt to make ext[34] async using this method, but I
> >>>>think it was ripped out. Yes, the mortal remains can still be seen with
> >>>>'git grep EIOCBQUEUED'.
> >>>>
> >>>>>>>It sounds to me that first and foremost you want to make sure you don't
> >>>>>>>have however many parallel operations you typically have running
> >>>>>>>contending on the same inodes or AGs. Hint: creating files under
> >>>>>>>separate subdirectories is a quick and easy way to allocate inodes under
> >>>>>>>separate AGs (the agno is encoded into the upper bits of the inode
> >>>>>>>number).
> >>>>>>Unfortunately our directory layout cannot be changed. And doesn't this
> >>>>>>require having agcount == O(number of active files)? That is easily in the
> >>>>>>thousands.
> >>>>>>
> >>>>>I think Glauber's O(nr_cpus) comment is probably the more likely
> >>>>>ballpark, but really it's something you'll probably just need to test to
> >>>>>see how far you need to go to avoid AG contention.
> >>>>>
> >>>>>I'm primarily throwing the subdir thing out there for testing purposes.
> >>>>>It's just an easy way to create inodes in a bunch of separate AGs so you
> >>>>>can determine whether/how much it really helps with modified AG counts.
> >>>>>I don't know enough about your application design to really comment on
> >>>>>that...
> >>>>We have O(cpus) shards that operate independently. Each shard writes 32MB
> >>>>commitlog files (that are pre-truncated to 32MB to allow concurrent writes
> >>>>without blocking); the files are then flushed and closed, and later removed.
> >>>>In parallel there are sequential writes and reads of large files using 128kB
> >>>>buffers), as well as random reads. Files are immutable (append-only), and
> >>>>if a file is being written, it is not concurrently read. In general files
> >>>>are not shared across shards. All I/O is async and O_DIRECT. open(),
> >>>>truncate(), fdatasync(), and friends are called from a helper thread.
> >>>>
> >>>>As far as I can tell it should a very friendly load for XFS and SSDs.
> >>>>
> >>>>>>> Reducing the frequency of block allocation/frees might also be
> >>>>>>>another help (e.g., preallocate and reuse files,
> >>>>>>Isn't that discouraged for SSDs?
> >>>>>>
> >>>>>Perhaps, if you're referring to the fact that the blocks are never freed
> >>>>>and thus never discarded..? Are you running fstrim?
> >>>>mount -o discard. And yes, overwrites are supposedly more expensive than
> >>>>trim old data + allocate new data, but maybe if you compare it with the work
> >>>>XFS has to do, perhaps the tradeoff is bad.
> >>>>
> >>>Ok, my understanding is that '-o discard' is not recommended in favor of
> >>>periodic fstrim for performance reasons, but that may or may not still
> >>>be the case.
> >>I understand that most SSDs have queued trim these days, but maybe I'm
> >>optimistic.
> >>
>
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
--
Carlos
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-01 17:09 ` Avi Kivity
2015-12-01 18:03 ` Carlos Maiolino
@ 2015-12-01 18:51 ` Brian Foster
2015-12-01 19:07 ` Glauber Costa
2015-12-01 19:26 ` Avi Kivity
1 sibling, 2 replies; 58+ messages in thread
From: Brian Foster @ 2015-12-01 18:51 UTC (permalink / raw)
To: Avi Kivity; +Cc: Glauber Costa, xfs
On Tue, Dec 01, 2015 at 07:09:29PM +0200, Avi Kivity wrote:
>
>
> On 12/01/2015 06:29 PM, Brian Foster wrote:
> >On Tue, Dec 01, 2015 at 06:08:51PM +0200, Avi Kivity wrote:
> >>
> >>On 12/01/2015 06:01 PM, Brian Foster wrote:
> >>>On Tue, Dec 01, 2015 at 05:22:38PM +0200, Avi Kivity wrote:
> >>>>On 12/01/2015 04:56 PM, Brian Foster wrote:
> >>>>>On Tue, Dec 01, 2015 at 03:58:28PM +0200, Avi Kivity wrote:
> >>>>>>On 12/01/2015 03:11 PM, Brian Foster wrote:
> >>>>>>>On Tue, Dec 01, 2015 at 11:08:47AM +0200, Avi Kivity wrote:
> >>>>>>>>On 11/30/2015 06:14 PM, Brian Foster wrote:
> >>>>>>>>>On Mon, Nov 30, 2015 at 04:29:13PM +0200, Avi Kivity wrote:
> >>>>>>>>>>On 11/30/2015 04:10 PM, Brian Foster wrote:
> >>>>>>>...
...
> >>>>>>Won't io_submit() also trigger metadata I/O? Or is that all deferred to
> >>>>>>async tasks? I don't mind them blocking each other as long as they let my
> >>>>>>io_submit alone.
> >>>>>>
> >>>>>Yeah, it can trigger metadata reads, force the log (the stale buffer
> >>>>>example) or push the AIL (wait on log space). Metadata changes made
> >>>>>directly via your I/O request are logged/committed via transactions,
> >>>>>which are generally processed asynchronously from that point on.
> >>>>>
> >>>>>>> io_submit() can probably block in a variety of
> >>>>>>>places afaict... it might have to read in the inode extent map, allocate
> >>>>>>>blocks, take inode/ag locks, reserve log space for transactions, etc.
> >>>>>>Any chance of changing all that to be asynchronous? Doesn't sound too hard,
> >>>>>>if somebody else has to do it.
> >>>>>>
> >>>>>I'm not following... if the fs needs to read in the inode extent map to
> >>>>>prepare for an allocation, what else can the thread do but wait? Are you
> >>>>>suggesting the request kick off whatever the blocking action happens to
> >>>>>be asynchronously and return with an error such that the request can be
> >>>>>retried later?
> >>>>Not quite, it should be invisible to the caller.
> >>>>
> >>>>That is, the code called by io_submit() (file_operations::write_iter, it
> >>>>seems to be called today) can kick off this operation and have it continue
> >>>>from where it left off.
> >>>Isn't that generally what happens today?
> >>You tell me. According to $subject, apparently not enough. Maybe we're
> >>triggering it more often, or we suffer more when it does trigger (the latter
> >>probably more likely).
> >>
> >The original mail describes looking at the sched:sched_switch tracepoint
> >which on a quick look, appears to fire whenever a cpu context switch
> >occurs. This likely triggers any time we wait on an I/O or a contended
> >lock (among other situations I'm sure), and it signifies that something
> >else is going to execute in our place until this thread can make
> >progress.
>
> For us, nothing else can execute in our place, we usually have exactly one
> thread per logical core. So we are heavily dependent on io_submit not
> sleeping.
>
Yes, this "coroutine model" makes more sense to me from the application
perspective. I'm just trying to understand what you're after from the
kernel perspective.
> The case of a contended lock is, to me, less worrying. It can be reduced by
> using more allocation groups, which is apparently the shared resource under
> contention.
>
Yep.
> The case of waiting for I/O is much more worrying, because I/O latency are
> much higher. But it seems like most of the DIO path does not trigger
> locking around I/O (and we are careful to avoid the ones that do, like
> writing beyond eof).
>
> (sorry for repeating myself, I have the feeling we are talking past each
> other and want to be on the same page)
>
Yeah, my point is just that just because the thread blocked on I/O,
doesn't mean the cpu can't carry on with some useful work for another
task.
> >
> >>> We submit an I/O which is
> >>>asynchronous in nature and wait on a completion, which causes the cpu to
> >>>schedule and execute another task until the completion is set by I/O
> >>>completion (via an async callback). At that point, the issuing thread
> >>>continues where it left off. I suspect I'm missing something... can you
> >>>elaborate on what you'd do differently here (and how it helps)?
> >>Just apply the same technique everywhere: convert locks to trylock +
> >>schedule a continuation on failure.
> >>
> >I'm certainly not an expert on the kernel scheduling, locking and
> >serialization mechanisms, but my understanding is that most things
> >outside of spin locks are reschedule points. For example, the
> >wait_for_completion() calls XFS uses to wait on I/O boil down to
> >schedule_timeout() calls. Buffer locks are implemented as semaphores and
> >down() can end up in the same place.
>
> But, for the most part, XFS seems to be able to avoid sleeping. The call to
> __blockdev_direct_IO only launches the I/O, so any locking is only around
> cpu operations and, unless there is contention, won't cause us to sleep in
> io_submit().
>
> Trying to follow the code, it looks like xfs_get_blocks_direct (and
> __blockdev_direct_IO's get_block parameter in general) is synchronous, so
> we're just lucky to have everything in cache. If it isn't, we block right
> there. I really hope I'm misreading this and some other magic is happening
> elsewhere instead of this.
>
Nope, it's synchronous from a code perspective. The
xfs_bmapi_read()->xfs_iread_extents() path could have to read in the
inode bmap metadata if it hasn't been done already. Note that this
should only happen once as everything is stored in-core, so in most
cases this is skipped. It's also possible extents are read in via some
other path/operation on the inode before an async I/O happens to be
submitted (e.g., see some of the other xfs_bmapi_read() callers).
Either way, the extents have to be read in at some point and I'd expect
that cpu to schedule onto some other task while that thread waits on I/O
to complete (read-ahead could also be a factor here, but I haven't
really dug into how that is triggered for buffers).
Brian
> >Brian
> >
> >>>>Seastar (the async user framework which we use to drive xfs) makes writing
> >>>>code like this easy, using continuations; but of course from ordinary
> >>>>threaded code it can be quite hard.
> >>>>
> >>>>btw, there was an attempt to make ext[34] async using this method, but I
> >>>>think it was ripped out. Yes, the mortal remains can still be seen with
> >>>>'git grep EIOCBQUEUED'.
> >>>>
> >>>>>>>It sounds to me that first and foremost you want to make sure you don't
> >>>>>>>have however many parallel operations you typically have running
> >>>>>>>contending on the same inodes or AGs. Hint: creating files under
> >>>>>>>separate subdirectories is a quick and easy way to allocate inodes under
> >>>>>>>separate AGs (the agno is encoded into the upper bits of the inode
> >>>>>>>number).
> >>>>>>Unfortunately our directory layout cannot be changed. And doesn't this
> >>>>>>require having agcount == O(number of active files)? That is easily in the
> >>>>>>thousands.
> >>>>>>
> >>>>>I think Glauber's O(nr_cpus) comment is probably the more likely
> >>>>>ballpark, but really it's something you'll probably just need to test to
> >>>>>see how far you need to go to avoid AG contention.
> >>>>>
> >>>>>I'm primarily throwing the subdir thing out there for testing purposes.
> >>>>>It's just an easy way to create inodes in a bunch of separate AGs so you
> >>>>>can determine whether/how much it really helps with modified AG counts.
> >>>>>I don't know enough about your application design to really comment on
> >>>>>that...
> >>>>We have O(cpus) shards that operate independently. Each shard writes 32MB
> >>>>commitlog files (that are pre-truncated to 32MB to allow concurrent writes
> >>>>without blocking); the files are then flushed and closed, and later removed.
> >>>>In parallel there are sequential writes and reads of large files using 128kB
> >>>>buffers), as well as random reads. Files are immutable (append-only), and
> >>>>if a file is being written, it is not concurrently read. In general files
> >>>>are not shared across shards. All I/O is async and O_DIRECT. open(),
> >>>>truncate(), fdatasync(), and friends are called from a helper thread.
> >>>>
> >>>>As far as I can tell it should a very friendly load for XFS and SSDs.
> >>>>
> >>>>>>> Reducing the frequency of block allocation/frees might also be
> >>>>>>>another help (e.g., preallocate and reuse files,
> >>>>>>Isn't that discouraged for SSDs?
> >>>>>>
> >>>>>Perhaps, if you're referring to the fact that the blocks are never freed
> >>>>>and thus never discarded..? Are you running fstrim?
> >>>>mount -o discard. And yes, overwrites are supposedly more expensive than
> >>>>trim old data + allocate new data, but maybe if you compare it with the work
> >>>>XFS has to do, perhaps the tradeoff is bad.
> >>>>
> >>>Ok, my understanding is that '-o discard' is not recommended in favor of
> >>>periodic fstrim for performance reasons, but that may or may not still
> >>>be the case.
> >>I understand that most SSDs have queued trim these days, but maybe I'm
> >>optimistic.
> >>
>
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-01 18:03 ` Carlos Maiolino
@ 2015-12-01 19:07 ` Avi Kivity
2015-12-01 21:19 ` Dave Chinner
0 siblings, 1 reply; 58+ messages in thread
From: Avi Kivity @ 2015-12-01 19:07 UTC (permalink / raw)
To: Glauber Costa, xfs
On 12/01/2015 08:03 PM, Carlos Maiolino wrote:
> Hi Avi,
>
>>> else is going to execute in our place until this thread can make
>>> progress.
>> For us, nothing else can execute in our place, we usually have exactly one
>> thread per logical core. So we are heavily dependent on io_submit not
>> sleeping.
>>
>> The case of a contended lock is, to me, less worrying. It can be reduced by
>> using more allocation groups, which is apparently the shared resource under
>> contention.
>>
> I apologize if I misread your previous comments, but, IIRC you said you can't
> change the directory structure your application is using, and IIRC your
> application does not spread files across several directories.
I miswrote somewhat: the application writes data files and commitlog
files. The data file directory structure is fixed due to compatibility
concerns (it is not a single directory, but some workloads will see most
access on files in a single directory. The commitlog directory
structure is more relaxed, and we can split it to a directory per shard
(=cpu) or something else.
If worst comes to worst, we'll hack around this and distribute the data
files into more directories, and provide some hack for compatibility.
> XFS spread files across the allocation groups, based on the directory these
> files are created,
Idea: create the files in some subdirectory, and immediately move them
to their required location.
> trying to keep files as close as possible from their
> metadata.
This is pointless for an SSD. Perhaps XFS should randomize the ag on
nonrotational media instead.
> Directories are spreaded across the AGs in a 'round-robin' way, each
> new directory, will be created in the next allocation group, and, xfs will try
> to allocate the files in the same AG as its parent directory. (Take a look at
> the 'rotorstep' sysctl option for xfs).
>
> So, unless you have the files distributed across enough directories, increasing
> the number of allocation groups may not change the lock contention you're
> facing in this case.
>
> I really don't remember if it has been mentioned already, but if not, it might
> be worth to take this point in consideration.
Thanks. I think you should really consider randomizing the ag for SSDs,
and meanwhile, we can just use the creation-directory hack to get the
same effect, at the cost of an extra system call. So at least for this
problem, there is a solution.
> anyway, just my 0.02
>
>> The case of waiting for I/O is much more worrying, because I/O latency are
>> much higher. But it seems like most of the DIO path does not trigger
>> locking around I/O (and we are careful to avoid the ones that do, like
>> writing beyond eof).
>>
>> (sorry for repeating myself, I have the feeling we are talking past each
>> other and want to be on the same page)
>>
>>>>> We submit an I/O which is
>>>>> asynchronous in nature and wait on a completion, which causes the cpu to
>>>>> schedule and execute another task until the completion is set by I/O
>>>>> completion (via an async callback). At that point, the issuing thread
>>>>> continues where it left off. I suspect I'm missing something... can you
>>>>> elaborate on what you'd do differently here (and how it helps)?
>>>> Just apply the same technique everywhere: convert locks to trylock +
>>>> schedule a continuation on failure.
>>>>
>>> I'm certainly not an expert on the kernel scheduling, locking and
>>> serialization mechanisms, but my understanding is that most things
>>> outside of spin locks are reschedule points. For example, the
>>> wait_for_completion() calls XFS uses to wait on I/O boil down to
>>> schedule_timeout() calls. Buffer locks are implemented as semaphores and
>>> down() can end up in the same place.
>> But, for the most part, XFS seems to be able to avoid sleeping. The call to
>> __blockdev_direct_IO only launches the I/O, so any locking is only around
>> cpu operations and, unless there is contention, won't cause us to sleep in
>> io_submit().
>>
>> Trying to follow the code, it looks like xfs_get_blocks_direct (and
>> __blockdev_direct_IO's get_block parameter in general) is synchronous, so
>> we're just lucky to have everything in cache. If it isn't, we block right
>> there. I really hope I'm misreading this and some other magic is happening
>> elsewhere instead of this.
>>
>>> Brian
>>>
>>>>>> Seastar (the async user framework which we use to drive xfs) makes writing
>>>>>> code like this easy, using continuations; but of course from ordinary
>>>>>> threaded code it can be quite hard.
>>>>>>
>>>>>> btw, there was an attempt to make ext[34] async using this method, but I
>>>>>> think it was ripped out. Yes, the mortal remains can still be seen with
>>>>>> 'git grep EIOCBQUEUED'.
>>>>>>
>>>>>>>>> It sounds to me that first and foremost you want to make sure you don't
>>>>>>>>> have however many parallel operations you typically have running
>>>>>>>>> contending on the same inodes or AGs. Hint: creating files under
>>>>>>>>> separate subdirectories is a quick and easy way to allocate inodes under
>>>>>>>>> separate AGs (the agno is encoded into the upper bits of the inode
>>>>>>>>> number).
>>>>>>>> Unfortunately our directory layout cannot be changed. And doesn't this
>>>>>>>> require having agcount == O(number of active files)? That is easily in the
>>>>>>>> thousands.
>>>>>>>>
>>>>>>> I think Glauber's O(nr_cpus) comment is probably the more likely
>>>>>>> ballpark, but really it's something you'll probably just need to test to
>>>>>>> see how far you need to go to avoid AG contention.
>>>>>>>
>>>>>>> I'm primarily throwing the subdir thing out there for testing purposes.
>>>>>>> It's just an easy way to create inodes in a bunch of separate AGs so you
>>>>>>> can determine whether/how much it really helps with modified AG counts.
>>>>>>> I don't know enough about your application design to really comment on
>>>>>>> that...
>>>>>> We have O(cpus) shards that operate independently. Each shard writes 32MB
>>>>>> commitlog files (that are pre-truncated to 32MB to allow concurrent writes
>>>>>> without blocking); the files are then flushed and closed, and later removed.
>>>>>> In parallel there are sequential writes and reads of large files using 128kB
>>>>>> buffers), as well as random reads. Files are immutable (append-only), and
>>>>>> if a file is being written, it is not concurrently read. In general files
>>>>>> are not shared across shards. All I/O is async and O_DIRECT. open(),
>>>>>> truncate(), fdatasync(), and friends are called from a helper thread.
>>>>>>
>>>>>> As far as I can tell it should a very friendly load for XFS and SSDs.
>>>>>>
>>>>>>>>> Reducing the frequency of block allocation/frees might also be
>>>>>>>>> another help (e.g., preallocate and reuse files,
>>>>>>>> Isn't that discouraged for SSDs?
>>>>>>>>
>>>>>>> Perhaps, if you're referring to the fact that the blocks are never freed
>>>>>>> and thus never discarded..? Are you running fstrim?
>>>>>> mount -o discard. And yes, overwrites are supposedly more expensive than
>>>>>> trim old data + allocate new data, but maybe if you compare it with the work
>>>>>> XFS has to do, perhaps the tradeoff is bad.
>>>>>>
>>>>> Ok, my understanding is that '-o discard' is not recommended in favor of
>>>>> periodic fstrim for performance reasons, but that may or may not still
>>>>> be the case.
>>>> I understand that most SSDs have queued trim these days, but maybe I'm
>>>> optimistic.
>>>>
>> _______________________________________________
>> xfs mailing list
>> xfs@oss.sgi.com
>> http://oss.sgi.com/mailman/listinfo/xfs
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-01 18:51 ` Brian Foster
@ 2015-12-01 19:07 ` Glauber Costa
2015-12-01 19:35 ` Brian Foster
2015-12-01 19:26 ` Avi Kivity
1 sibling, 1 reply; 58+ messages in thread
From: Glauber Costa @ 2015-12-01 19:07 UTC (permalink / raw)
To: Brian Foster; +Cc: Avi Kivity, xfs
Hi Brian,
>
> Either way, the extents have to be read in at some point and I'd expect
> that cpu to schedule onto some other task while that thread waits on I/O
> to complete (read-ahead could also be a factor here, but I haven't
> really dug into how that is triggered for buffers).
>
Being a datastore, we expect to run practically alone in any box we're
at. That means that there is no other task to run. If io_submit
blocks, the system blocks. The assumption that blocking will just
yield the processor for another thread makes sense in the general case
where you assume more than one application running and/or more than
one thread within the same application.
>From our user's perspective, however, every time that happens we can't
make progress. It doesn't really matter where it blocks.
If io_submit returns without blocking, we can still push more work,
even though the kernel is still not ready to proceed. If it blocks,
we're dead.
> Brian
>
>> >Brian
>> >
>> >>>>Seastar (the async user framework which we use to drive xfs) makes writing
>> >>>>code like this easy, using continuations; but of course from ordinary
>> >>>>threaded code it can be quite hard.
>> >>>>
>> >>>>btw, there was an attempt to make ext[34] async using this method, but I
>> >>>>think it was ripped out. Yes, the mortal remains can still be seen with
>> >>>>'git grep EIOCBQUEUED'.
>> >>>>
>> >>>>>>>It sounds to me that first and foremost you want to make sure you don't
>> >>>>>>>have however many parallel operations you typically have running
>> >>>>>>>contending on the same inodes or AGs. Hint: creating files under
>> >>>>>>>separate subdirectories is a quick and easy way to allocate inodes under
>> >>>>>>>separate AGs (the agno is encoded into the upper bits of the inode
>> >>>>>>>number).
>> >>>>>>Unfortunately our directory layout cannot be changed. And doesn't this
>> >>>>>>require having agcount == O(number of active files)? That is easily in the
>> >>>>>>thousands.
>> >>>>>>
>> >>>>>I think Glauber's O(nr_cpus) comment is probably the more likely
>> >>>>>ballpark, but really it's something you'll probably just need to test to
>> >>>>>see how far you need to go to avoid AG contention.
>> >>>>>
>> >>>>>I'm primarily throwing the subdir thing out there for testing purposes.
>> >>>>>It's just an easy way to create inodes in a bunch of separate AGs so you
>> >>>>>can determine whether/how much it really helps with modified AG counts.
>> >>>>>I don't know enough about your application design to really comment on
>> >>>>>that...
>> >>>>We have O(cpus) shards that operate independently. Each shard writes 32MB
>> >>>>commitlog files (that are pre-truncated to 32MB to allow concurrent writes
>> >>>>without blocking); the files are then flushed and closed, and later removed.
>> >>>>In parallel there are sequential writes and reads of large files using 128kB
>> >>>>buffers), as well as random reads. Files are immutable (append-only), and
>> >>>>if a file is being written, it is not concurrently read. In general files
>> >>>>are not shared across shards. All I/O is async and O_DIRECT. open(),
>> >>>>truncate(), fdatasync(), and friends are called from a helper thread.
>> >>>>
>> >>>>As far as I can tell it should a very friendly load for XFS and SSDs.
>> >>>>
>> >>>>>>> Reducing the frequency of block allocation/frees might also be
>> >>>>>>>another help (e.g., preallocate and reuse files,
>> >>>>>>Isn't that discouraged for SSDs?
>> >>>>>>
>> >>>>>Perhaps, if you're referring to the fact that the blocks are never freed
>> >>>>>and thus never discarded..? Are you running fstrim?
>> >>>>mount -o discard. And yes, overwrites are supposedly more expensive than
>> >>>>trim old data + allocate new data, but maybe if you compare it with the work
>> >>>>XFS has to do, perhaps the tradeoff is bad.
>> >>>>
>> >>>Ok, my understanding is that '-o discard' is not recommended in favor of
>> >>>periodic fstrim for performance reasons, but that may or may not still
>> >>>be the case.
>> >>I understand that most SSDs have queued trim these days, but maybe I'm
>> >>optimistic.
>> >>
>>
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-01 18:51 ` Brian Foster
2015-12-01 19:07 ` Glauber Costa
@ 2015-12-01 19:26 ` Avi Kivity
2015-12-01 19:41 ` Christoph Hellwig
2015-12-02 0:13 ` Brian Foster
1 sibling, 2 replies; 58+ messages in thread
From: Avi Kivity @ 2015-12-01 19:26 UTC (permalink / raw)
To: Brian Foster; +Cc: Glauber Costa, xfs
On 12/01/2015 08:51 PM, Brian Foster wrote:
> On Tue, Dec 01, 2015 at 07:09:29PM +0200, Avi Kivity wrote:
>>
>> On 12/01/2015 06:29 PM, Brian Foster wrote:
>>> On Tue, Dec 01, 2015 at 06:08:51PM +0200, Avi Kivity wrote:
>>>> On 12/01/2015 06:01 PM, Brian Foster wrote:
>>>>> On Tue, Dec 01, 2015 at 05:22:38PM +0200, Avi Kivity wrote:
>>>>>> On 12/01/2015 04:56 PM, Brian Foster wrote:
>>>>>>> On Tue, Dec 01, 2015 at 03:58:28PM +0200, Avi Kivity wrote:
>>>>>>>> On 12/01/2015 03:11 PM, Brian Foster wrote:
>>>>>>>>> On Tue, Dec 01, 2015 at 11:08:47AM +0200, Avi Kivity wrote:
>>>>>>>>>> On 11/30/2015 06:14 PM, Brian Foster wrote:
>>>>>>>>>>> On Mon, Nov 30, 2015 at 04:29:13PM +0200, Avi Kivity wrote:
>>>>>>>>>>>> On 11/30/2015 04:10 PM, Brian Foster wrote:
>>>>>>>>> ...
> ...
>>>>>>>> Won't io_submit() also trigger metadata I/O? Or is that all deferred to
>>>>>>>> async tasks? I don't mind them blocking each other as long as they let my
>>>>>>>> io_submit alone.
>>>>>>>>
>>>>>>> Yeah, it can trigger metadata reads, force the log (the stale buffer
>>>>>>> example) or push the AIL (wait on log space). Metadata changes made
>>>>>>> directly via your I/O request are logged/committed via transactions,
>>>>>>> which are generally processed asynchronously from that point on.
>>>>>>>
>>>>>>>>> io_submit() can probably block in a variety of
>>>>>>>>> places afaict... it might have to read in the inode extent map, allocate
>>>>>>>>> blocks, take inode/ag locks, reserve log space for transactions, etc.
>>>>>>>> Any chance of changing all that to be asynchronous? Doesn't sound too hard,
>>>>>>>> if somebody else has to do it.
>>>>>>>>
>>>>>>> I'm not following... if the fs needs to read in the inode extent map to
>>>>>>> prepare for an allocation, what else can the thread do but wait? Are you
>>>>>>> suggesting the request kick off whatever the blocking action happens to
>>>>>>> be asynchronously and return with an error such that the request can be
>>>>>>> retried later?
>>>>>> Not quite, it should be invisible to the caller.
>>>>>>
>>>>>> That is, the code called by io_submit() (file_operations::write_iter, it
>>>>>> seems to be called today) can kick off this operation and have it continue
>>>>> >from where it left off.
>>>>> Isn't that generally what happens today?
>>>> You tell me. According to $subject, apparently not enough. Maybe we're
>>>> triggering it more often, or we suffer more when it does trigger (the latter
>>>> probably more likely).
>>>>
>>> The original mail describes looking at the sched:sched_switch tracepoint
>>> which on a quick look, appears to fire whenever a cpu context switch
>>> occurs. This likely triggers any time we wait on an I/O or a contended
>>> lock (among other situations I'm sure), and it signifies that something
>>> else is going to execute in our place until this thread can make
>>> progress.
>> For us, nothing else can execute in our place, we usually have exactly one
>> thread per logical core. So we are heavily dependent on io_submit not
>> sleeping.
>>
> Yes, this "coroutine model" makes more sense to me from the application
> perspective. I'm just trying to understand what you're after from the
> kernel perspective.
It's basically the same thing. To to this, we'd have get_block either
return the block's address (if it was in some metadata cache), or, if it
was not, issue an I/O that fills (part of) that cache, and as its
completion function, a continuation that reruns __blockdev_direct_IO
from the point it was stopped so it can submit the data I/O (if the
metadata cache was completely updated) or issue the next I/O aiming to
fill that metadata cache, if it was not.
Without that (and the more complicated code for the write path)
io_submit is basically unusable. Yes parts of it are asynchronous, but
if other parts of it are still synchronous, we end up requiring
thread_count > cpu_count and now we have to context switch constantly.
>
>> The case of a contended lock is, to me, less worrying. It can be reduced by
>> using more allocation groups, which is apparently the shared resource under
>> contention.
>>
> Yep.
>
>> The case of waiting for I/O is much more worrying, because I/O latency are
>> much higher. But it seems like most of the DIO path does not trigger
>> locking around I/O (and we are careful to avoid the ones that do, like
>> writing beyond eof).
>>
>> (sorry for repeating myself, I have the feeling we are talking past each
>> other and want to be on the same page)
>>
> Yeah, my point is just that just because the thread blocked on I/O,
> doesn't mean the cpu can't carry on with some useful work for another
> task.
In our case, there is no other task. We run one thread per logical
core, so if that thread gets blocked, the cpu idles.
The whole point of io_submit() is to issue an I/O and let the caller
continue processing immediately. It is the equivalent of O_NONBLOCK for
networking code. If O_NONBLOCK did block from time to time, practically
all modern network applications would see a huge performance drop.
>
>>>>> We submit an I/O which is
>>>>> asynchronous in nature and wait on a completion, which causes the cpu to
>>>>> schedule and execute another task until the completion is set by I/O
>>>>> completion (via an async callback). At that point, the issuing thread
>>>>> continues where it left off. I suspect I'm missing something... can you
>>>>> elaborate on what you'd do differently here (and how it helps)?
>>>> Just apply the same technique everywhere: convert locks to trylock +
>>>> schedule a continuation on failure.
>>>>
>>> I'm certainly not an expert on the kernel scheduling, locking and
>>> serialization mechanisms, but my understanding is that most things
>>> outside of spin locks are reschedule points. For example, the
>>> wait_for_completion() calls XFS uses to wait on I/O boil down to
>>> schedule_timeout() calls. Buffer locks are implemented as semaphores and
>>> down() can end up in the same place.
>> But, for the most part, XFS seems to be able to avoid sleeping. The call to
>> __blockdev_direct_IO only launches the I/O, so any locking is only around
>> cpu operations and, unless there is contention, won't cause us to sleep in
>> io_submit().
>>
>> Trying to follow the code, it looks like xfs_get_blocks_direct (and
>> __blockdev_direct_IO's get_block parameter in general) is synchronous, so
>> we're just lucky to have everything in cache. If it isn't, we block right
>> there. I really hope I'm misreading this and some other magic is happening
>> elsewhere instead of this.
>>
> Nope, it's synchronous from a code perspective. The
> xfs_bmapi_read()->xfs_iread_extents() path could have to read in the
> inode bmap metadata if it hasn't been done already. Note that this
> should only happen once as everything is stored in-core, so in most
> cases this is skipped. It's also possible extents are read in via some
> other path/operation on the inode before an async I/O happens to be
> submitted (e.g., see some of the other xfs_bmapi_read() callers).
Is there (could we add) some ioctl to prime this cache? We could call
it from a worker thread where we don't mind blocking during open.
What is the eviction policy for this cache? Is it simply the block
device's page cache?
What about the write path, will we see the same problems there? I would
guess the problem is less severe there if the metadata is written with
writeback policy.
>
> Either way, the extents have to be read in at some point and I'd expect
> that cpu to schedule onto some other task while that thread waits on I/O
> to complete (read-ahead could also be a factor here, but I haven't
> really dug into how that is triggered for buffers).
To provide an example, our application, which is a database, faces this
problem exact at a higher level. Data is stored in data files, and data
items' locations are stored in index files. When we read a bit of data,
we issue an index read, and pass it a continuation to be executed when
the read completes. This latter continuation parses the data and passes
it to the code that prepares it for merging with data from other data
files, and an eventual return to the user.
Having written code for over a year in this style, I've come to expect
it to be used everywhere asynchronous I/O is used, but I realize it is
fairly hard without good support from a framework that allows
continuations to be composed in a natural way.
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-01 19:07 ` Glauber Costa
@ 2015-12-01 19:35 ` Brian Foster
2015-12-01 19:45 ` Avi Kivity
0 siblings, 1 reply; 58+ messages in thread
From: Brian Foster @ 2015-12-01 19:35 UTC (permalink / raw)
To: Glauber Costa; +Cc: Avi Kivity, xfs
On Tue, Dec 01, 2015 at 02:07:41PM -0500, Glauber Costa wrote:
> Hi Brian,
>
>
> >
> > Either way, the extents have to be read in at some point and I'd expect
> > that cpu to schedule onto some other task while that thread waits on I/O
> > to complete (read-ahead could also be a factor here, but I haven't
> > really dug into how that is triggered for buffers).
> >
>
>
> Being a datastore, we expect to run practically alone in any box we're
> at. That means that there is no other task to run. If io_submit
> blocks, the system blocks. The assumption that blocking will just
> yield the processor for another thread makes sense in the general case
> where you assume more than one application running and/or more than
> one thread within the same application.
>
Hmm, well that helps me understand the concern a bit more. That said, I
still question how likely this condition is. Even if this is a
completely stripped down userspace with no other applications running,
the kernel (or even XFS) alone might have plenty of threads/work items
to execute to take care of "background" tasks for various subsystems.
Of course, we don't have all of the details of your environment so
perhaps this is not the case. Perhaps a more productive approach here
might be to find a way to detect this particular case (once you've
worked out the other AG count tunings and whatnot that you want to use)
where a thread into the fs is blocked and actually has nothing else to
do and work from there. I _think_ there is such a thing as an idle task
somewhere that might be useful to help quantify this, but I'd have to
dig around to understand it better.
That actually gives us a concrete scenario to work with, try to
reproduce and improve on. It also facilitates improvements that might be
beneficial to the general use case as opposed to tailored for this
particular use case and highly specific environment. For example, if we
find a particular sustained workload that repetitively blocks with
nothing else to do, document and characterize it for the list and I'm
sure people will come up with a variety of ideas to try and address it.
Otherwise, we're kind of just looking around for context switch points
and assuming that they will all just block with nothing else to do. For
one, I don't think that's really accurate. It's also not very productive
an approach and doesn't have any measurable benefit if it doesn't come
along with a test case or reproducible condition.
Brian
> From our user's perspective, however, every time that happens we can't
> make progress. It doesn't really matter where it blocks.
>
> If io_submit returns without blocking, we can still push more work,
> even though the kernel is still not ready to proceed. If it blocks,
> we're dead.
>
> > Brian
> >
> >> >Brian
> >> >
> >> >>>>Seastar (the async user framework which we use to drive xfs) makes writing
> >> >>>>code like this easy, using continuations; but of course from ordinary
> >> >>>>threaded code it can be quite hard.
> >> >>>>
> >> >>>>btw, there was an attempt to make ext[34] async using this method, but I
> >> >>>>think it was ripped out. Yes, the mortal remains can still be seen with
> >> >>>>'git grep EIOCBQUEUED'.
> >> >>>>
> >> >>>>>>>It sounds to me that first and foremost you want to make sure you don't
> >> >>>>>>>have however many parallel operations you typically have running
> >> >>>>>>>contending on the same inodes or AGs. Hint: creating files under
> >> >>>>>>>separate subdirectories is a quick and easy way to allocate inodes under
> >> >>>>>>>separate AGs (the agno is encoded into the upper bits of the inode
> >> >>>>>>>number).
> >> >>>>>>Unfortunately our directory layout cannot be changed. And doesn't this
> >> >>>>>>require having agcount == O(number of active files)? That is easily in the
> >> >>>>>>thousands.
> >> >>>>>>
> >> >>>>>I think Glauber's O(nr_cpus) comment is probably the more likely
> >> >>>>>ballpark, but really it's something you'll probably just need to test to
> >> >>>>>see how far you need to go to avoid AG contention.
> >> >>>>>
> >> >>>>>I'm primarily throwing the subdir thing out there for testing purposes.
> >> >>>>>It's just an easy way to create inodes in a bunch of separate AGs so you
> >> >>>>>can determine whether/how much it really helps with modified AG counts.
> >> >>>>>I don't know enough about your application design to really comment on
> >> >>>>>that...
> >> >>>>We have O(cpus) shards that operate independently. Each shard writes 32MB
> >> >>>>commitlog files (that are pre-truncated to 32MB to allow concurrent writes
> >> >>>>without blocking); the files are then flushed and closed, and later removed.
> >> >>>>In parallel there are sequential writes and reads of large files using 128kB
> >> >>>>buffers), as well as random reads. Files are immutable (append-only), and
> >> >>>>if a file is being written, it is not concurrently read. In general files
> >> >>>>are not shared across shards. All I/O is async and O_DIRECT. open(),
> >> >>>>truncate(), fdatasync(), and friends are called from a helper thread.
> >> >>>>
> >> >>>>As far as I can tell it should a very friendly load for XFS and SSDs.
> >> >>>>
> >> >>>>>>> Reducing the frequency of block allocation/frees might also be
> >> >>>>>>>another help (e.g., preallocate and reuse files,
> >> >>>>>>Isn't that discouraged for SSDs?
> >> >>>>>>
> >> >>>>>Perhaps, if you're referring to the fact that the blocks are never freed
> >> >>>>>and thus never discarded..? Are you running fstrim?
> >> >>>>mount -o discard. And yes, overwrites are supposedly more expensive than
> >> >>>>trim old data + allocate new data, but maybe if you compare it with the work
> >> >>>>XFS has to do, perhaps the tradeoff is bad.
> >> >>>>
> >> >>>Ok, my understanding is that '-o discard' is not recommended in favor of
> >> >>>periodic fstrim for performance reasons, but that may or may not still
> >> >>>be the case.
> >> >>I understand that most SSDs have queued trim these days, but maybe I'm
> >> >>optimistic.
> >> >>
> >>
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-01 19:26 ` Avi Kivity
@ 2015-12-01 19:41 ` Christoph Hellwig
2015-12-01 19:50 ` Avi Kivity
2015-12-02 0:13 ` Brian Foster
1 sibling, 1 reply; 58+ messages in thread
From: Christoph Hellwig @ 2015-12-01 19:41 UTC (permalink / raw)
To: Avi Kivity; +Cc: Brian Foster, Glauber Costa, xfs
On Tue, Dec 01, 2015 at 09:26:42PM +0200, Avi Kivity wrote:
> It's basically the same thing. To to this, we'd have get_block either
> return the block's address (if it was in some metadata cache), or, if it was
> not, issue an I/O that fills (part of) that cache, and as its completion
> function, a continuation that reruns __blockdev_direct_IO from the point it
> was stopped so it can submit the data I/O (if the metadata cache was
> completely updated) or issue the next I/O aiming to fill that metadata
> cache, if it was not.
We did something this for blocking reads with great results, and it could be
done similarly for direct I/O I think:
https://lwn.net/Articles/612483/
Unfortunately Andrew shut it down for odd reasons so it didn't get in.
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-01 19:35 ` Brian Foster
@ 2015-12-01 19:45 ` Avi Kivity
0 siblings, 0 replies; 58+ messages in thread
From: Avi Kivity @ 2015-12-01 19:45 UTC (permalink / raw)
To: Brian Foster, Glauber Costa; +Cc: xfs
On 12/01/2015 09:35 PM, Brian Foster wrote:
> On Tue, Dec 01, 2015 at 02:07:41PM -0500, Glauber Costa wrote:
>> Hi Brian,
>>
>>
>>> Either way, the extents have to be read in at some point and I'd expect
>>> that cpu to schedule onto some other task while that thread waits on I/O
>>> to complete (read-ahead could also be a factor here, but I haven't
>>> really dug into how that is triggered for buffers).
>>>
>>
>> Being a datastore, we expect to run practically alone in any box we're
>> at. That means that there is no other task to run. If io_submit
>> blocks, the system blocks. The assumption that blocking will just
>> yield the processor for another thread makes sense in the general case
>> where you assume more than one application running and/or more than
>> one thread within the same application.
>>
> Hmm, well that helps me understand the concern a bit more. That said, I
> still question how likely this condition is. Even if this is a
> completely stripped down userspace with no other applications running,
> the kernel (or even XFS) alone might have plenty of threads/work items
> to execute to take care of "background" tasks for various subsystems.
There are not. We grab almost all of memory. All our I/O is O_DIRECT
so there is no page cache to write back. There may be softirq work from
networking, but in one mode we have (not yet in production), we use a
userspace networking stack, so no softirq at all.
That said, I doubt this is a problem now. Because the files are large
and well laid out, the amount of metadata is small and can easily be cached.
We might prime the metadata cache before launching the application, or
just ignore the whole problem. It would be much worse with small files,
but that isn't the case for us.
>
> Of course, we don't have all of the details of your environment so
> perhaps this is not the case. Perhaps a more productive approach here
> might be to find a way to detect this particular case (once you've
> worked out the other AG count tunings and whatnot that you want to use)
> where a thread into the fs is blocked and actually has nothing else to
> do and work from there. I _think_ there is such a thing as an idle task
> somewhere that might be useful to help quantify this, but I'd have to
> dig around to understand it better.
We simply observe the idle cpu counter going above zero.
Once we resolve the other issues, we'll instrument the kernel with
systemtap and see where the other blockages come from.
> That actually gives us a concrete scenario to work with, try to
> reproduce and improve on. It also facilitates improvements that might be
> beneficial to the general use case as opposed to tailored for this
> particular use case and highly specific environment. For example, if we
> find a particular sustained workload that repetitively blocks with
> nothing else to do, document and characterize it for the list and I'm
> sure people will come up with a variety of ideas to try and address it.
> Otherwise, we're kind of just looking around for context switch points
> and assuming that they will all just block with nothing else to do. For
> one, I don't think that's really accurate. It's also not very productive
> an approach and doesn't have any measurable benefit if it doesn't come
> along with a test case or reproducible condition.
I agree completely. We'll try to find better probe points than
schedule(). We'll also be able to come up with reproducers, this should
not be too hard once we have good instrumentation.
> Brian
>
>> From our user's perspective, however, every time that happens we can't
>> make progress. It doesn't really matter where it blocks.
>>
>> If io_submit returns without blocking, we can still push more work,
>> even though the kernel is still not ready to proceed. If it blocks,
>> we're dead.
>>
>>> Brian
>>>
>>>>> Brian
>>>>>
>>>>>>>> Seastar (the async user framework which we use to drive xfs) makes writing
>>>>>>>> code like this easy, using continuations; but of course from ordinary
>>>>>>>> threaded code it can be quite hard.
>>>>>>>>
>>>>>>>> btw, there was an attempt to make ext[34] async using this method, but I
>>>>>>>> think it was ripped out. Yes, the mortal remains can still be seen with
>>>>>>>> 'git grep EIOCBQUEUED'.
>>>>>>>>
>>>>>>>>>>> It sounds to me that first and foremost you want to make sure you don't
>>>>>>>>>>> have however many parallel operations you typically have running
>>>>>>>>>>> contending on the same inodes or AGs. Hint: creating files under
>>>>>>>>>>> separate subdirectories is a quick and easy way to allocate inodes under
>>>>>>>>>>> separate AGs (the agno is encoded into the upper bits of the inode
>>>>>>>>>>> number).
>>>>>>>>>> Unfortunately our directory layout cannot be changed. And doesn't this
>>>>>>>>>> require having agcount == O(number of active files)? That is easily in the
>>>>>>>>>> thousands.
>>>>>>>>>>
>>>>>>>>> I think Glauber's O(nr_cpus) comment is probably the more likely
>>>>>>>>> ballpark, but really it's something you'll probably just need to test to
>>>>>>>>> see how far you need to go to avoid AG contention.
>>>>>>>>>
>>>>>>>>> I'm primarily throwing the subdir thing out there for testing purposes.
>>>>>>>>> It's just an easy way to create inodes in a bunch of separate AGs so you
>>>>>>>>> can determine whether/how much it really helps with modified AG counts.
>>>>>>>>> I don't know enough about your application design to really comment on
>>>>>>>>> that...
>>>>>>>> We have O(cpus) shards that operate independently. Each shard writes 32MB
>>>>>>>> commitlog files (that are pre-truncated to 32MB to allow concurrent writes
>>>>>>>> without blocking); the files are then flushed and closed, and later removed.
>>>>>>>> In parallel there are sequential writes and reads of large files using 128kB
>>>>>>>> buffers), as well as random reads. Files are immutable (append-only), and
>>>>>>>> if a file is being written, it is not concurrently read. In general files
>>>>>>>> are not shared across shards. All I/O is async and O_DIRECT. open(),
>>>>>>>> truncate(), fdatasync(), and friends are called from a helper thread.
>>>>>>>>
>>>>>>>> As far as I can tell it should a very friendly load for XFS and SSDs.
>>>>>>>>
>>>>>>>>>>> Reducing the frequency of block allocation/frees might also be
>>>>>>>>>>> another help (e.g., preallocate and reuse files,
>>>>>>>>>> Isn't that discouraged for SSDs?
>>>>>>>>>>
>>>>>>>>> Perhaps, if you're referring to the fact that the blocks are never freed
>>>>>>>>> and thus never discarded..? Are you running fstrim?
>>>>>>>> mount -o discard. And yes, overwrites are supposedly more expensive than
>>>>>>>> trim old data + allocate new data, but maybe if you compare it with the work
>>>>>>>> XFS has to do, perhaps the tradeoff is bad.
>>>>>>>>
>>>>>>> Ok, my understanding is that '-o discard' is not recommended in favor of
>>>>>>> periodic fstrim for performance reasons, but that may or may not still
>>>>>>> be the case.
>>>>>> I understand that most SSDs have queued trim these days, but maybe I'm
>>>>>> optimistic.
>>>>>>
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-01 19:41 ` Christoph Hellwig
@ 2015-12-01 19:50 ` Avi Kivity
0 siblings, 0 replies; 58+ messages in thread
From: Avi Kivity @ 2015-12-01 19:50 UTC (permalink / raw)
To: Christoph Hellwig; +Cc: Brian Foster, Glauber Costa, xfs
On 12/01/2015 09:41 PM, Christoph Hellwig wrote:
> On Tue, Dec 01, 2015 at 09:26:42PM +0200, Avi Kivity wrote:
>> It's basically the same thing. To to this, we'd have get_block either
>> return the block's address (if it was in some metadata cache), or, if it was
>> not, issue an I/O that fills (part of) that cache, and as its completion
>> function, a continuation that reruns __blockdev_direct_IO from the point it
>> was stopped so it can submit the data I/O (if the metadata cache was
>> completely updated) or issue the next I/O aiming to fill that metadata
>> cache, if it was not.
> We did something this for blocking reads with great results, and it could be
> done similarly for direct I/O I think:
>
> https://lwn.net/Articles/612483/
>
> Unfortunately Andrew shut it down for odd reasons so it didn't get in.
How would this work? io_submit() returns -ENOTALLMETADATAISINCACHE,
user calls io_submit() again from a worker thread, where he doesn't mind
blocking?
In fact sys_io_submit() could catch this error and resubmit the I/O on
its own using a work item, and io_submit() would become non-blocking, at
least on I/O (lock contention may still be a problem, but a smaller one).
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-11-30 23:51 ` Glauber Costa
@ 2015-12-01 20:30 ` Dave Chinner
0 siblings, 0 replies; 58+ messages in thread
From: Dave Chinner @ 2015-12-01 20:30 UTC (permalink / raw)
To: Glauber Costa; +Cc: Avi Kivity, xfs
On Mon, Nov 30, 2015 at 06:51:51PM -0500, Glauber Costa wrote:
> On Mon, Nov 30, 2015 at 6:10 PM, Dave Chinner <david@fromorbit.com> wrote:
> > Let me have a think about how we can implement lazytime in a sane
> > way, such that fsync() works correctly, we don't throw away
> > timstamp changes in memory reclaim and we don't write unlogged
> > changes to the on-disk locations....
>
> I trust you fully for matters related to speed.
>
> Keep in mind, though, that at least for us the fact that it blocks is
> a lot worse than the fact that it is slow. We can work around slow,
> but blocking basically means that we won't have any more work to push
> - since we don't do threading. The processor that stales just sits
> idle until the lock is released. So any non-blocking solution to this
> would already be a win for us.
Right, the blocking is on the inode lock needed to do the
transactional update of the timestamp. lazytime would need to avoid
the timestamp update transaction completely, but we still need to
capture the timestamp and run the transaction later or capture it in
a subsequent change before we write back the inode.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-01 14:01 ` Glauber Costa
2015-12-01 14:37 ` Avi Kivity
@ 2015-12-01 20:45 ` Dave Chinner
2015-12-01 20:56 ` Avi Kivity
1 sibling, 1 reply; 58+ messages in thread
From: Dave Chinner @ 2015-12-01 20:45 UTC (permalink / raw)
To: Glauber Costa; +Cc: Avi Kivity, Brian Foster, xfs
On Tue, Dec 01, 2015 at 09:01:13AM -0500, Glauber Costa wrote:
> On Tue, Dec 1, 2015 at 8:58 AM, Avi Kivity <avi@scylladb.com> wrote:
> > On 12/01/2015 03:11 PM, Brian Foster wrote:
> >> It sounds to me that first and foremost you want to make sure you don't
> >> have however many parallel operations you typically have running
> >> contending on the same inodes or AGs. Hint: creating files under
> >> separate subdirectories is a quick and easy way to allocate inodes under
> >> separate AGs (the agno is encoded into the upper bits of the inode
> >> number).
> >
> >
> > Unfortunately our directory layout cannot be changed. And doesn't this
> > require having agcount == O(number of active files)? That is easily in the
> > thousands.
>
> Actually, wouldn't agcount == O(nr_cpus) be good enough?
Not quite. What you need is agcount ~= O(nr_active_allocations).
The difference is an allocation can block waiting on IO, and the
CPU can then go off and run another process, which then tries to do
an allocation. So you might only have 4 CPUs, but a workload that
can have a hundred active allocations at once (not uncommon in
file server workloads).
On worklaods that are roughly 1 process per CPU, it's typical that
agcount = 2 * N cpus gives pretty good results on large filesystems.
If you've got 400GB filesystems or you are using spinning disks,
then you probably don't want to go above 16 AGs, because then you
have problems with maintaining continugous free space and you'll
seek the spinning disks to death....
> >> 'mount -o ikeep,'
> >
> >
> > Interesting. Our files are large so we could try this.
Keep in mind that ikeep means that inode allocation permanently
fragments free space, which can affect how large files are allocated
once you truncate/rm the original files.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-01 20:45 ` Dave Chinner
@ 2015-12-01 20:56 ` Avi Kivity
2015-12-01 23:41 ` Dave Chinner
0 siblings, 1 reply; 58+ messages in thread
From: Avi Kivity @ 2015-12-01 20:56 UTC (permalink / raw)
To: Dave Chinner, Glauber Costa; +Cc: Brian Foster, xfs
On 12/01/2015 10:45 PM, Dave Chinner wrote:
> On Tue, Dec 01, 2015 at 09:01:13AM -0500, Glauber Costa wrote:
>> On Tue, Dec 1, 2015 at 8:58 AM, Avi Kivity <avi@scylladb.com> wrote:
>>> On 12/01/2015 03:11 PM, Brian Foster wrote:
>>>> It sounds to me that first and foremost you want to make sure you don't
>>>> have however many parallel operations you typically have running
>>>> contending on the same inodes or AGs. Hint: creating files under
>>>> separate subdirectories is a quick and easy way to allocate inodes under
>>>> separate AGs (the agno is encoded into the upper bits of the inode
>>>> number).
>>>
>>> Unfortunately our directory layout cannot be changed. And doesn't this
>>> require having agcount == O(number of active files)? That is easily in the
>>> thousands.
>> Actually, wouldn't agcount == O(nr_cpus) be good enough?
> Not quite. What you need is agcount ~= O(nr_active_allocations).
Yes, this is what I mean by "active files".
>
> The difference is an allocation can block waiting on IO, and the
> CPU can then go off and run another process, which then tries to do
> an allocation. So you might only have 4 CPUs, but a workload that
> can have a hundred active allocations at once (not uncommon in
> file server workloads).
But for us, probably not much more. We try to restrict active I/Os to
the effective disk queue depth (more than that and they just turn sour
waiting in the disk queue).
> On worklaods that are roughly 1 process per CPU, it's typical that
> agcount = 2 * N cpus gives pretty good results on large filesystems.
This is probably using sync calls. Using async calls you can have many
more I/Os in progress (but still limited by effective disk queue depth).
> If you've got 400GB filesystems or you are using spinning disks,
> then you probably don't want to go above 16 AGs, because then you
> have problems with maintaining continugous free space and you'll
> seek the spinning disks to death....
We're concentrating on SSDs for now.
>
>>>> 'mount -o ikeep,'
>>>
>>> Interesting. Our files are large so we could try this.
> Keep in mind that ikeep means that inode allocation permanently
> fragments free space, which can affect how large files are allocated
> once you truncate/rm the original files.
>
>
We can try to prime this by allocating a lot of inodes up front, then
removing them, so that this doesn't happen.
Hurray ext2.
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-01 15:22 ` Avi Kivity
2015-12-01 16:01 ` Brian Foster
@ 2015-12-01 21:04 ` Dave Chinner
2015-12-01 21:10 ` Glauber Costa
2015-12-01 21:24 ` Avi Kivity
1 sibling, 2 replies; 58+ messages in thread
From: Dave Chinner @ 2015-12-01 21:04 UTC (permalink / raw)
To: Avi Kivity; +Cc: Brian Foster, Glauber Costa, xfs
On Tue, Dec 01, 2015 at 05:22:38PM +0200, Avi Kivity wrote:
> On 12/01/2015 04:56 PM, Brian Foster wrote:
> >On Tue, Dec 01, 2015 at 03:58:28PM +0200, Avi Kivity wrote:
> >>> io_submit() can probably block in a variety of
> >>>places afaict... it might have to read in the inode extent map, allocate
> >>>blocks, take inode/ag locks, reserve log space for transactions, etc.
> >>Any chance of changing all that to be asynchronous? Doesn't sound too hard,
> >>if somebody else has to do it.
> >>
> >I'm not following... if the fs needs to read in the inode extent map to
> >prepare for an allocation, what else can the thread do but wait? Are you
> >suggesting the request kick off whatever the blocking action happens to
> >be asynchronously and return with an error such that the request can be
> >retried later?
>
> Not quite, it should be invisible to the caller.
I have a pony I can sell you.
> That is, the code called by io_submit()
> (file_operations::write_iter, it seems to be called today) can kick
> off this operation and have it continue from where it left off.
This is a problem that people have tried to solve in the past (e.g.
syslets, etc) where the thread executes until it has to block, and
then it's handled off to a worker thread/syslet to block and the
main process returns with EIOCBQUEUED.
Basically, you're asking for a real AIO infrastructure to
beintroduced into the kernel, and I think that's beyond what us XFS
guys can do...
> >>> Reducing the frequency of block allocation/frees might also be
> >>>another help (e.g., preallocate and reuse files,
> >>Isn't that discouraged for SSDs?
> >>
> >Perhaps, if you're referring to the fact that the blocks are never freed
> >and thus never discarded..? Are you running fstrim?
>
> mount -o discard. And yes, overwrites are supposedly more expensive
> than trim old data + allocate new data, but maybe if you compare it
> with the work XFS has to do, perhaps the tradeoff is bad.
Oh, you do realise that using "-o discard" causes significant delays
in journal commit processing? i.e. the journal commit completion
blocks until all the discards have been submitted and waited on
*synchronously*. This is a problem with the linux block layer in
that blkdev_issue_discard() is a synchronous operation.....
Hence if you are seeing delays in transactions (e.g. timestamp updates)
it's entirely possible that things will get much better if you
remove the discard mount option. It's much better from a performance
perspective to use the fstrim command every so often - fstrim issues
discard operations in the context of the fstrim process - it does
not interact with the transaction subsystem at all.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-01 21:04 ` Dave Chinner
@ 2015-12-01 21:10 ` Glauber Costa
2015-12-01 21:39 ` Dave Chinner
2015-12-01 21:24 ` Avi Kivity
1 sibling, 1 reply; 58+ messages in thread
From: Glauber Costa @ 2015-12-01 21:10 UTC (permalink / raw)
To: Dave Chinner; +Cc: Avi Kivity, Brian Foster, xfs
On Tue, Dec 1, 2015 at 4:04 PM, Dave Chinner <david@fromorbit.com> wrote:
> On Tue, Dec 01, 2015 at 05:22:38PM +0200, Avi Kivity wrote:
>> On 12/01/2015 04:56 PM, Brian Foster wrote:
>> >On Tue, Dec 01, 2015 at 03:58:28PM +0200, Avi Kivity wrote:
>> >>> io_submit() can probably block in a variety of
>> >>>places afaict... it might have to read in the inode extent map, allocate
>> >>>blocks, take inode/ag locks, reserve log space for transactions, etc.
>> >>Any chance of changing all that to be asynchronous? Doesn't sound too hard,
>> >>if somebody else has to do it.
>> >>
>> >I'm not following... if the fs needs to read in the inode extent map to
>> >prepare for an allocation, what else can the thread do but wait? Are you
>> >suggesting the request kick off whatever the blocking action happens to
>> >be asynchronously and return with an error such that the request can be
>> >retried later?
>>
>> Not quite, it should be invisible to the caller.
>
> I have a pony I can sell you.
>
>> That is, the code called by io_submit()
>> (file_operations::write_iter, it seems to be called today) can kick
>> off this operation and have it continue from where it left off.
>
> This is a problem that people have tried to solve in the past (e.g.
> syslets, etc) where the thread executes until it has to block, and
> then it's handled off to a worker thread/syslet to block and the
> main process returns with EIOCBQUEUED.
>
> Basically, you're asking for a real AIO infrastructure to
> beintroduced into the kernel, and I think that's beyond what us XFS
> guys can do...
>
>> >>> Reducing the frequency of block allocation/frees might also be
>> >>>another help (e.g., preallocate and reuse files,
>> >>Isn't that discouraged for SSDs?
>> >>
>> >Perhaps, if you're referring to the fact that the blocks are never freed
>> >and thus never discarded..? Are you running fstrim?
>>
>> mount -o discard. And yes, overwrites are supposedly more expensive
>> than trim old data + allocate new data, but maybe if you compare it
>> with the work XFS has to do, perhaps the tradeoff is bad.
>
> Oh, you do realise that using "-o discard" causes significant delays
> in journal commit processing? i.e. the journal commit completion
> blocks until all the discards have been submitted and waited on
> *synchronously*. This is a problem with the linux block layer in
> that blkdev_issue_discard() is a synchronous operation.....
>
> Hence if you are seeing delays in transactions (e.g. timestamp updates)
> it's entirely possible that things will get much better if you
> remove the discard mount option. It's much better from a performance
> perspective to use the fstrim command every so often - fstrim issues
> discard operations in the context of the fstrim process - it does
> not interact with the transaction subsystem at all.
Hi Dave,
This is news to me.
However, in the disk that we have used during the acquisition of this
trace, discard doesn't seem to be supported:
$ sudo fstrim /data/
fstrim: /data/: the discard operation is not supported
In that case, if I understand correctly the discard mount option
should be a noop, no?
That recommendation is great for our general case, though.
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-01 19:07 ` Avi Kivity
@ 2015-12-01 21:19 ` Dave Chinner
2015-12-01 21:38 ` Avi Kivity
0 siblings, 1 reply; 58+ messages in thread
From: Dave Chinner @ 2015-12-01 21:19 UTC (permalink / raw)
To: Avi Kivity; +Cc: Glauber Costa, xfs
On Tue, Dec 01, 2015 at 09:07:14PM +0200, Avi Kivity wrote:
> On 12/01/2015 08:03 PM, Carlos Maiolino wrote:
> >Hi Avi,
> >
> >>>else is going to execute in our place until this thread can make
> >>>progress.
> >>For us, nothing else can execute in our place, we usually have exactly one
> >>thread per logical core. So we are heavily dependent on io_submit not
> >>sleeping.
> >>
> >>The case of a contended lock is, to me, less worrying. It can be reduced by
> >>using more allocation groups, which is apparently the shared resource under
> >>contention.
> >>
> >I apologize if I misread your previous comments, but, IIRC you said you can't
> >change the directory structure your application is using, and IIRC your
> >application does not spread files across several directories.
>
> I miswrote somewhat: the application writes data files and commitlog
> files. The data file directory structure is fixed due to
> compatibility concerns (it is not a single directory, but some
> workloads will see most access on files in a single directory. The
> commitlog directory structure is more relaxed, and we can split it
> to a directory per shard (=cpu) or something else.
>
> If worst comes to worst, we'll hack around this and distribute the
> data files into more directories, and provide some hack for
> compatibility.
>
> >XFS spread files across the allocation groups, based on the directory these
> >files are created,
>
> Idea: create the files in some subdirectory, and immediately move
> them to their required location.
See xfs_fsr.
>
> > trying to keep files as close as possible from their
> >metadata.
>
> This is pointless for an SSD. Perhaps XFS should randomize the ag on
> nonrotational media instead.
Actually, no, it is not pointless. SSDs do not require optimisation
for minimal seek time, but data locality is still just as important
as spinning disks, if not moreso. Why? Because the garbage
collection routines in the SSDs are all about locality and we can't
drive garbage collection effectively via discard operations if the
filesystem is not keeping temporally related files close together in
it's block address space.
e.g. If the files in a directory are all close together, and the
directory is removed, we then leave a big empty contiguous region in
the filesystem free space map, and when we send discards over that
we end up with a single big trim and the drive handles that far more
effectively than lots of little trims (i.e. one per file) that the
drive cannot do anything useful with because they are all smaller
than the internal SSD page/block sizes and so get ignored. This is
one of the reasons fstrim is so much more efficient and effective
than using the discard mount option.
And, well, XFS is designed to operate on storage devices made up of
more than one drive, so the way AGs are selected is designed to
given long term load balancing (both for space usage and
instantenous performance). With the existing algorithms we've not
had any issues with SSD lifetimes, long term performance
degradation, etc, so there's no evidence that we actually need to
change the fundamental allocation algorithms specially for SSDs.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-01 21:04 ` Dave Chinner
2015-12-01 21:10 ` Glauber Costa
@ 2015-12-01 21:24 ` Avi Kivity
2015-12-01 21:31 ` Glauber Costa
1 sibling, 1 reply; 58+ messages in thread
From: Avi Kivity @ 2015-12-01 21:24 UTC (permalink / raw)
To: Dave Chinner; +Cc: Brian Foster, Glauber Costa, xfs
On 12/01/2015 11:04 PM, Dave Chinner wrote:
> On Tue, Dec 01, 2015 at 05:22:38PM +0200, Avi Kivity wrote:
>> On 12/01/2015 04:56 PM, Brian Foster wrote:
>>> On Tue, Dec 01, 2015 at 03:58:28PM +0200, Avi Kivity wrote:
>>>>> io_submit() can probably block in a variety of
>>>>> places afaict... it might have to read in the inode extent map, allocate
>>>>> blocks, take inode/ag locks, reserve log space for transactions, etc.
>>>> Any chance of changing all that to be asynchronous? Doesn't sound too hard,
>>>> if somebody else has to do it.
>>>>
>>> I'm not following... if the fs needs to read in the inode extent map to
>>> prepare for an allocation, what else can the thread do but wait? Are you
>>> suggesting the request kick off whatever the blocking action happens to
>>> be asynchronously and return with an error such that the request can be
>>> retried later?
>> Not quite, it should be invisible to the caller.
> I have a pony I can sell you.
You already sold me a pony.
>> That is, the code called by io_submit()
>> (file_operations::write_iter, it seems to be called today) can kick
>> off this operation and have it continue from where it left off.
> This is a problem that people have tried to solve in the past (e.g.
> syslets, etc) where the thread executes until it has to block, and
> then it's handled off to a worker thread/syslet to block and the
> main process returns with EIOCBQUEUED.
Yes, I remember that.
> Basically, you're asking for a real AIO infrastructure to
> beintroduced into the kernel, and I think that's beyond what us XFS
> guys can do...
Sure you can, Dave. In fact you feel an irresistible urge to do it.
But I don't think the EIOCBQUEUED thing need be repeated. We can have a
simpler implementation:
- Add a task flag TIF_AIO, which causes any new I/O to fail with
EAIOWOULDBLOCK.
- have __blockdev_direct_IO() do its block-mapping operations with
TIF_AIO set (but remove it just before issuing the bio).
- sys_aio_submit() catches EAIOWOULDBLOCK and resubmits the aio in a
work item, this time without TIF_AIO games.
The effect would be similar to EIOCBQUEUED, but simpler, as instead of
issuing any metadata I/O you abort the operation and restart it from
scratch.
>
>>>>> Reducing the frequency of block allocation/frees might also be
>>>>> another help (e.g., preallocate and reuse files,
>>>> Isn't that discouraged for SSDs?
>>>>
>>> Perhaps, if you're referring to the fact that the blocks are never freed
>>> and thus never discarded..? Are you running fstrim?
>> mount -o discard. And yes, overwrites are supposedly more expensive
>> than trim old data + allocate new data, but maybe if you compare it
>> with the work XFS has to do, perhaps the tradeoff is bad.
> Oh, you do realise that using "-o discard" causes significant delays
> in journal commit processing? i.e. the journal commit completion
> blocks until all the discards have been submitted and waited on
> *synchronously*. This is a problem with the linux block layer in
> that blkdev_issue_discard() is a synchronous operation.....
I do now. What's the unicode for a crying face?
> Hence if you are seeing delays in transactions (e.g. timestamp updates)
> it's entirely possible that things will get much better if you
> remove the discard mount option. It's much better from a performance
> perspective to use the fstrim command every so often - fstrim issues
> discard operations in the context of the fstrim process - it does
> not interact with the transaction subsystem at all.
>
>
All right. On the other hand we have to know when to issue it. That
would be when nn% of the disk area have been rewritten. Is there some
counter I can poll every minute or so for this? Not doing the fstrim in
time would cause the disk performance to tank.
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-01 21:24 ` Avi Kivity
@ 2015-12-01 21:31 ` Glauber Costa
0 siblings, 0 replies; 58+ messages in thread
From: Glauber Costa @ 2015-12-01 21:31 UTC (permalink / raw)
To: Avi Kivity; +Cc: Brian Foster, xfs
>
>>> That is, the code called by io_submit()
>>> (file_operations::write_iter, it seems to be called today) can kick
>>> off this operation and have it continue from where it left off.
>>
>> This is a problem that people have tried to solve in the past (e.g.
>> syslets, etc) where the thread executes until it has to block, and
>> then it's handled off to a worker thread/syslet to block and the
>> main process returns with EIOCBQUEUED.
>
>
> Yes, I remember that.
>
>> Basically, you're asking for a real AIO infrastructure to
>> beintroduced into the kernel, and I think that's beyond what us XFS
>> guys can do...
>
>
> Sure you can, Dave. In fact you feel an irresistible urge to do it.
What is that? Are you that anxious for the Star Wars premiere that you
are trying your very own jedi mind tricks??
> I do now. What's the unicode for a crying face?
>
>> Hence if you are seeing delays in transactions (e.g. timestamp updates)
>> it's entirely possible that things will get much better if you
>> remove the discard mount option. It's much better from a performance
>> perspective to use the fstrim command every so often - fstrim issues
>> discard operations in the context of the fstrim process - it does
>> not interact with the transaction subsystem at all.
>>
>>
>
> All right. On the other hand we have to know when to issue it. That would
> be when nn% of the disk area have been rewritten. Is there some counter I
> can poll every minute or so for this? Not doing the fstrim in time would
> cause the disk performance to tank.
Note, as I said, that while this is a really good general
recommendation from down under, that was not likely to have had any
effect in the current trace - that disk does not support discard, and
I am assuming the mount option becomes a noop in this case.
>
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-01 21:19 ` Dave Chinner
@ 2015-12-01 21:38 ` Avi Kivity
2015-12-01 23:06 ` Dave Chinner
0 siblings, 1 reply; 58+ messages in thread
From: Avi Kivity @ 2015-12-01 21:38 UTC (permalink / raw)
To: Dave Chinner; +Cc: Glauber Costa, xfs
On 12/01/2015 11:19 PM, Dave Chinner wrote:
> On Tue, Dec 01, 2015 at 09:07:14PM +0200, Avi Kivity wrote:
>> On 12/01/2015 08:03 PM, Carlos Maiolino wrote:
>>> Hi Avi,
>>>
>>>>> else is going to execute in our place until this thread can make
>>>>> progress.
>>>> For us, nothing else can execute in our place, we usually have exactly one
>>>> thread per logical core. So we are heavily dependent on io_submit not
>>>> sleeping.
>>>>
>>>> The case of a contended lock is, to me, less worrying. It can be reduced by
>>>> using more allocation groups, which is apparently the shared resource under
>>>> contention.
>>>>
>>> I apologize if I misread your previous comments, but, IIRC you said you can't
>>> change the directory structure your application is using, and IIRC your
>>> application does not spread files across several directories.
>> I miswrote somewhat: the application writes data files and commitlog
>> files. The data file directory structure is fixed due to
>> compatibility concerns (it is not a single directory, but some
>> workloads will see most access on files in a single directory. The
>> commitlog directory structure is more relaxed, and we can split it
>> to a directory per shard (=cpu) or something else.
>>
>> If worst comes to worst, we'll hack around this and distribute the
>> data files into more directories, and provide some hack for
>> compatibility.
>>
>>> XFS spread files across the allocation groups, based on the directory these
>>> files are created,
>> Idea: create the files in some subdirectory, and immediately move
>> them to their required location.
> See xfs_fsr.
Can you elaborate? I don't see how it is applicable.
My hack involves creating the file in a random directory, and while it
is still zero sized, move it to its final directory. This is simply to
defeat the ag selection heuristic. No data is copied.
>>> trying to keep files as close as possible from their
>>> metadata.
>> This is pointless for an SSD. Perhaps XFS should randomize the ag on
>> nonrotational media instead.
> Actually, no, it is not pointless. SSDs do not require optimisation
> for minimal seek time, but data locality is still just as important
> as spinning disks, if not moreso. Why? Because the garbage
> collection routines in the SSDs are all about locality and we can't
> drive garbage collection effectively via discard operations if the
> filesystem is not keeping temporally related files close together in
> it's block address space.
In my case, files in the same directory are not temporally related. But
I understand where the heuristic comes from.
Maybe an ioctl to set a directory attribute "the files in this directory
are not temporally related"?
I imagine this will be useful for many server applications.
> e.g. If the files in a directory are all close together, and the
> directory is removed, we then leave a big empty contiguous region in
> the filesystem free space map, and when we send discards over that
> we end up with a single big trim and the drive handles that far more
Would this not be defeated if a directory that happens to share the
allocation group gets populated simultaneously?
> effectively than lots of little trims (i.e. one per file) that the
> drive cannot do anything useful with because they are all smaller
> than the internal SSD page/block sizes and so get ignored. This is
> one of the reasons fstrim is so much more efficient and effective
> than using the discard mount option.
In my use case, the files are fairly large, and there is constant
rewriting (not in-place: files are read, merged, and written back). So
I'm worried an fstrim can happen too late.
>
> And, well, XFS is designed to operate on storage devices made up of
> more than one drive, so the way AGs are selected is designed to
> given long term load balancing (both for space usage and
> instantenous performance). With the existing algorithms we've not
> had any issues with SSD lifetimes, long term performance
> degradation, etc, so there's no evidence that we actually need to
> change the fundamental allocation algorithms specially for SSDs.
>
Ok. Maybe the SSDs can deal with untrimmed overwrites efficiently,
provided the io sizes are large enough.
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-01 21:10 ` Glauber Costa
@ 2015-12-01 21:39 ` Dave Chinner
0 siblings, 0 replies; 58+ messages in thread
From: Dave Chinner @ 2015-12-01 21:39 UTC (permalink / raw)
To: Glauber Costa; +Cc: Avi Kivity, Brian Foster, xfs
On Tue, Dec 01, 2015 at 04:10:45PM -0500, Glauber Costa wrote:
> On Tue, Dec 1, 2015 at 4:04 PM, Dave Chinner <david@fromorbit.com> wrote:
> > On Tue, Dec 01, 2015 at 05:22:38PM +0200, Avi Kivity wrote:
> >> On 12/01/2015 04:56 PM, Brian Foster wrote:
> >> mount -o discard. And yes, overwrites are supposedly more expensive
> >> than trim old data + allocate new data, but maybe if you compare it
> >> with the work XFS has to do, perhaps the tradeoff is bad.
> >
> > Oh, you do realise that using "-o discard" causes significant delays
> > in journal commit processing? i.e. the journal commit completion
> > blocks until all the discards have been submitted and waited on
> > *synchronously*. This is a problem with the linux block layer in
> > that blkdev_issue_discard() is a synchronous operation.....
> >
> > Hence if you are seeing delays in transactions (e.g. timestamp updates)
> > it's entirely possible that things will get much better if you
> > remove the discard mount option. It's much better from a performance
> > perspective to use the fstrim command every so often - fstrim issues
> > discard operations in the context of the fstrim process - it does
> > not interact with the transaction subsystem at all.
>
> Hi Dave,
>
> This is news to me.
>
> However, in the disk that we have used during the acquisition of this
> trace, discard doesn't seem to be supported:
> $ sudo fstrim /data/
> fstrim: /data/: the discard operation is not supported
>
> In that case, if I understand correctly the discard mount option
> should be a noop, no?
XFS still makes the blkdev_issue_discard() calls, though, because
the block device can turn discard support on and off dynamically.
e.g. raid devices where a faulty drive is replaced temporarily with
a drive that doesn't have discard support. The block device suddenly
starts returning -EOPNOTSUPP to the filesystem from
blkdev_issue_discard() calls. However, the admin then replaces that
drive with a new one that des have discard support, and now
blkdev_issue_discard() works as exepected.
IOWs, if you set the mount option, XFS will always attempt to issue
discards...
> That recommendation is great for our general case, though.
For the moment. Given lots of time, reworking this code could
greatly reduce the impact/overhead of it and so make it practical to
enable. There's a lot of work to get to that point, though...
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-01 21:38 ` Avi Kivity
@ 2015-12-01 23:06 ` Dave Chinner
2015-12-02 9:02 ` Avi Kivity
0 siblings, 1 reply; 58+ messages in thread
From: Dave Chinner @ 2015-12-01 23:06 UTC (permalink / raw)
To: Avi Kivity; +Cc: Glauber Costa, xfs
On Tue, Dec 01, 2015 at 11:38:29PM +0200, Avi Kivity wrote:
> On 12/01/2015 11:19 PM, Dave Chinner wrote:
> >On Tue, Dec 01, 2015 at 09:07:14PM +0200, Avi Kivity wrote:
> >>On 12/01/2015 08:03 PM, Carlos Maiolino wrote:
> >>>Hi Avi,
> >>>
> >>>>>else is going to execute in our place until this thread can make
> >>>>>progress.
> >>>>For us, nothing else can execute in our place, we usually have exactly one
> >>>>thread per logical core. So we are heavily dependent on io_submit not
> >>>>sleeping.
> >>>>
> >>>>The case of a contended lock is, to me, less worrying. It can be reduced by
> >>>>using more allocation groups, which is apparently the shared resource under
> >>>>contention.
> >>>>
> >>>I apologize if I misread your previous comments, but, IIRC you said you can't
> >>>change the directory structure your application is using, and IIRC your
> >>>application does not spread files across several directories.
> >>I miswrote somewhat: the application writes data files and commitlog
> >>files. The data file directory structure is fixed due to
> >>compatibility concerns (it is not a single directory, but some
> >>workloads will see most access on files in a single directory. The
> >>commitlog directory structure is more relaxed, and we can split it
> >>to a directory per shard (=cpu) or something else.
> >>
> >>If worst comes to worst, we'll hack around this and distribute the
> >>data files into more directories, and provide some hack for
> >>compatibility.
> >>
> >>>XFS spread files across the allocation groups, based on the directory these
> >>>files are created,
> >>Idea: create the files in some subdirectory, and immediately move
> >>them to their required location.
> >See xfs_fsr.
>
> Can you elaborate? I don't see how it is applicable.
Just pointing out that this is what xfs_fsr does to control locality
of allocation for files it is defragmenting. Except that rather than
moving files, it uses XFS_IOC_SWAPEXT to switch the data between two
inodes atomically...
> My hack involves creating the file in a random directory, and while
> it is still zero sized, move it to its final directory. This is
> simply to defeat the ag selection heuristic.
Which you really don't want to do.
> >>> trying to keep files as close as possible from their
> >>>metadata.
> >>This is pointless for an SSD. Perhaps XFS should randomize the ag on
> >>nonrotational media instead.
> >Actually, no, it is not pointless. SSDs do not require optimisation
> >for minimal seek time, but data locality is still just as important
> >as spinning disks, if not moreso. Why? Because the garbage
> >collection routines in the SSDs are all about locality and we can't
> >drive garbage collection effectively via discard operations if the
> >filesystem is not keeping temporally related files close together in
> >it's block address space.
>
> In my case, files in the same directory are not temporally related.
> But I understand where the heuristic comes from.
>
> Maybe an ioctl to set a directory attribute "the files in this
> directory are not temporally related"?
And exactly what does that gain us? Exactly what problem are you
trying to solve by manipulating file locality that can't be solved
by existing knobs and config options?
Perhaps you'd like to read up on how the inode32 allocator behaves?
> >e.g. If the files in a directory are all close together, and the
> >directory is removed, we then leave a big empty contiguous region in
> >the filesystem free space map, and when we send discards over that
> >we end up with a single big trim and the drive handles that far more
>
> Would this not be defeated if a directory that happens to share the
> allocation group gets populated simultaneously?
Sure. But this sort of thing is rare in the real world, and when
they do occur, it generally only takes small tweaks to algorithms
and layouts make them go away. I don't care to bikeshed about
theoretical problems - I'm in the business of finding the root cause
of the problems users are having and solving those problems. So far
what you've given us is a ball of "there's blocking in AIO
submission", and the only one that is clear cut is the timestamp
update.
Go back and categorise the types of blocking that you are seeing -
whether it be on the AGIs during inode manipulation, one the AGFs
becuse of concurrent extent allocation, on log forces because of
slow discards in the transcation completion, on the transaction
subsystem because of a lack of log space for concurrent
reservations, etc. And then determine if changing the layout of the
filesystem (e.g. number of AGs, size of log, etc) and different
mount options (e.g. turning off discard, using inode32 allocator,
etc) make any difference to the blocking issues you are seeing.
Once we know which of the different algorithms is causing the
blocking issues, we'll know a lot more about why we're having
problems and a better idea of what problems we actually need to
solve.
> >effectively than lots of little trims (i.e. one per file) that the
> >drive cannot do anything useful with because they are all smaller
> >than the internal SSD page/block sizes and so get ignored. This is
> >one of the reasons fstrim is so much more efficient and effective
> >than using the discard mount option.
>
> In my use case, the files are fairly large, and there is constant
> rewriting (not in-place: files are read, merged, and written back).
> So I'm worried an fstrim can happen too late.
Have you measured the SSD performance degradation over time due to
large overwrites? If not, then again it is a good chance you are
trying to solve a theoretical problem rather than a real problem....
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-01 20:56 ` Avi Kivity
@ 2015-12-01 23:41 ` Dave Chinner
2015-12-02 8:23 ` Avi Kivity
0 siblings, 1 reply; 58+ messages in thread
From: Dave Chinner @ 2015-12-01 23:41 UTC (permalink / raw)
To: Avi Kivity; +Cc: Brian Foster, Glauber Costa, xfs
On Tue, Dec 01, 2015 at 10:56:01PM +0200, Avi Kivity wrote:
> On 12/01/2015 10:45 PM, Dave Chinner wrote:
> >On Tue, Dec 01, 2015 at 09:01:13AM -0500, Glauber Costa wrote:
> >The difference is an allocation can block waiting on IO, and the
> >CPU can then go off and run another process, which then tries to do
> >an allocation. So you might only have 4 CPUs, but a workload that
> >can have a hundred active allocations at once (not uncommon in
> >file server workloads).
>
> But for us, probably not much more. We try to restrict active I/Os
> to the effective disk queue depth (more than that and they just turn
> sour waiting in the disk queue).
>
>
> >On worklaods that are roughly 1 process per CPU, it's typical that
> >agcount = 2 * N cpus gives pretty good results on large filesystems.
>
> This is probably using sync calls. Using async calls you can have
> many more I/Os in progress (but still limited by effective disk
> queue depth).
Ah, no. Even with async IO you don't want unbound allocation
concurrency. The allocation algorithms rely on having contiguous
free space extents that are much larger than the allocations being
done to work effeectively and minimise file fragmentation. If you
chop the filesystem up into lots of small AGs, then it accelerates
the rate at which the free space gets chopped up into smaller
extents and performance then suffers. It's the same problem as
running a large filesystem near ENOSPC for an extended period of
time, which again is something we most definitely don't recommend
you do in production systems.
> >If you've got 400GB filesystems or you are using spinning disks,
> >then you probably don't want to go above 16 AGs, because then you
> >have problems with maintaining continugous free space and you'll
> >seek the spinning disks to death....
>
> We're concentrating on SSDs for now.
Sure, so "problems with maintaining continugous free space" is what
you need to be concerned about.
> >>>>'mount -o ikeep,'
> >>>
> >>>Interesting. Our files are large so we could try this.
> >Keep in mind that ikeep means that inode allocation permanently
> >fragments free space, which can affect how large files are allocated
> >once you truncate/rm the original files.
>
> We can try to prime this by allocating a lot of inodes up front,
> then removing them, so that this doesn't happen.
Again - what problem have you measured that inode preallocation will
solves in your application? Don't make changes just because you
*think* it will fix what you *think* is a problem. Measure, analyse,
solve, in that order.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-01 19:26 ` Avi Kivity
2015-12-01 19:41 ` Christoph Hellwig
@ 2015-12-02 0:13 ` Brian Foster
2015-12-02 0:57 ` Dave Chinner
2015-12-02 8:34 ` Avi Kivity
1 sibling, 2 replies; 58+ messages in thread
From: Brian Foster @ 2015-12-02 0:13 UTC (permalink / raw)
To: Avi Kivity; +Cc: Glauber Costa, xfs
On Tue, Dec 01, 2015 at 09:26:42PM +0200, Avi Kivity wrote:
> On 12/01/2015 08:51 PM, Brian Foster wrote:
> >On Tue, Dec 01, 2015 at 07:09:29PM +0200, Avi Kivity wrote:
> >>
> >>On 12/01/2015 06:29 PM, Brian Foster wrote:
> >>>On Tue, Dec 01, 2015 at 06:08:51PM +0200, Avi Kivity wrote:
> >>>>On 12/01/2015 06:01 PM, Brian Foster wrote:
> >>>>>On Tue, Dec 01, 2015 at 05:22:38PM +0200, Avi Kivity wrote:
> >>>>>>On 12/01/2015 04:56 PM, Brian Foster wrote:
> >>>>>>>On Tue, Dec 01, 2015 at 03:58:28PM +0200, Avi Kivity wrote:
> >>>>>>>>On 12/01/2015 03:11 PM, Brian Foster wrote:
> >>>>>>>>>On Tue, Dec 01, 2015 at 11:08:47AM +0200, Avi Kivity wrote:
> >>>>>>>>>>On 11/30/2015 06:14 PM, Brian Foster wrote:
> >>>>>>>>>>>On Mon, Nov 30, 2015 at 04:29:13PM +0200, Avi Kivity wrote:
> >>>>>>>>>>>>On 11/30/2015 04:10 PM, Brian Foster wrote:
...
> >>The case of waiting for I/O is much more worrying, because I/O latency are
> >>much higher. But it seems like most of the DIO path does not trigger
> >>locking around I/O (and we are careful to avoid the ones that do, like
> >>writing beyond eof).
> >>
> >>(sorry for repeating myself, I have the feeling we are talking past each
> >>other and want to be on the same page)
> >>
> >Yeah, my point is just that just because the thread blocked on I/O,
> >doesn't mean the cpu can't carry on with some useful work for another
> >task.
>
> In our case, there is no other task. We run one thread per logical core, so
> if that thread gets blocked, the cpu idles.
>
> The whole point of io_submit() is to issue an I/O and let the caller
> continue processing immediately. It is the equivalent of O_NONBLOCK for
> networking code. If O_NONBLOCK did block from time to time, practically all
> modern network applications would see a huge performance drop.
>
Ok, but my understanding is that O_NONBLOCK would return an error code
in the blocking case such that userspace can do something else or retry
from a blockable context. I think this is similar to what hch posted wrt
to the pwrite2() bits for nonblocking buffered I/O or what I was asking
about earlier on with regard to returning an error if some blocking
would otherwise occur.
> >
> >>>>> We submit an I/O which is
> >>>>>asynchronous in nature and wait on a completion, which causes the cpu to
> >>>>>schedule and execute another task until the completion is set by I/O
> >>>>>completion (via an async callback). At that point, the issuing thread
> >>>>>continues where it left off. I suspect I'm missing something... can you
> >>>>>elaborate on what you'd do differently here (and how it helps)?
> >>>>Just apply the same technique everywhere: convert locks to trylock +
> >>>>schedule a continuation on failure.
> >>>>
> >>>I'm certainly not an expert on the kernel scheduling, locking and
> >>>serialization mechanisms, but my understanding is that most things
> >>>outside of spin locks are reschedule points. For example, the
> >>>wait_for_completion() calls XFS uses to wait on I/O boil down to
> >>>schedule_timeout() calls. Buffer locks are implemented as semaphores and
> >>>down() can end up in the same place.
> >>But, for the most part, XFS seems to be able to avoid sleeping. The call to
> >>__blockdev_direct_IO only launches the I/O, so any locking is only around
> >>cpu operations and, unless there is contention, won't cause us to sleep in
> >>io_submit().
> >>
> >>Trying to follow the code, it looks like xfs_get_blocks_direct (and
> >>__blockdev_direct_IO's get_block parameter in general) is synchronous, so
> >>we're just lucky to have everything in cache. If it isn't, we block right
> >>there. I really hope I'm misreading this and some other magic is happening
> >>elsewhere instead of this.
> >>
> >Nope, it's synchronous from a code perspective. The
> >xfs_bmapi_read()->xfs_iread_extents() path could have to read in the
> >inode bmap metadata if it hasn't been done already. Note that this
> >should only happen once as everything is stored in-core, so in most
> >cases this is skipped. It's also possible extents are read in via some
> >other path/operation on the inode before an async I/O happens to be
> >submitted (e.g., see some of the other xfs_bmapi_read() callers).
>
> Is there (could we add) some ioctl to prime this cache? We could call it
> from a worker thread where we don't mind blocking during open.
>
I suppose that's possible, or the worker thread could perform some
existing operation known to prime the cache. I don't think it's worth
getting into without a concrete example, however. The extent read
example we're batting around might not ever be a problem (as you've
noted due to file size), if files are truncated and recycled, for
example.
> What is the eviction policy for this cache? Is it simply the block
> device's page cache?
>
IIUC the extent list stays around until the inode is reclaimed. There's
a separate buffer cache for metadata buffers. Both types of objects
would be reclaimed based on memory pressure.
> What about the write path, will we see the same problems there? I would
> guess the problem is less severe there if the metadata is written with
> writeback policy.
>
Metadata is modified in-core and handed off to the logging
infrastructure via a transaction. The log is flushed to disk some time
later and metadata writeback occurs asynchronously via the xfsaild
thread.
Brian
> >
> >Either way, the extents have to be read in at some point and I'd expect
> >that cpu to schedule onto some other task while that thread waits on I/O
> >to complete (read-ahead could also be a factor here, but I haven't
> >really dug into how that is triggered for buffers).
>
> To provide an example, our application, which is a database, faces this
> problem exact at a higher level. Data is stored in data files, and data
> items' locations are stored in index files. When we read a bit of data, we
> issue an index read, and pass it a continuation to be executed when the read
> completes. This latter continuation parses the data and passes it to the
> code that prepares it for merging with data from other data files, and an
> eventual return to the user.
>
> Having written code for over a year in this style, I've come to expect it to
> be used everywhere asynchronous I/O is used, but I realize it is fairly hard
> without good support from a framework that allows continuations to be
> composed in a natural way.
>
>
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-02 0:13 ` Brian Foster
@ 2015-12-02 0:57 ` Dave Chinner
2015-12-02 8:38 ` Avi Kivity
2015-12-02 8:34 ` Avi Kivity
1 sibling, 1 reply; 58+ messages in thread
From: Dave Chinner @ 2015-12-02 0:57 UTC (permalink / raw)
To: Brian Foster; +Cc: Avi Kivity, Glauber Costa, xfs
On Tue, Dec 01, 2015 at 07:13:29PM -0500, Brian Foster wrote:
> On Tue, Dec 01, 2015 at 09:26:42PM +0200, Avi Kivity wrote:
> > On 12/01/2015 08:51 PM, Brian Foster wrote:
> > >On Tue, Dec 01, 2015 at 07:09:29PM +0200, Avi Kivity wrote:
> > >Nope, it's synchronous from a code perspective. The
> > >xfs_bmapi_read()->xfs_iread_extents() path could have to read in the
> > >inode bmap metadata if it hasn't been done already. Note that this
> > >should only happen once as everything is stored in-core, so in most
> > >cases this is skipped. It's also possible extents are read in via some
> > >other path/operation on the inode before an async I/O happens to be
> > >submitted (e.g., see some of the other xfs_bmapi_read() callers).
> >
> > Is there (could we add) some ioctl to prime this cache? We could call it
> > from a worker thread where we don't mind blocking during open.
> >
>
> I suppose that's possible, or the worker thread could perform some
> existing operation known to prime the cache. I don't think it's worth
> getting into without a concrete example, however.
You mean like EXT4_IOC_PRECACHE_EXTENTS?
You know, that ioctl that the ext4 googlers needed to add because
they already had AIO applications that depend on it and they hadn't
realised that the could do exactly the same thing with a FIEMAP
call? i.e. this call to count the number of extents in the file:
struct fiemap fm = {
.offset = 0,
.length = FIEMAP_MAX_OFFSET,
};
res = ioctl(fd, FS_IOC_FIEMAP, &fm);
will cause XFS to read in the extent map and cache it.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-01 23:41 ` Dave Chinner
@ 2015-12-02 8:23 ` Avi Kivity
0 siblings, 0 replies; 58+ messages in thread
From: Avi Kivity @ 2015-12-02 8:23 UTC (permalink / raw)
To: Dave Chinner; +Cc: Brian Foster, Glauber Costa, xfs
On 12/02/2015 01:41 AM, Dave Chinner wrote:
> On Tue, Dec 01, 2015 at 10:56:01PM +0200, Avi Kivity wrote:
>> On 12/01/2015 10:45 PM, Dave Chinner wrote:
>>> On Tue, Dec 01, 2015 at 09:01:13AM -0500, Glauber Costa wrote:
>>> The difference is an allocation can block waiting on IO, and the
>>> CPU can then go off and run another process, which then tries to do
>>> an allocation. So you might only have 4 CPUs, but a workload that
>>> can have a hundred active allocations at once (not uncommon in
>>> file server workloads).
>> But for us, probably not much more. We try to restrict active I/Os
>> to the effective disk queue depth (more than that and they just turn
>> sour waiting in the disk queue).
>>
>>
>>> On worklaods that are roughly 1 process per CPU, it's typical that
>>> agcount = 2 * N cpus gives pretty good results on large filesystems.
>> This is probably using sync calls. Using async calls you can have
>> many more I/Os in progress (but still limited by effective disk
>> queue depth).
> Ah, no. Even with async IO you don't want unbound allocation
> concurrency.
Unbound, certainly not.
But if my disk want 100 concurrent operations to deliver maximum
bandwidth, and XFS wants fewer concurrent allocations to satisfy some
internal constraint, then I can't satisfy both.
To be fair, the number 100 was measured for 4k reads. It's sure to be
much lower for 128k writes, and since we set an extent size hint of 1MB,
only 1/8th of those will be allocating. So I expect things to work in
practice, at least with the current generation of disks. Unfortunately
disk bandwidth is growing faster than latency is improving, which means
that the effective concurrency is increasing.
> The allocation algorithms rely on having contiguous
> free space extents that are much larger than the allocations being
> done to work effeectively and minimise file fragmentation. If you
> chop the filesystem up into lots of small AGs, then it accelerates
> the rate at which the free space gets chopped up into smaller
> extents and performance then suffers. It's the same problem as
> running a large filesystem near ENOSPC for an extended period of
> time, which again is something we most definitely don't recommend
> you do in production systems.
I understand. I guess it makes ag randomization even more important,
for our use case.
What happens when an ag fills up? Can a file overflow to another ag?
>
>>> If you've got 400GB filesystems or you are using spinning disks,
>>> then you probably don't want to go above 16 AGs, because then you
>>> have problems with maintaining continugous free space and you'll
>>> seek the spinning disks to death....
>> We're concentrating on SSDs for now.
> Sure, so "problems with maintaining continugous free space" is what
> you need to be concerned about.
Right. Luckily our allocation patterns are very friendly towards that.
We have append-only files that grow rapidly, then are immutable for a
time, then are deleted. (It is a log-structured database so a natual fit
for SSDs).
We can increase our extent size hint if it will help the SSD any.
>
>>>>>> 'mount -o ikeep,'
>>>>> Interesting. Our files are large so we could try this.
>>> Keep in mind that ikeep means that inode allocation permanently
>>> fragments free space, which can affect how large files are allocated
>>> once you truncate/rm the original files.
>> We can try to prime this by allocating a lot of inodes up front,
>> then removing them, so that this doesn't happen.
> Again - what problem have you measured that inode preallocation will
> solves in your application? Don't make changes just because you
> *think* it will fix what you *think* is a problem. Measure, analyse,
> solve, in that order.
We are now investigating what we can do to fix the problem, we aren't
committing to any solution yet. Certainly we plan to be certain of what
the problem is before we fix it.
Up until a few days ago we never saw any blocks with XFS, and were very
happy -- but that was with 90us, 450k IOPS disks. With the slower
disks, accessed through a certain hypervisor, we do see XFS block, and
it is very worrying.
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-02 0:13 ` Brian Foster
2015-12-02 0:57 ` Dave Chinner
@ 2015-12-02 8:34 ` Avi Kivity
2015-12-08 6:03 ` Dave Chinner
1 sibling, 1 reply; 58+ messages in thread
From: Avi Kivity @ 2015-12-02 8:34 UTC (permalink / raw)
To: Brian Foster; +Cc: Glauber Costa, xfs
On 12/02/2015 02:13 AM, Brian Foster wrote:
> On Tue, Dec 01, 2015 at 09:26:42PM +0200, Avi Kivity wrote:
>> On 12/01/2015 08:51 PM, Brian Foster wrote:
>>> On Tue, Dec 01, 2015 at 07:09:29PM +0200, Avi Kivity wrote:
>>>> On 12/01/2015 06:29 PM, Brian Foster wrote:
>>>>> On Tue, Dec 01, 2015 at 06:08:51PM +0200, Avi Kivity wrote:
>>>>>> On 12/01/2015 06:01 PM, Brian Foster wrote:
>>>>>>> On Tue, Dec 01, 2015 at 05:22:38PM +0200, Avi Kivity wrote:
>>>>>>>> On 12/01/2015 04:56 PM, Brian Foster wrote:
>>>>>>>>> On Tue, Dec 01, 2015 at 03:58:28PM +0200, Avi Kivity wrote:
>>>>>>>>>> On 12/01/2015 03:11 PM, Brian Foster wrote:
>>>>>>>>>>> On Tue, Dec 01, 2015 at 11:08:47AM +0200, Avi Kivity wrote:
>>>>>>>>>>>> On 11/30/2015 06:14 PM, Brian Foster wrote:
>>>>>>>>>>>>> On Mon, Nov 30, 2015 at 04:29:13PM +0200, Avi Kivity wrote:
>>>>>>>>>>>>>> On 11/30/2015 04:10 PM, Brian Foster wrote:
> ...
>>>> The case of waiting for I/O is much more worrying, because I/O latency are
>>>> much higher. But it seems like most of the DIO path does not trigger
>>>> locking around I/O (and we are careful to avoid the ones that do, like
>>>> writing beyond eof).
>>>>
>>>> (sorry for repeating myself, I have the feeling we are talking past each
>>>> other and want to be on the same page)
>>>>
>>> Yeah, my point is just that just because the thread blocked on I/O,
>>> doesn't mean the cpu can't carry on with some useful work for another
>>> task.
>> In our case, there is no other task. We run one thread per logical core, so
>> if that thread gets blocked, the cpu idles.
>>
>> The whole point of io_submit() is to issue an I/O and let the caller
>> continue processing immediately. It is the equivalent of O_NONBLOCK for
>> networking code. If O_NONBLOCK did block from time to time, practically all
>> modern network applications would see a huge performance drop.
>>
> Ok, but my understanding is that O_NONBLOCK would return an error code
> in the blocking case such that userspace can do something else or retry
> from a blockable context.
I did not mean the exact equivalent, but in the spirit of allowing a
thread to perform an I/O task (networking or file I/O) in parallel with
computation.
For networking, returning an error is fine because there exists a
notification (epoll) to tell userspace when a retry would succeed. For
file I/O, there isn't one. Still, returning an error is better than
nothing because then, as you say, you can retry in a blockable context.
> I think this is similar to what hch posted wrt
> to the pwrite2() bits for nonblocking buffered I/O or what I was asking
> about earlier on with regard to returning an error if some blocking
> would otherwise occur.
Yes. Anything except silently blocking!
>
>>>>>>> We submit an I/O which is
>>>>>>> asynchronous in nature and wait on a completion, which causes the cpu to
>>>>>>> schedule and execute another task until the completion is set by I/O
>>>>>>> completion (via an async callback). At that point, the issuing thread
>>>>>>> continues where it left off. I suspect I'm missing something... can you
>>>>>>> elaborate on what you'd do differently here (and how it helps)?
>>>>>> Just apply the same technique everywhere: convert locks to trylock +
>>>>>> schedule a continuation on failure.
>>>>>>
>>>>> I'm certainly not an expert on the kernel scheduling, locking and
>>>>> serialization mechanisms, but my understanding is that most things
>>>>> outside of spin locks are reschedule points. For example, the
>>>>> wait_for_completion() calls XFS uses to wait on I/O boil down to
>>>>> schedule_timeout() calls. Buffer locks are implemented as semaphores and
>>>>> down() can end up in the same place.
>>>> But, for the most part, XFS seems to be able to avoid sleeping. The call to
>>>> __blockdev_direct_IO only launches the I/O, so any locking is only around
>>>> cpu operations and, unless there is contention, won't cause us to sleep in
>>>> io_submit().
>>>>
>>>> Trying to follow the code, it looks like xfs_get_blocks_direct (and
>>>> __blockdev_direct_IO's get_block parameter in general) is synchronous, so
>>>> we're just lucky to have everything in cache. If it isn't, we block right
>>>> there. I really hope I'm misreading this and some other magic is happening
>>>> elsewhere instead of this.
>>>>
>>> Nope, it's synchronous from a code perspective. The
>>> xfs_bmapi_read()->xfs_iread_extents() path could have to read in the
>>> inode bmap metadata if it hasn't been done already. Note that this
>>> should only happen once as everything is stored in-core, so in most
>>> cases this is skipped. It's also possible extents are read in via some
>>> other path/operation on the inode before an async I/O happens to be
>>> submitted (e.g., see some of the other xfs_bmapi_read() callers).
>> Is there (could we add) some ioctl to prime this cache? We could call it
>> from a worker thread where we don't mind blocking during open.
>>
> I suppose that's possible, or the worker thread could perform some
> existing operation known to prime the cache. I don't think it's worth
> getting into without a concrete example, however. The extent read
> example we're batting around might not ever be a problem (as you've
> noted due to file size), if files are truncated and recycled, for
> example.
>
>> What is the eviction policy for this cache? Is it simply the block
>> device's page cache?
>>
> IIUC the extent list stays around until the inode is reclaimed. There's
> a separate buffer cache for metadata buffers. Both types of objects
> would be reclaimed based on memory pressure.
It comes down to size of disk, size of memory, and average file size. I
expect that with current disk and memory sizes the metadata is quite
small, so this might not be a problem, and even a cold start would
self-prime in a reasonably short time.
>
>> What about the write path, will we see the same problems there? I would
>> guess the problem is less severe there if the metadata is written with
>> writeback policy.
>>
> Metadata is modified in-core and handed off to the logging
> infrastructure via a transaction. The log is flushed to disk some time
> later and metadata writeback occurs asynchronously via the xfsaild
> thread.
Unless, I expect, if the log is full. Since we're hammering on the disk
quite heavily, the log would be fighting with user I/O and possibly losing.
Does XFS throttle user I/O in order to get the log buffers recycled faster?
Is there any way for us to keep track of it, and reduce disk pressure
when it gets full?
Oh you answered that already, /sys/fs/xfs/device/log/*.
>
> Brian
>
>>> Either way, the extents have to be read in at some point and I'd expect
>>> that cpu to schedule onto some other task while that thread waits on I/O
>>> to complete (read-ahead could also be a factor here, but I haven't
>>> really dug into how that is triggered for buffers).
>> To provide an example, our application, which is a database, faces this
>> problem exact at a higher level. Data is stored in data files, and data
>> items' locations are stored in index files. When we read a bit of data, we
>> issue an index read, and pass it a continuation to be executed when the read
>> completes. This latter continuation parses the data and passes it to the
>> code that prepares it for merging with data from other data files, and an
>> eventual return to the user.
>>
>> Having written code for over a year in this style, I've come to expect it to
>> be used everywhere asynchronous I/O is used, but I realize it is fairly hard
>> without good support from a framework that allows continuations to be
>> composed in a natural way.
>>
>>
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-02 0:57 ` Dave Chinner
@ 2015-12-02 8:38 ` Avi Kivity
0 siblings, 0 replies; 58+ messages in thread
From: Avi Kivity @ 2015-12-02 8:38 UTC (permalink / raw)
To: Dave Chinner, Brian Foster; +Cc: Glauber Costa, xfs
On 12/02/2015 02:57 AM, Dave Chinner wrote:
> On Tue, Dec 01, 2015 at 07:13:29PM -0500, Brian Foster wrote:
>> On Tue, Dec 01, 2015 at 09:26:42PM +0200, Avi Kivity wrote:
>>> On 12/01/2015 08:51 PM, Brian Foster wrote:
>>>> On Tue, Dec 01, 2015 at 07:09:29PM +0200, Avi Kivity wrote:
>>>> Nope, it's synchronous from a code perspective. The
>>>> xfs_bmapi_read()->xfs_iread_extents() path could have to read in the
>>>> inode bmap metadata if it hasn't been done already. Note that this
>>>> should only happen once as everything is stored in-core, so in most
>>>> cases this is skipped. It's also possible extents are read in via some
>>>> other path/operation on the inode before an async I/O happens to be
>>>> submitted (e.g., see some of the other xfs_bmapi_read() callers).
>>> Is there (could we add) some ioctl to prime this cache? We could call it
>>> from a worker thread where we don't mind blocking during open.
>>>
>> I suppose that's possible, or the worker thread could perform some
>> existing operation known to prime the cache. I don't think it's worth
>> getting into without a concrete example, however.
> You mean like EXT4_IOC_PRECACHE_EXTENTS?
>
> You know, that ioctl that the ext4 googlers needed to add because
> they already had AIO applications that depend on it and they hadn't
> realised that the could do exactly the same thing with a FIEMAP
> call? i.e. this call to count the number of extents in the file:
>
> struct fiemap fm = {
> .offset = 0,
> .length = FIEMAP_MAX_OFFSET,
> };
>
> res = ioctl(fd, FS_IOC_FIEMAP, &fm);
>
> will cause XFS to read in the extent map and cache it.
>
Cool, it even appears to be callable with CAP_WHATEVER. So we would use
this to prime the metadata caches before startup, if they turn out to be
a problem in practice.
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-01 23:06 ` Dave Chinner
@ 2015-12-02 9:02 ` Avi Kivity
2015-12-02 12:57 ` Carlos Maiolino
2015-12-02 23:19 ` Dave Chinner
0 siblings, 2 replies; 58+ messages in thread
From: Avi Kivity @ 2015-12-02 9:02 UTC (permalink / raw)
To: Dave Chinner; +Cc: Glauber Costa, xfs
On 12/02/2015 01:06 AM, Dave Chinner wrote:
> On Tue, Dec 01, 2015 at 11:38:29PM +0200, Avi Kivity wrote:
>> On 12/01/2015 11:19 PM, Dave Chinner wrote:
>>> On Tue, Dec 01, 2015 at 09:07:14PM +0200, Avi Kivity wrote:
>>>> On 12/01/2015 08:03 PM, Carlos Maiolino wrote:
>>>>> Hi Avi,
>>>>>
>>>>>>> else is going to execute in our place until this thread can make
>>>>>>> progress.
>>>>>> For us, nothing else can execute in our place, we usually have exactly one
>>>>>> thread per logical core. So we are heavily dependent on io_submit not
>>>>>> sleeping.
>>>>>>
>>>>>> The case of a contended lock is, to me, less worrying. It can be reduced by
>>>>>> using more allocation groups, which is apparently the shared resource under
>>>>>> contention.
>>>>>>
>>>>> I apologize if I misread your previous comments, but, IIRC you said you can't
>>>>> change the directory structure your application is using, and IIRC your
>>>>> application does not spread files across several directories.
>>>> I miswrote somewhat: the application writes data files and commitlog
>>>> files. The data file directory structure is fixed due to
>>>> compatibility concerns (it is not a single directory, but some
>>>> workloads will see most access on files in a single directory. The
>>>> commitlog directory structure is more relaxed, and we can split it
>>>> to a directory per shard (=cpu) or something else.
>>>>
>>>> If worst comes to worst, we'll hack around this and distribute the
>>>> data files into more directories, and provide some hack for
>>>> compatibility.
>>>>
>>>>> XFS spread files across the allocation groups, based on the directory these
>>>>> files are created,
>>>> Idea: create the files in some subdirectory, and immediately move
>>>> them to their required location.
>>> See xfs_fsr.
>> Can you elaborate? I don't see how it is applicable.
> Just pointing out that this is what xfs_fsr does to control locality
> of allocation for files it is defragmenting. Except that rather than
> moving files, it uses XFS_IOC_SWAPEXT to switch the data between two
> inodes atomically...
Ok, thanks.
>
>> My hack involves creating the file in a random directory, and while
>> it is still zero sized, move it to its final directory. This is
>> simply to defeat the ag selection heuristic.
> Which you really don't want to do.
Why not? For my directory structure, files in the same directory do not
share temporal locality. What does the ag selection heuristic give me?
>
>>>>> trying to keep files as close as possible from their
>>>>> metadata.
>>>> This is pointless for an SSD. Perhaps XFS should randomize the ag on
>>>> nonrotational media instead.
>>> Actually, no, it is not pointless. SSDs do not require optimisation
>>> for minimal seek time, but data locality is still just as important
>>> as spinning disks, if not moreso. Why? Because the garbage
>>> collection routines in the SSDs are all about locality and we can't
>>> drive garbage collection effectively via discard operations if the
>>> filesystem is not keeping temporally related files close together in
>>> it's block address space.
>> In my case, files in the same directory are not temporally related.
>> But I understand where the heuristic comes from.
>>
>> Maybe an ioctl to set a directory attribute "the files in this
>> directory are not temporally related"?
> And exactly what does that gain us?
I have a directory with commitlog files that are constantly and rapidly
being created, appended to, and removed, from all logical cores in the
system. Does this not put pressure on that allocation group's locks?
> Exactly what problem are you
> trying to solve by manipulating file locality that can't be solved
> by existing knobs and config options?
I admit I don't know much about the existing knobs and config options.
Pointers are appreciated.
>
> Perhaps you'd like to read up on how the inode32 allocator behaves?
Indeed I would, pointers are appreciated.
>
>>> e.g. If the files in a directory are all close together, and the
>>> directory is removed, we then leave a big empty contiguous region in
>>> the filesystem free space map, and when we send discards over that
>>> we end up with a single big trim and the drive handles that far more
>> Would this not be defeated if a directory that happens to share the
>> allocation group gets populated simultaneously?
> Sure. But this sort of thing is rare in the real world, and when
> they do occur, it generally only takes small tweaks to algorithms
> and layouts make them go away. I don't care to bikeshed about
> theoretical problems - I'm in the business of finding the root cause
> of the problems users are having and solving those problems. So far
> what you've given us is a ball of "there's blocking in AIO
> submission", and the only one that is clear cut is the timestamp
> update.
>
> Go back and categorise the types of blocking that you are seeing -
> whether it be on the AGIs during inode manipulation, one the AGFs
> becuse of concurrent extent allocation, on log forces because of
> slow discards in the transcation completion, on the transaction
> subsystem because of a lack of log space for concurrent
> reservations, etc. And then determine if changing the layout of the
> filesystem (e.g. number of AGs, size of log, etc) and different
> mount options (e.g. turning off discard, using inode32 allocator,
> etc) make any difference to the blocking issues you are seeing.
>
> Once we know which of the different algorithms is causing the
> blocking issues, we'll know a lot more about why we're having
> problems and a better idea of what problems we actually need to
> solve.
I'm happy to hack off the lowest hanging fruit and then go after the
next one. I understand you're annoyed at having to defend against what
may be non-problems; but for me it is an opportunity to learn about the
file system. For us it is the weakest spot in our system, because on
the one hand we heavily depend on async behavior and on the other hand
Linux is notoriously bad at it. So we are very nervous when blocking
happens.
>
>>> effectively than lots of little trims (i.e. one per file) that the
>>> drive cannot do anything useful with because they are all smaller
>>> than the internal SSD page/block sizes and so get ignored. This is
>>> one of the reasons fstrim is so much more efficient and effective
>>> than using the discard mount option.
>> In my use case, the files are fairly large, and there is constant
>> rewriting (not in-place: files are read, merged, and written back).
>> So I'm worried an fstrim can happen too late.
> Have you measured the SSD performance degradation over time due to
> large overwrites? If not, then again it is a good chance you are
> trying to solve a theoretical problem rather than a real problem....
>
I'm not worried about that (maybe I should be) but about the SSD
reaching internal ENOSPC due to the fstrim happening too late.
Consider this scenario, which is quite typical for us:
1. Fill 1/3rd of the disk with a few large files.
2. Copy/merge the data into a new file, occupying another 1/3rd of the disk.
3. Repeat 1+2.
If this is repeated few times, the disk can see 100% of its space
occupied (depending on how free space is allocated), even if from a
user's perspective it is never more than 2/3rds full.
Maybe a simple countermeasure is to issue an fstrim every time we write
10%-20% of the disk's capacity.
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-02 9:02 ` Avi Kivity
@ 2015-12-02 12:57 ` Carlos Maiolino
2015-12-02 23:19 ` Dave Chinner
1 sibling, 0 replies; 58+ messages in thread
From: Carlos Maiolino @ 2015-12-02 12:57 UTC (permalink / raw)
To: Avi Kivity; +Cc: Glauber Costa, xfs
> >>>>compatibility.
> >>>>
> >>>>>XFS spread files across the allocation groups, based on the directory these
> >>>>>files are created,
> >>>>Idea: create the files in some subdirectory, and immediately move
> >>>>them to their required location.
> >>>See xfs_fsr.
> >>Can you elaborate? I don't see how it is applicable.
> >Just pointing out that this is what xfs_fsr does to control locality
> >of allocation for files it is defragmenting. Except that rather than
> >moving files, it uses XFS_IOC_SWAPEXT to switch the data between two
> >inodes atomically...
>
> Ok, thanks.
>
> >
> >>My hack involves creating the file in a random directory, and while
> >>it is still zero sized, move it to its final directory. This is
> >>simply to defeat the ag selection heuristic.
> >Which you really don't want to do.
>
> Why not? For my directory structure, files in the same directory do not
> share temporal locality. What does the ag selection heuristic give me?
>
>
>
> >
> >>>>> trying to keep files as close as possible from their
> >>>>>metadata.
> >>>>This is pointless for an SSD. Perhaps XFS should randomize the ag on
> >>>>nonrotational media instead.
> >>>Actually, no, it is not pointless. SSDs do not require optimisation
> >>>for minimal seek time, but data locality is still just as important
> >>>as spinning disks, if not moreso. Why? Because the garbage
> >>>collection routines in the SSDs are all about locality and we can't
> >>>drive garbage collection effectively via discard operations if the
> >>>filesystem is not keeping temporally related files close together in
> >>>it's block address space.
> >>In my case, files in the same directory are not temporally related.
> >>But I understand where the heuristic comes from.
> >>
> >>Maybe an ioctl to set a directory attribute "the files in this
> >>directory are not temporally related"?
> >And exactly what does that gain us?
>
> I have a directory with commitlog files that are constantly and rapidly
> being created, appended to, and removed, from all logical cores in the
> system. Does this not put pressure on that allocation group's locks?
>
> >Exactly what problem are you
> >trying to solve by manipulating file locality that can't be solved
> >by existing knobs and config options?
>
> I admit I don't know much about the existing knobs and config options.
> Pointers are appreciated.
>
>
> >
> >Perhaps you'd like to read up on how the inode32 allocator behaves?
>
> Indeed I would, pointers are appreciated.
>
inode32 mount option limit inode allocations to the first AG of the
filesystem, instead of allocate inodes across the whole AGs.
This exists basically because since inode numbers are based on where the inode
is allocated, inodes allocated beyond the first TB of the filesystem, will have
64bit inode numbers, which, in case, might cause compatibility problems with
applications which are not able to read 64bit inode numbers.
> >
> >>>e.g. If the files in a directory are all close together, and the
> >>>directory is removed, we then leave a big empty contiguous region in
> >>>the filesystem free space map, and when we send discards over that
> >>>we end up with a single big trim and the drive handles that far more
> >>Would this not be defeated if a directory that happens to share the
> >>allocation group gets populated simultaneously?
> >Sure. But this sort of thing is rare in the real world, and when
> >they do occur, it generally only takes small tweaks to algorithms
> >and layouts make them go away. I don't care to bikeshed about
> >theoretical problems - I'm in the business of finding the root cause
> >of the problems users are having and solving those problems. So far
> >what you've given us is a ball of "there's blocking in AIO
> >submission", and the only one that is clear cut is the timestamp
> >update.
> >
> >Go back and categorise the types of blocking that you are seeing -
> >whether it be on the AGIs during inode manipulation, one the AGFs
> >becuse of concurrent extent allocation, on log forces because of
> >slow discards in the transcation completion, on the transaction
> >subsystem because of a lack of log space for concurrent
> >reservations, etc. And then determine if changing the layout of the
> >filesystem (e.g. number of AGs, size of log, etc) and different
> >mount options (e.g. turning off discard, using inode32 allocator,
> >etc) make any difference to the blocking issues you are seeing.
> >
> >Once we know which of the different algorithms is causing the
> >blocking issues, we'll know a lot more about why we're having
> >problems and a better idea of what problems we actually need to
> >solve.
>
> I'm happy to hack off the lowest hanging fruit and then go after the next
> one. I understand you're annoyed at having to defend against what may be
> non-problems; but for me it is an opportunity to learn about the file
> system. For us it is the weakest spot in our system, because on the one
> hand we heavily depend on async behavior and on the other hand Linux is
> notoriously bad at it. So we are very nervous when blocking happens.
>
> >
> >>>effectively than lots of little trims (i.e. one per file) that the
> >>>drive cannot do anything useful with because they are all smaller
> >>>than the internal SSD page/block sizes and so get ignored. This is
> >>>one of the reasons fstrim is so much more efficient and effective
> >>>than using the discard mount option.
> >>In my use case, the files are fairly large, and there is constant
> >>rewriting (not in-place: files are read, merged, and written back).
> >>So I'm worried an fstrim can happen too late.
> >Have you measured the SSD performance degradation over time due to
> >large overwrites? If not, then again it is a good chance you are
> >trying to solve a theoretical problem rather than a real problem....
> >
>
> I'm not worried about that (maybe I should be) but about the SSD reaching
> internal ENOSPC due to the fstrim happening too late.
>
> Consider this scenario, which is quite typical for us:
>
> 1. Fill 1/3rd of the disk with a few large files.
> 2. Copy/merge the data into a new file, occupying another 1/3rd of the disk.
> 3. Repeat 1+2.
>
> If this is repeated few times, the disk can see 100% of its space occupied
> (depending on how free space is allocated), even if from a user's
> perspective it is never more than 2/3rds full.
>
> Maybe a simple countermeasure is to issue an fstrim every time we write
> 10%-20% of the disk's capacity.
>
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
--
Carlos
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-02 9:02 ` Avi Kivity
2015-12-02 12:57 ` Carlos Maiolino
@ 2015-12-02 23:19 ` Dave Chinner
2015-12-03 12:52 ` Avi Kivity
1 sibling, 1 reply; 58+ messages in thread
From: Dave Chinner @ 2015-12-02 23:19 UTC (permalink / raw)
To: Avi Kivity; +Cc: Glauber Costa, xfs
On Wed, Dec 02, 2015 at 11:02:08AM +0200, Avi Kivity wrote:
> On 12/02/2015 01:06 AM, Dave Chinner wrote:
> >On Tue, Dec 01, 2015 at 11:38:29PM +0200, Avi Kivity wrote:
> >>On 12/01/2015 11:19 PM, Dave Chinner wrote:
> >>>>>XFS spread files across the allocation groups, based on the directory these
> >>>>>files are created,
> >>>>Idea: create the files in some subdirectory, and immediately move
> >>>>them to their required location.
....
> >>My hack involves creating the file in a random directory, and while
> >>it is still zero sized, move it to its final directory. This is
> >>simply to defeat the ag selection heuristic.
> >Which you really don't want to do.
>
> Why not? For my directory structure, files in the same directory do
> not share temporal locality. What does the ag selection heuristic
> give me?
Wrong question. The right question is this: what problems does
subverting the AG selection heuristic cause me?
If you can't answer that question, then you can't quantify the risks
involved with making such a behavioural change.
> >>>>> trying to keep files as close as possible from their
> >>>>>metadata.
> >>>>This is pointless for an SSD. Perhaps XFS should randomize the ag on
> >>>>nonrotational media instead.
> >>>Actually, no, it is not pointless. SSDs do not require optimisation
> >>>for minimal seek time, but data locality is still just as important
> >>>as spinning disks, if not moreso. Why? Because the garbage
> >>>collection routines in the SSDs are all about locality and we can't
> >>>drive garbage collection effectively via discard operations if the
> >>>filesystem is not keeping temporally related files close together in
> >>>it's block address space.
> >>In my case, files in the same directory are not temporally related.
> >>But I understand where the heuristic comes from.
> >>
> >>Maybe an ioctl to set a directory attribute "the files in this
> >>directory are not temporally related"?
> >And exactly what does that gain us?
>
> I have a directory with commitlog files that are constantly and
> rapidly being created, appended to, and removed, from all logical
> cores in the system. Does this not put pressure on that allocation
> group's locks?
Not usually, because if an AG is contended, the allocation algorithm
skips the contended AG and selects the next uncontended AG to
allocate in. And given that the append algorithm used by the
allocator attempts to use the last block of the last extent as the
target for the new extent (i.e. contiguous allocation) once a file
has skipped to a different AG all allocations will continue in that
new AG until it is either full or it becomes contended....
IOWs, when AG contention occurs, the filesystem automatically
spreads out the load over multiple AGs. Put simply, we optimise for
locality first, but we're willing to compromise on locality to
minimise contention when it occurs. But, also, keep in mind that
in minimising contention we are still selecting the most local of
possible alternatives, and that's something you can't do in
userspace....
> >Exactly what problem are you
> >trying to solve by manipulating file locality that can't be solved
> >by existing knobs and config options?
>
> I admit I don't know much about the existing knobs and config
> options. Pointers are appreciated.
You can find some work in progress here:
https://git.kernel.org/cgit/fs/xfs/xfs-documentation.git/
looks like there's some problem with xfs.org wiki, so the links
to the user/training info on this page:
http://xfs.org/index.php/XFS_Papers_and_Documentation
aren't working.
> >Perhaps you'd like to read up on how the inode32 allocator behaves?
>
> Indeed I would, pointers are appreciated.
Inode allocation section here:
https://git.kernel.org/cgit/fs/xfs/xfs-documentation.git/tree/admin/XFS_Performance_Tuning/filesystem_tunables.asciidoc
> >Once we know which of the different algorithms is causing the
> >blocking issues, we'll know a lot more about why we're having
> >problems and a better idea of what problems we actually need to
> >solve.
>
> I'm happy to hack off the lowest hanging fruit and then go after the
> next one. I understand you're annoyed at having to defend against
> what may be non-problems; but for me it is an opportunity to learn
> about the file system.
No, I'm not annoyed. I just don't want to be chasing ghosts and so
we need to be on the same page about how to track down these issues.
And, beleive me, you'll learn a lot about how the filesystem behaves
just by watching how the different configs react to the same
input...
> For us it is the weakest spot in our system,
> because on the one hand we heavily depend on async behavior and on
> the other hand Linux is notoriously bad at it. So we are very
> nervous when blocking happens.
I can't disagree with you there - we really need to fix what we can
within the constraints of the OS first, then we once we have it
working as well as we can, then we can look to solving the remaining
"notoriously bad" AIO problems...
> >>>effectively than lots of little trims (i.e. one per file) that the
> >>>drive cannot do anything useful with because they are all smaller
> >>>than the internal SSD page/block sizes and so get ignored. This is
> >>>one of the reasons fstrim is so much more efficient and effective
> >>>than using the discard mount option.
> >>In my use case, the files are fairly large, and there is constant
> >>rewriting (not in-place: files are read, merged, and written back).
> >>So I'm worried an fstrim can happen too late.
> >Have you measured the SSD performance degradation over time due to
> >large overwrites? If not, then again it is a good chance you are
> >trying to solve a theoretical problem rather than a real problem....
> >
>
> I'm not worried about that (maybe I should be) but about the SSD
> reaching internal ENOSPC due to the fstrim happening too late.
>
> Consider this scenario, which is quite typical for us:
>
> 1. Fill 1/3rd of the disk with a few large files.
> 2. Copy/merge the data into a new file, occupying another 1/3rd of the disk.
> 3. Repeat 1+2.
>
> If this is repeated few times, the disk can see 100% of its space
> occupied (depending on how free space is allocated), even if from a
> user's perspective it is never more than 2/3rds full.
I don't think that's true. SSD behaviour largely depends on how much
of the LBA space has been written to (i.e. marked used) and so that
metric tends to determine how the SSD behaves under such workloads.
This is one of the reasons that overprovisioning SSD space (e.g.
leaving 25% of the LBA space completely unused) results in better
performance under overwrite workloads - there's lots more scratch
space for the garbage collector to work with...
Hence as long as the filesystem is reusing the same LBA regions for
the files, TRIM will probably not make a significant difference to
performance because there's still 1/3rd of the LBA region that is
"unused". Hence the overwrites go into the unused 1/3rd of the SSD,
and the underlying SSD blocks associated with the "overwritten" LBA
region are immediately marked free, just like if you issued a trim
for that region before you start the overwrite.
With the way the XFS allocator works, it fills AGs from lowest to
highest blocks, and if you free lots of space down low in the AG
then that tends to get reused before the higher offset free space.
hence the XFS allocates space in the above workload would result in
roughly 1/3rd of the LBA space associated with the filesystem
remaining unused. This is another allocator behaviour designed for
spinning disks (to keep the data on the faster outer edges of
drives) that maps very well to internal SSD allocation/reclaim
algorithms....
FWIW, did you know that TRIM generally doesn't return the disk to
the performance of a pristine, empty disk? Generally only a secure
erase will guarantee that a SSD returns to "empty disk" performance,
but that also removes all data from then entire SSD. Hence the
baseline "sustained performance" you should be using is not "empty
disk" performance, but the performance once the disk has been
overwritten completely at least once. Only them will you tend to see
what effect TRIM will actually have.
> Maybe a simple countermeasure is to issue an fstrim every time we
> write 10%-20% of the disk's capacity.
Run the workload to steady state performance and measure the
degradation as it continues to run and overwrite the SSDs
repeatedly. To do this properly you are going to have to sacrifice
some SSDs, because you're going to need to overwrite them quite a
few times to get an idea of the degradation characteristics and
whether a periodic trim makes any difference or not.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-02 23:19 ` Dave Chinner
@ 2015-12-03 12:52 ` Avi Kivity
2015-12-04 3:16 ` Dave Chinner
0 siblings, 1 reply; 58+ messages in thread
From: Avi Kivity @ 2015-12-03 12:52 UTC (permalink / raw)
To: Dave Chinner; +Cc: Glauber Costa, xfs
On 12/03/2015 01:19 AM, Dave Chinner wrote:
> On Wed, Dec 02, 2015 at 11:02:08AM +0200, Avi Kivity wrote:
>> On 12/02/2015 01:06 AM, Dave Chinner wrote:
>>> On Tue, Dec 01, 2015 at 11:38:29PM +0200, Avi Kivity wrote:
>>>> On 12/01/2015 11:19 PM, Dave Chinner wrote:
>>>>>>> XFS spread files across the allocation groups, based on the directory these
>>>>>>> files are created,
>>>>>> Idea: create the files in some subdirectory, and immediately move
>>>>>> them to their required location.
> ....
>>>> My hack involves creating the file in a random directory, and while
>>>> it is still zero sized, move it to its final directory. This is
>>>> simply to defeat the ag selection heuristic.
>>> Which you really don't want to do.
>> Why not? For my directory structure, files in the same directory do
>> not share temporal locality. What does the ag selection heuristic
>> give me?
> Wrong question. The right question is this: what problems does
> subverting the AG selection heuristic cause me?
>
> If you can't answer that question, then you can't quantify the risks
> involved with making such a behavioural change.
Okay. Any hint about the answer to that question?
>
>>>>>>> trying to keep files as close as possible from their
>>>>>>> metadata.
>>>>>> This is pointless for an SSD. Perhaps XFS should randomize the ag on
>>>>>> nonrotational media instead.
>>>>> Actually, no, it is not pointless. SSDs do not require optimisation
>>>>> for minimal seek time, but data locality is still just as important
>>>>> as spinning disks, if not moreso. Why? Because the garbage
>>>>> collection routines in the SSDs are all about locality and we can't
>>>>> drive garbage collection effectively via discard operations if the
>>>>> filesystem is not keeping temporally related files close together in
>>>>> it's block address space.
>>>> In my case, files in the same directory are not temporally related.
>>>> But I understand where the heuristic comes from.
>>>>
>>>> Maybe an ioctl to set a directory attribute "the files in this
>>>> directory are not temporally related"?
>>> And exactly what does that gain us?
>> I have a directory with commitlog files that are constantly and
>> rapidly being created, appended to, and removed, from all logical
>> cores in the system. Does this not put pressure on that allocation
>> group's locks?
> Not usually, because if an AG is contended, the allocation algorithm
> skips the contended AG and selects the next uncontended AG to
> allocate in. And given that the append algorithm used by the
> allocator attempts to use the last block of the last extent as the
> target for the new extent (i.e. contiguous allocation) once a file
> has skipped to a different AG all allocations will continue in that
> new AG until it is either full or it becomes contended....
>
> IOWs, when AG contention occurs, the filesystem automatically
> spreads out the load over multiple AGs. Put simply, we optimise for
> locality first, but we're willing to compromise on locality to
> minimise contention when it occurs. But, also, keep in mind that
> in minimising contention we are still selecting the most local of
> possible alternatives, and that's something you can't do in
> userspace....
Cool. I don't think "nearly-local" matters much for an SSD (it's either
contiguous or it is not), but it's good to know that it's self-tuning
wrt. contention.
In some good news, Glauber hacked our I/O engine not to throw so many
concurrent I/Os at the filesystem, and indeed so the contention
reduced. So it's likely we were pushing the fs so hard all the ags were
contended, but this is no longer the case.
>
>>> Exactly what problem are you
>>> trying to solve by manipulating file locality that can't be solved
>>> by existing knobs and config options?
>> I admit I don't know much about the existing knobs and config
>> options. Pointers are appreciated.
> You can find some work in progress here:
>
> https://git.kernel.org/cgit/fs/xfs/xfs-documentation.git/
>
> looks like there's some problem with xfs.org wiki, so the links
> to the user/training info on this page:
>
> http://xfs.org/index.php/XFS_Papers_and_Documentation
>
> aren't working.
>
>>> Perhaps you'd like to read up on how the inode32 allocator behaves?
>> Indeed I would, pointers are appreciated.
> Inode allocation section here:
>
> https://git.kernel.org/cgit/fs/xfs/xfs-documentation.git/tree/admin/XFS_Performance_Tuning/filesystem_tunables.asciidoc
Thanks for all the links, I'll study them and see what we can do to tune
for our workload.
>>> Once we know which of the different algorithms is causing the
>>> blocking issues, we'll know a lot more about why we're having
>>> problems and a better idea of what problems we actually need to
>>> solve.
>> I'm happy to hack off the lowest hanging fruit and then go after the
>> next one. I understand you're annoyed at having to defend against
>> what may be non-problems; but for me it is an opportunity to learn
>> about the file system.
> No, I'm not annoyed. I just don't want to be chasing ghosts and so
> we need to be on the same page about how to track down these issues.
> And, beleive me, you'll learn a lot about how the filesystem behaves
> just by watching how the different configs react to the same
> input...
Ok. Looks like I have a lot of homework.
>
>> For us it is the weakest spot in our system,
>> because on the one hand we heavily depend on async behavior and on
>> the other hand Linux is notoriously bad at it. So we are very
>> nervous when blocking happens.
> I can't disagree with you there - we really need to fix what we can
> within the constraints of the OS first, then we once we have it
> working as well as we can, then we can look to solving the remaining
> "notoriously bad" AIO problems...
There are lots of users who will be eternally grateful to you if you can
get this fixed. Linux has a very bad reputation in this area with the
accepted wisdom that you can only use aio reliably against block
devices. XFS comes very close, it will make a huge impact if it can be
used to do aio reliably, without a lot of constraints on the application.
>
>>>>> effectively than lots of little trims (i.e. one per file) that the
>>>>> drive cannot do anything useful with because they are all smaller
>>>>> than the internal SSD page/block sizes and so get ignored. This is
>>>>> one of the reasons fstrim is so much more efficient and effective
>>>>> than using the discard mount option.
>>>> In my use case, the files are fairly large, and there is constant
>>>> rewriting (not in-place: files are read, merged, and written back).
>>>> So I'm worried an fstrim can happen too late.
>>> Have you measured the SSD performance degradation over time due to
>>> large overwrites? If not, then again it is a good chance you are
>>> trying to solve a theoretical problem rather than a real problem....
>>>
>> I'm not worried about that (maybe I should be) but about the SSD
>> reaching internal ENOSPC due to the fstrim happening too late.
>>
>> Consider this scenario, which is quite typical for us:
>>
>> 1. Fill 1/3rd of the disk with a few large files.
>> 2. Copy/merge the data into a new file, occupying another 1/3rd of the disk.
>> 3. Repeat 1+2.
>>
>> If this is repeated few times, the disk can see 100% of its space
>> occupied (depending on how free space is allocated), even if from a
>> user's perspective it is never more than 2/3rds full.
> I don't think that's true. SSD behaviour largely depends on how much
> of the LBA space has been written to (i.e. marked used) and so that
> metric tends to determine how the SSD behaves under such workloads.
> This is one of the reasons that overprovisioning SSD space (e.g.
> leaving 25% of the LBA space completely unused) results in better
> performance under overwrite workloads - there's lots more scratch
> space for the garbage collector to work with...
>
> Hence as long as the filesystem is reusing the same LBA regions for
> the files, TRIM will probably not make a significant difference to
> performance because there's still 1/3rd of the LBA region that is
> "unused". Hence the overwrites go into the unused 1/3rd of the SSD,
> and the underlying SSD blocks associated with the "overwritten" LBA
> region are immediately marked free, just like if you issued a trim
> for that region before you start the overwrite.
>
> With the way the XFS allocator works, it fills AGs from lowest to
> highest blocks, and if you free lots of space down low in the AG
> then that tends to get reused before the higher offset free space.
> hence the XFS allocates space in the above workload would result in
> roughly 1/3rd of the LBA space associated with the filesystem
> remaining unused. This is another allocator behaviour designed for
> spinning disks (to keep the data on the faster outer edges of
> drives) that maps very well to internal SSD allocation/reclaim
> algorithms....
Cool. So we'll keep fstrim usage to daily, or something similarly low.
>
> FWIW, did you know that TRIM generally doesn't return the disk to
> the performance of a pristine, empty disk? Generally only a secure
> erase will guarantee that a SSD returns to "empty disk" performance,
> but that also removes all data from then entire SSD. Hence the
> baseline "sustained performance" you should be using is not "empty
> disk" performance, but the performance once the disk has been
> overwritten completely at least once. Only them will you tend to see
> what effect TRIM will actually have.
I did not know that. Maybe that's another factor in why cloud SSDs are
so slow.
>
>> Maybe a simple countermeasure is to issue an fstrim every time we
>> write 10%-20% of the disk's capacity.
> Run the workload to steady state performance and measure the
> degradation as it continues to run and overwrite the SSDs
> repeatedly. To do this properly you are going to have to sacrifice
> some SSDs, because you're going to need to overwrite them quite a
> few times to get an idea of the degradation characteristics and
> whether a periodic trim makes any difference or not.
Enterprise SSDs are guaranteed for something like N full writes / day
for several years, are they not? So such a test can take weeks or
months, depending on the ratio between disk size and bandwidth.
Still, I guess it has to be done.
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-03 12:52 ` Avi Kivity
@ 2015-12-04 3:16 ` Dave Chinner
2015-12-08 13:52 ` Avi Kivity
0 siblings, 1 reply; 58+ messages in thread
From: Dave Chinner @ 2015-12-04 3:16 UTC (permalink / raw)
To: Avi Kivity; +Cc: Glauber Costa, xfs
On Thu, Dec 03, 2015 at 02:52:08PM +0200, Avi Kivity wrote:
>
>
> On 12/03/2015 01:19 AM, Dave Chinner wrote:
> >On Wed, Dec 02, 2015 at 11:02:08AM +0200, Avi Kivity wrote:
> >>On 12/02/2015 01:06 AM, Dave Chinner wrote:
> >>>On Tue, Dec 01, 2015 at 11:38:29PM +0200, Avi Kivity wrote:
> >>>>On 12/01/2015 11:19 PM, Dave Chinner wrote:
> >>>>>>>XFS spread files across the allocation groups, based on the directory these
> >>>>>>>files are created,
> >>>>>>Idea: create the files in some subdirectory, and immediately move
> >>>>>>them to their required location.
> >....
> >>>>My hack involves creating the file in a random directory, and while
> >>>>it is still zero sized, move it to its final directory. This is
> >>>>simply to defeat the ag selection heuristic.
> >>>Which you really don't want to do.
> >>Why not? For my directory structure, files in the same directory do
> >>not share temporal locality. What does the ag selection heuristic
> >>give me?
> >Wrong question. The right question is this: what problems does
> >subverting the AG selection heuristic cause me?
> >
> >If you can't answer that question, then you can't quantify the risks
> >involved with making such a behavioural change.
>
> Okay. Any hint about the answer to that question?
If your file set is randomly distributed across the filesystem, then
it's quite likely that the filesystem will use all of the LBA space
rather than reusing the same AGs and hence LBA regions. That's going
to slowly fragment free space as metadata (which has different
lifetimes to data) and long term data gets more widely distributed.
That, in term will slowly result in the working dataset being made
up of more and smaller extents, whcih will also slowly get more
distributed over time, which them means allocation and freeing of
extents takes longer, trim becomes less effective because it's
workingwith smaller spaces, the SSD's "LBA in use" mapping becomes
more fragmented so garbage collection becomes harder, etc...
But, really, the only way to tell is to test, measure, observe and
analyse....
> >>>>>>This is pointless for an SSD. Perhaps XFS should randomize the ag on
> >>>>>>nonrotational media instead.
> >>>>>Actually, no, it is not pointless. SSDs do not require optimisation
> >>>>>for minimal seek time, but data locality is still just as important
> >>>>>as spinning disks, if not moreso. Why? Because the garbage
> >>>>>collection routines in the SSDs are all about locality and we can't
> >>>>>drive garbage collection effectively via discard operations if the
> >>>>>filesystem is not keeping temporally related files close together in
> >>>>>it's block address space.
> >>>>In my case, files in the same directory are not temporally related.
> >>>>But I understand where the heuristic comes from.
> >>>>
> >>>>Maybe an ioctl to set a directory attribute "the files in this
> >>>>directory are not temporally related"?
> >>>And exactly what does that gain us?
> >>I have a directory with commitlog files that are constantly and
> >>rapidly being created, appended to, and removed, from all logical
> >>cores in the system. Does this not put pressure on that allocation
> >>group's locks?
> >Not usually, because if an AG is contended, the allocation algorithm
> >skips the contended AG and selects the next uncontended AG to
> >allocate in. And given that the append algorithm used by the
> >allocator attempts to use the last block of the last extent as the
> >target for the new extent (i.e. contiguous allocation) once a file
> >has skipped to a different AG all allocations will continue in that
> >new AG until it is either full or it becomes contended....
> >
> >IOWs, when AG contention occurs, the filesystem automatically
> >spreads out the load over multiple AGs. Put simply, we optimise for
> >locality first, but we're willing to compromise on locality to
> >minimise contention when it occurs. But, also, keep in mind that
> >in minimising contention we are still selecting the most local of
> >possible alternatives, and that's something you can't do in
> >userspace....
>
> Cool. I don't think "nearly-local" matters much for an SSD (it's
> either contiguous or it is not), but it's good to know that it's
> self-tuning wrt. contention.
"Nearly local" matters a lot for filesystem free space management
and hence minimising the amount o LBA space the filesystem actually
uses in the long term given a relatively predicatable workload....
> In some good news, Glauber hacked our I/O engine not to throw so
> many concurrent I/Os at the filesystem, and indeed so the contention
> reduced. So it's likely we were pushing the fs so hard all the ags
> were contended, but this is no longer the case.
What is the xfs_info output of the filesystem you tested on?
> >With the way the XFS allocator works, it fills AGs from lowest to
> >highest blocks, and if you free lots of space down low in the AG
> >then that tends to get reused before the higher offset free space.
> >hence the XFS allocates space in the above workload would result in
> >roughly 1/3rd of the LBA space associated with the filesystem
> >remaining unused. This is another allocator behaviour designed for
> >spinning disks (to keep the data on the faster outer edges of
> >drives) that maps very well to internal SSD allocation/reclaim
> >algorithms....
>
> Cool. So we'll keep fstrim usage to daily, or something similarly low.
Well, it's something you'll need to monitor to determine what the
best frequency is, as even fstrim doesn't come for free (esp. if the
storage does not support queued TRIM commands).
> >FWIW, did you know that TRIM generally doesn't return the disk to
> >the performance of a pristine, empty disk? Generally only a secure
> >erase will guarantee that a SSD returns to "empty disk" performance,
> >but that also removes all data from then entire SSD. Hence the
> >baseline "sustained performance" you should be using is not "empty
> >disk" performance, but the performance once the disk has been
> >overwritten completely at least once. Only them will you tend to see
> >what effect TRIM will actually have.
>
> I did not know that. Maybe that's another factor in why cloud SSDs
> are so slow.
Have a look at the random write performance consistency graphs for
the different enterprise SSDs here:
http://www.anandtech.com/show/9430/micron-m510dc-480gb-enterprise-sata-ssd-review/3
You'll see just how different sustained write load performance is to
the empty drive performance (which is only the first few hundred
seconds of each graph) across the different drives that have been
tested. The next page has similar results for mixed random
read/write workloads....
That will give you a good idea of how the current enterprise SSDs
behave under sustained write load. It's a *lot* better than the way
the 1st and 2nd generation drives performed....
> >>write 10%-20% of the disk's capacity.
> >Run the workload to steady state performance and measure the
> >degradation as it continues to run and overwrite the SSDs
> >repeatedly. To do this properly you are going to have to sacrifice
> >some SSDs, because you're going to need to overwrite them quite a
> >few times to get an idea of the degradation characteristics and
> >whether a periodic trim makes any difference or not.
>
> Enterprise SSDs are guaranteed for something like N full writes /
> day for several years, are they not?
Yes, usually somewhere between 3-15 DWPD (Drive Writes Per Day) - it
typically works out at around 5000 full drive write cycles for
enterprise drives. However, at both the low capacity end of the
scale or the high performance end (i.e. pcie cards capable of multiple
GB/s writes), it's not uncommon to be able to burn a DW cycle in
under 10 minutes and so you can easily burn the life out of a drive
in a couple of weeks of intense testing....
> So such a test can take weeks
> or months, depending on the ratio between disk size and bandwidth.
> Still, I guess it has to be done.
*nod*
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-02 8:34 ` Avi Kivity
@ 2015-12-08 6:03 ` Dave Chinner
2015-12-08 13:56 ` Avi Kivity
0 siblings, 1 reply; 58+ messages in thread
From: Dave Chinner @ 2015-12-08 6:03 UTC (permalink / raw)
To: Avi Kivity; +Cc: Brian Foster, Glauber Costa, xfs
On Wed, Dec 02, 2015 at 10:34:14AM +0200, Avi Kivity wrote:
> On 12/02/2015 02:13 AM, Brian Foster wrote:
> >Metadata is modified in-core and handed off to the logging
> >infrastructure via a transaction. The log is flushed to disk some time
> >later and metadata writeback occurs asynchronously via the xfsaild
> >thread.
>
> Unless, I expect, if the log is full. Since we're hammering on the
> disk quite heavily, the log would be fighting with user I/O and
> possibly losing.
>
> Does XFS throttle user I/O in order to get the log buffers recycled faster?
No. XFS tags the metadata IO with REQ_META that the IO schedulers
can tell the difference between metadata and data IO, and schedule
them appropriately. Further. log buffers are also tagged with
REQ_SYNC to indicate they are latency sensitive IOs, whcih the IO
schedulers again treat differently to minimise latency in the face
of bulk async IO which is not latency sensitive.
IOWs, IO prioritisation and dispatch scheduling is the job of the IO
scheduler, not the filesystem. The filesystem just tells the
scheduler how to treat the different types of IO...
> Is there any way for us to keep track of it, and reduce disk
> pressure when it gets full?
Only if you want to make more problems for yourself - second
guessing what the filesystem is going to do will only lead you to
dancing the Charlie Foxtrot on a regular basis. :/
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-04 3:16 ` Dave Chinner
@ 2015-12-08 13:52 ` Avi Kivity
2015-12-08 23:13 ` Dave Chinner
0 siblings, 1 reply; 58+ messages in thread
From: Avi Kivity @ 2015-12-08 13:52 UTC (permalink / raw)
To: Dave Chinner; +Cc: Glauber Costa, xfs
On 12/04/2015 05:16 AM, Dave Chinner wrote:
> On Thu, Dec 03, 2015 at 02:52:08PM +0200, Avi Kivity wrote:
>>
>> On 12/03/2015 01:19 AM, Dave Chinner wrote:
>>> On Wed, Dec 02, 2015 at 11:02:08AM +0200, Avi Kivity wrote:
>>>> On 12/02/2015 01:06 AM, Dave Chinner wrote:
>>>>> On Tue, Dec 01, 2015 at 11:38:29PM +0200, Avi Kivity wrote:
>>>>>> On 12/01/2015 11:19 PM, Dave Chinner wrote:
>>>>>>>>> XFS spread files across the allocation groups, based on the directory these
>>>>>>>>> files are created,
>>>>>>>> Idea: create the files in some subdirectory, and immediately move
>>>>>>>> them to their required location.
>>> ....
>>>>>> My hack involves creating the file in a random directory, and while
>>>>>> it is still zero sized, move it to its final directory. This is
>>>>>> simply to defeat the ag selection heuristic.
>>>>> Which you really don't want to do.
>>>> Why not? For my directory structure, files in the same directory do
>>>> not share temporal locality. What does the ag selection heuristic
>>>> give me?
>>> Wrong question. The right question is this: what problems does
>>> subverting the AG selection heuristic cause me?
>>>
>>> If you can't answer that question, then you can't quantify the risks
>>> involved with making such a behavioural change.
>> Okay. Any hint about the answer to that question?
> If your file set is randomly distributed across the filesystem,
I think that happens whether or not I break the "files in the same
directory are related" heuristic, because I have many directories. It's
just that some of them get churned more than others.
> then
> it's quite likely that the filesystem will use all of the LBA space
> rather than reusing the same AGs and hence LBA regions. That's going
> to slowly fragment free space as metadata (which has different
> lifetimes to data) and long term data gets more widely distributed.
> That, in term will slowly result in the working dataset being made
> up of more and smaller extents, whcih will also slowly get more
> distributed over time, which them means allocation and freeing of
> extents takes longer, trim becomes less effective because it's
> workingwith smaller spaces, the SSD's "LBA in use" mapping becomes
> more fragmented so garbage collection becomes harder, etc...
>
> But, really, the only way to tell is to test, measure, observe and
> analyse....
Sure.
>
>>>>>>>> This is pointless for an SSD. Perhaps XFS should randomize the ag on
>>>>>>>> nonrotational media instead.
>>>>>>> Actually, no, it is not pointless. SSDs do not require optimisation
>>>>>>> for minimal seek time, but data locality is still just as important
>>>>>>> as spinning disks, if not moreso. Why? Because the garbage
>>>>>>> collection routines in the SSDs are all about locality and we can't
>>>>>>> drive garbage collection effectively via discard operations if the
>>>>>>> filesystem is not keeping temporally related files close together in
>>>>>>> it's block address space.
>>>>>> In my case, files in the same directory are not temporally related.
>>>>>> But I understand where the heuristic comes from.
>>>>>>
>>>>>> Maybe an ioctl to set a directory attribute "the files in this
>>>>>> directory are not temporally related"?
>>>>> And exactly what does that gain us?
>>>> I have a directory with commitlog files that are constantly and
>>>> rapidly being created, appended to, and removed, from all logical
>>>> cores in the system. Does this not put pressure on that allocation
>>>> group's locks?
>>> Not usually, because if an AG is contended, the allocation algorithm
>>> skips the contended AG and selects the next uncontended AG to
>>> allocate in. And given that the append algorithm used by the
>>> allocator attempts to use the last block of the last extent as the
>>> target for the new extent (i.e. contiguous allocation) once a file
>>> has skipped to a different AG all allocations will continue in that
>>> new AG until it is either full or it becomes contended....
>>>
>>> IOWs, when AG contention occurs, the filesystem automatically
>>> spreads out the load over multiple AGs. Put simply, we optimise for
>>> locality first, but we're willing to compromise on locality to
>>> minimise contention when it occurs. But, also, keep in mind that
>>> in minimising contention we are still selecting the most local of
>>> possible alternatives, and that's something you can't do in
>>> userspace....
>> Cool. I don't think "nearly-local" matters much for an SSD (it's
>> either contiguous or it is not), but it's good to know that it's
>> self-tuning wrt. contention.
> "Nearly local" matters a lot for filesystem free space management
> and hence minimising the amount o LBA space the filesystem actually
> uses in the long term given a relatively predicatable workload....
>
>> In some good news, Glauber hacked our I/O engine not to throw so
>> many concurrent I/Os at the filesystem, and indeed so the contention
>> reduced. So it's likely we were pushing the fs so hard all the ags
>> were contended, but this is no longer the case.
> What is the xfs_info output of the filesystem you tested on?
It was a cloud disk so someone else now has the pleasure...
>
>>> With the way the XFS allocator works, it fills AGs from lowest to
>>> highest blocks, and if you free lots of space down low in the AG
>>> then that tends to get reused before the higher offset free space.
>>> hence the XFS allocates space in the above workload would result in
>>> roughly 1/3rd of the LBA space associated with the filesystem
>>> remaining unused. This is another allocator behaviour designed for
>>> spinning disks (to keep the data on the faster outer edges of
>>> drives) that maps very well to internal SSD allocation/reclaim
>>> algorithms....
>> Cool. So we'll keep fstrim usage to daily, or something similarly low.
> Well, it's something you'll need to monitor to determine what the
> best frequency is, as even fstrim doesn't come for free (esp. if the
> storage does not support queued TRIM commands).
I was able to trigger a load where discard caused io_submit to sleep
even on my super-fast nvme drive.
The bad news is, disabling discard and running fstrim in parallel with
this load also caused io_submit to sleep.
>
>>> FWIW, did you know that TRIM generally doesn't return the disk to
>>> the performance of a pristine, empty disk? Generally only a secure
>>> erase will guarantee that a SSD returns to "empty disk" performance,
>>> but that also removes all data from then entire SSD. Hence the
>>> baseline "sustained performance" you should be using is not "empty
>>> disk" performance, but the performance once the disk has been
>>> overwritten completely at least once. Only them will you tend to see
>>> what effect TRIM will actually have.
>> I did not know that. Maybe that's another factor in why cloud SSDs
>> are so slow.
> Have a look at the random write performance consistency graphs for
> the different enterprise SSDs here:
>
> http://www.anandtech.com/show/9430/micron-m510dc-480gb-enterprise-sata-ssd-review/3
>
> You'll see just how different sustained write load performance is to
> the empty drive performance (which is only the first few hundred
> seconds of each graph) across the different drives that have been
> tested. The next page has similar results for mixed random
> read/write workloads....
>
> That will give you a good idea of how the current enterprise SSDs
> behave under sustained write load. It's a *lot* better than the way
> the 1st and 2nd generation drives performed....
>
>>>> write 10%-20% of the disk's capacity.
>>> Run the workload to steady state performance and measure the
>>> degradation as it continues to run and overwrite the SSDs
>>> repeatedly. To do this properly you are going to have to sacrifice
>>> some SSDs, because you're going to need to overwrite them quite a
>>> few times to get an idea of the degradation characteristics and
>>> whether a periodic trim makes any difference or not.
>> Enterprise SSDs are guaranteed for something like N full writes /
>> day for several years, are they not?
> Yes, usually somewhere between 3-15 DWPD (Drive Writes Per Day) - it
> typically works out at around 5000 full drive write cycles for
> enterprise drives. However, at both the low capacity end of the
> scale or the high performance end (i.e. pcie cards capable of multiple
> GB/s writes), it's not uncommon to be able to burn a DW cycle in
> under 10 minutes and so you can easily burn the life out of a drive
> in a couple of weeks of intense testing....
>
>> So such a test can take weeks
>> or months, depending on the ratio between disk size and bandwidth.
>> Still, I guess it has to be done.
> *nod*
>
> Cheers,
>
> Dave.
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-08 6:03 ` Dave Chinner
@ 2015-12-08 13:56 ` Avi Kivity
2015-12-08 23:32 ` Dave Chinner
0 siblings, 1 reply; 58+ messages in thread
From: Avi Kivity @ 2015-12-08 13:56 UTC (permalink / raw)
To: Dave Chinner; +Cc: Brian Foster, Glauber Costa, xfs
On 12/08/2015 08:03 AM, Dave Chinner wrote:
> On Wed, Dec 02, 2015 at 10:34:14AM +0200, Avi Kivity wrote:
>> On 12/02/2015 02:13 AM, Brian Foster wrote:
>>> Metadata is modified in-core and handed off to the logging
>>> infrastructure via a transaction. The log is flushed to disk some time
>>> later and metadata writeback occurs asynchronously via the xfsaild
>>> thread.
>> Unless, I expect, if the log is full. Since we're hammering on the
>> disk quite heavily, the log would be fighting with user I/O and
>> possibly losing.
>>
>> Does XFS throttle user I/O in order to get the log buffers recycled faster?
> No. XFS tags the metadata IO with REQ_META that the IO schedulers
> can tell the difference between metadata and data IO, and schedule
> them appropriately. Further. log buffers are also tagged with
> REQ_SYNC to indicate they are latency sensitive IOs, whcih the IO
> schedulers again treat differently to minimise latency in the face
> of bulk async IO which is not latency sensitive.
>
> IOWs, IO prioritisation and dispatch scheduling is the job of the IO
> scheduler, not the filesystem. The filesystem just tells the
> scheduler how to treat the different types of IO...
>
>> Is there any way for us to keep track of it, and reduce disk
>> pressure when it gets full?
> Only if you want to make more problems for yourself - second
> guessing what the filesystem is going to do will only lead you to
> dancing the Charlie Foxtrot on a regular basis. :/
So far the best approach I found that doesn't conflict with this is to
limit io_submit iodepth to the natural disk iodepth (or a small multiple
thereof). This seems to keep XFS in its comfort zone, and is good for
latency anyway.
The only issue is that the only way to obtain this parameter is to
measure it.
I wrote a small tool to do this [1], but it's a hassle for users.
[1] https://github.com/avikivity/diskplorer
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-08 13:52 ` Avi Kivity
@ 2015-12-08 23:13 ` Dave Chinner
0 siblings, 0 replies; 58+ messages in thread
From: Dave Chinner @ 2015-12-08 23:13 UTC (permalink / raw)
To: Avi Kivity; +Cc: Glauber Costa, xfs
On Tue, Dec 08, 2015 at 03:52:52PM +0200, Avi Kivity wrote:
> >>>With the way the XFS allocator works, it fills AGs from lowest to
> >>>highest blocks, and if you free lots of space down low in the AG
> >>>then that tends to get reused before the higher offset free space.
> >>>hence the XFS allocates space in the above workload would result in
> >>>roughly 1/3rd of the LBA space associated with the filesystem
> >>>remaining unused. This is another allocator behaviour designed for
> >>>spinning disks (to keep the data on the faster outer edges of
> >>>drives) that maps very well to internal SSD allocation/reclaim
> >>>algorithms....
> >>Cool. So we'll keep fstrim usage to daily, or something similarly low.
> >Well, it's something you'll need to monitor to determine what the
> >best frequency is, as even fstrim doesn't come for free (esp. if the
> >storage does not support queued TRIM commands).
>
> I was able to trigger a load where discard caused io_submit to sleep
> even on my super-fast nvme drive.
>
> The bad news is, disabling discard and running fstrim in parallel
> with this load also caused io_submit to sleep.
Well, yes. fstrim is not a magic bullet that /prevents/ discard
from interrupting your application's IO - it's just a method under
which the impact can be /somewhat controlled/ as it can be scheduled
for periods where the impact has minimal interruption (e.g. when
load is likely to be light, such as at 3am just before nightly
backups are run).
Regardless, it sounds like your steady state load could be described
as "throwing as much IO as we possible can at the device", but you
are then then having "blocking trouble" when maintenance (expensive)
operations like TRIM need to be are run. I'm not sure this
"blocking" can be prevented completely, because it assumes that you
have a device of infinite IO capacity.
That is, if you exceed the device's command queue depth and the IO
scheduler request queue depth, the block layer will block in the IO
scheduler waiting for a request queue slot to come free. Put simply:
if you overload the IO subsystem, it will block. There's nothing we
can do in the filesystem about this - this is the way the block
layer works, and it's architected this way to provide the necessary
feedback control for buffered write IO throttling and other
congestion control mechanisms in the kernel.
Sure, you can set the IO scheduler request queue depth to be really
deep to avoid blocking, but this then simply increases your average
and worst-case IO latency in overload situations. At some point you
have to consider the IO subsystem is overloaded and the application
driving it needs to back off. Something has to block when this
happens...
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-08 13:56 ` Avi Kivity
@ 2015-12-08 23:32 ` Dave Chinner
2015-12-09 8:37 ` Avi Kivity
0 siblings, 1 reply; 58+ messages in thread
From: Dave Chinner @ 2015-12-08 23:32 UTC (permalink / raw)
To: Avi Kivity; +Cc: Brian Foster, Glauber Costa, xfs
On Tue, Dec 08, 2015 at 03:56:52PM +0200, Avi Kivity wrote:
> On 12/08/2015 08:03 AM, Dave Chinner wrote:
> >On Wed, Dec 02, 2015 at 10:34:14AM +0200, Avi Kivity wrote:
> >>On 12/02/2015 02:13 AM, Brian Foster wrote:
> >>>Metadata is modified in-core and handed off to the logging
> >>>infrastructure via a transaction. The log is flushed to disk some time
> >>>later and metadata writeback occurs asynchronously via the xfsaild
> >>>thread.
> >>Unless, I expect, if the log is full. Since we're hammering on the
> >>disk quite heavily, the log would be fighting with user I/O and
> >>possibly losing.
> >>
> >>Does XFS throttle user I/O in order to get the log buffers recycled faster?
> >No. XFS tags the metadata IO with REQ_META that the IO schedulers
> >can tell the difference between metadata and data IO, and schedule
> >them appropriately. Further. log buffers are also tagged with
> >REQ_SYNC to indicate they are latency sensitive IOs, whcih the IO
> >schedulers again treat differently to minimise latency in the face
> >of bulk async IO which is not latency sensitive.
> >
> >IOWs, IO prioritisation and dispatch scheduling is the job of the IO
> >scheduler, not the filesystem. The filesystem just tells the
> >scheduler how to treat the different types of IO...
> >
> >>Is there any way for us to keep track of it, and reduce disk
> >>pressure when it gets full?
> >Only if you want to make more problems for yourself - second
> >guessing what the filesystem is going to do will only lead you to
> >dancing the Charlie Foxtrot on a regular basis. :/
>
> So far the best approach I found that doesn't conflict with this is
> to limit io_submit iodepth to the natural disk iodepth (or a small
> multiple thereof). This seems to keep XFS in its comfort zone, and
> is good for latency anyway.
That's pretty much what I just explained in my previous reply. ;)
> The only issue is that the only way to obtain this parameter is to
> measure it.
Yup, exactly what I've been saying ;)
However, You can get a pretty good guess on max concurrency from the
device characteristics in sysfs:
/sys/block/<dev>/queue/nr_requests
gives you the maximum IO scheduler request queue depth, and
/sys/block/<dev>/device/queue_depth
gives you the hardware command queue depth.
E.g. a random iscsi device I have attached to a test VM:
$ cat /sys/block/sdc/device/queue_depth
32
$ cat /sys/block/sdc/queue/nr_requests
127
Which means 32 physical IOs can be in flight concurrently, and the
IO scheduler will queue up to roughly another 100 discrete IOs
before it starts blocking incoming IO requests (127 is the typical
io scheduler queue depth default). That means maximum non-blocking
concurrency is going to be around 100-130 IOs in flight at once.
> I wrote a small tool to do this [1], but it's a hassle for users.
>
> [1] https://github.com/avikivity/diskplorer
I note that the NVMe device you tested in the description hits
maximum performance with concurrency at around 110-120 read IOs in
flight. :)
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
* Re: sleeps and waits during io_submit
2015-12-08 23:32 ` Dave Chinner
@ 2015-12-09 8:37 ` Avi Kivity
0 siblings, 0 replies; 58+ messages in thread
From: Avi Kivity @ 2015-12-09 8:37 UTC (permalink / raw)
To: Dave Chinner; +Cc: Brian Foster, Glauber Costa, xfs
On 12/09/2015 01:32 AM, Dave Chinner wrote:
> On Tue, Dec 08, 2015 at 03:56:52PM +0200, Avi Kivity wrote:
>> On 12/08/2015 08:03 AM, Dave Chinner wrote:
>>> On Wed, Dec 02, 2015 at 10:34:14AM +0200, Avi Kivity wrote:
>>>> On 12/02/2015 02:13 AM, Brian Foster wrote:
>>>>> Metadata is modified in-core and handed off to the logging
>>>>> infrastructure via a transaction. The log is flushed to disk some time
>>>>> later and metadata writeback occurs asynchronously via the xfsaild
>>>>> thread.
>>>> Unless, I expect, if the log is full. Since we're hammering on the
>>>> disk quite heavily, the log would be fighting with user I/O and
>>>> possibly losing.
>>>>
>>>> Does XFS throttle user I/O in order to get the log buffers recycled faster?
>>> No. XFS tags the metadata IO with REQ_META that the IO schedulers
>>> can tell the difference between metadata and data IO, and schedule
>>> them appropriately. Further. log buffers are also tagged with
>>> REQ_SYNC to indicate they are latency sensitive IOs, whcih the IO
>>> schedulers again treat differently to minimise latency in the face
>>> of bulk async IO which is not latency sensitive.
>>>
>>> IOWs, IO prioritisation and dispatch scheduling is the job of the IO
>>> scheduler, not the filesystem. The filesystem just tells the
>>> scheduler how to treat the different types of IO...
>>>
>>>> Is there any way for us to keep track of it, and reduce disk
>>>> pressure when it gets full?
>>> Only if you want to make more problems for yourself - second
>>> guessing what the filesystem is going to do will only lead you to
>>> dancing the Charlie Foxtrot on a regular basis. :/
>> So far the best approach I found that doesn't conflict with this is
>> to limit io_submit iodepth to the natural disk iodepth (or a small
>> multiple thereof). This seems to keep XFS in its comfort zone, and
>> is good for latency anyway.
> That's pretty much what I just explained in my previous reply. ;)
>
>> The only issue is that the only way to obtain this parameter is to
>> measure it.
> Yup, exactly what I've been saying ;)
>
> However, You can get a pretty good guess on max concurrency from the
> device characteristics in sysfs:
>
> /sys/block/<dev>/queue/nr_requests
That's just a fixed number. AFAICT, it isn't derived from the actual device.
"measure it" is better than nothing, but when you want to distribute
software that works out of the box and does not need extensive tuning,
it leaves something to be desired.
I'm thinking about detecting the limit dynamically (below the limit,
throughput is roughly proportional to concurrency; above the limit,
throughput is fixed while latency is proportional to concurrency). The
problem is that the measurement is very noisy, the more so because we
are two layers above the hardware, and driving it from cores that try
very hard not to communicate.
The right place to do this is the block layer.
> gives you the maximum IO scheduler request queue depth, and
>
> /sys/block/<dev>/device/queue_depth
>
> gives you the hardware command queue depth.
That's more useful, but it really describes the bus/link/protocol rather
than the device itself.
I don't have this queue_depth attribute for my nvme0n1 device (4.1.7).
>
> E.g. a random iscsi device I have attached to a test VM:
>
> $ cat /sys/block/sdc/device/queue_depth
> 32
> $ cat /sys/block/sdc/queue/nr_requests
> 127
>
> Which means 32 physical IOs can be in flight concurrently, and the
> IO scheduler will queue up to roughly another 100 discrete IOs
> before it starts blocking incoming IO requests (127 is the typical
> io scheduler queue depth default). That means maximum non-blocking
> concurrency is going to be around 100-130 IOs in flight at once.
>
>> I wrote a small tool to do this [1], but it's a hassle for users.
>>
>> [1] https://github.com/avikivity/diskplorer
> I note that the NVMe device you tested in the description hits
> maximum performance with concurrency at around 110-120 read IOs in
> flight. :)
>
>
We increased nr_requests for the test so it wouldn't block. So it's the
actual device characteristics, not an artifact of the software stack.
If you consider a RAID of these, you can easily need a few hundred
concurrent ops.
IIRC nvme's maximum iodepth is 64k.
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 58+ messages in thread
end of thread, other threads:[~2015-12-09 8:37 UTC | newest]
Thread overview: 58+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-11-28 2:43 sleeps and waits during io_submit Glauber Costa
2015-11-30 14:10 ` Brian Foster
2015-11-30 14:29 ` Avi Kivity
2015-11-30 16:14 ` Brian Foster
2015-12-01 9:08 ` Avi Kivity
2015-12-01 13:11 ` Brian Foster
2015-12-01 13:58 ` Avi Kivity
2015-12-01 14:01 ` Glauber Costa
2015-12-01 14:37 ` Avi Kivity
2015-12-01 20:45 ` Dave Chinner
2015-12-01 20:56 ` Avi Kivity
2015-12-01 23:41 ` Dave Chinner
2015-12-02 8:23 ` Avi Kivity
2015-12-01 14:56 ` Brian Foster
2015-12-01 15:22 ` Avi Kivity
2015-12-01 16:01 ` Brian Foster
2015-12-01 16:08 ` Avi Kivity
2015-12-01 16:29 ` Brian Foster
2015-12-01 17:09 ` Avi Kivity
2015-12-01 18:03 ` Carlos Maiolino
2015-12-01 19:07 ` Avi Kivity
2015-12-01 21:19 ` Dave Chinner
2015-12-01 21:38 ` Avi Kivity
2015-12-01 23:06 ` Dave Chinner
2015-12-02 9:02 ` Avi Kivity
2015-12-02 12:57 ` Carlos Maiolino
2015-12-02 23:19 ` Dave Chinner
2015-12-03 12:52 ` Avi Kivity
2015-12-04 3:16 ` Dave Chinner
2015-12-08 13:52 ` Avi Kivity
2015-12-08 23:13 ` Dave Chinner
2015-12-01 18:51 ` Brian Foster
2015-12-01 19:07 ` Glauber Costa
2015-12-01 19:35 ` Brian Foster
2015-12-01 19:45 ` Avi Kivity
2015-12-01 19:26 ` Avi Kivity
2015-12-01 19:41 ` Christoph Hellwig
2015-12-01 19:50 ` Avi Kivity
2015-12-02 0:13 ` Brian Foster
2015-12-02 0:57 ` Dave Chinner
2015-12-02 8:38 ` Avi Kivity
2015-12-02 8:34 ` Avi Kivity
2015-12-08 6:03 ` Dave Chinner
2015-12-08 13:56 ` Avi Kivity
2015-12-08 23:32 ` Dave Chinner
2015-12-09 8:37 ` Avi Kivity
2015-12-01 21:04 ` Dave Chinner
2015-12-01 21:10 ` Glauber Costa
2015-12-01 21:39 ` Dave Chinner
2015-12-01 21:24 ` Avi Kivity
2015-12-01 21:31 ` Glauber Costa
2015-11-30 15:49 ` Glauber Costa
2015-12-01 13:11 ` Brian Foster
2015-12-01 13:39 ` Glauber Costa
2015-12-01 14:02 ` Brian Foster
2015-11-30 23:10 ` Dave Chinner
2015-11-30 23:51 ` Glauber Costa
2015-12-01 20:30 ` Dave Chinner
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.