* linux-next memleak after IO on dax mountpoint
@ 2016-05-27 8:46 Xiong Zhou
2016-05-28 4:05 ` Xiong Zhou
0 siblings, 1 reply; 8+ messages in thread
From: Xiong Zhou @ 2016-05-27 8:46 UTC (permalink / raw)
To: sfr, axboe, linux-next, linux-nvdimm; +Cc: linux-kernel
Hi,
Reporting an oom/memleak issue in linux-next tree:
#Description:
dbench invokes oom-killer, make host unavaiable.
dbench was doing IO on nvdimm device mounted fs with dax mount option.
It happens on both xfs and ext4 filesystems.
It does not happen testing without dax mountoption.
Seems like memleak keep happening untill system run out of memory. On
good kernels, memory get freed after every dbench run.
#Hardware
lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 48
On-line CPU(s) list: 0-47
Thread(s) per core: 2
Core(s) per socket: 12
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 63
Model name: Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
Stepping: 2
CPU MHz: 2596.781
BogoMIPS: 5200.05
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 30720K
NUMA node0 CPU(s): 0-11,24-35
NUMA node1 CPU(s): 12-23,36-47
free -g
total used free shared buff/cache available
Mem: 31 0 30 0 0 30
Swap: 9 0 9
#Version:
Since next-20150517 tree, till latest 0526 tree.
0516 tree survives testing.
dbench version 4.00 - Copyright Andrew Tridgell 1999-2004
#How reproducible:
always
#Reproduce steps:
Repeating fstests[1] generic/241, 30 times, which maybe is relative
to system total ram.
#bisect info
Bisect point to this commit:
commit d6cab70166b5bf2cbeec0c566e51725c793e3aed
Merge: cb9553d 661806a
Author: Stephen Rothwell <sfr@canb.auug.org.au>
Date: Tue May 17 11:18:34 2016 +1000
Merge remote-tracking branch 'block/for-next'
which is forwarding to this:
commit 661806a319890962aaa839dc1dbf7ea356aa6b92
Merge: 1335822 b3a834b
Author: Jens Axboe <axboe@fb.com>
Date: Mon May 16 09:55:01 2016 -0600
Merge branch 'for-4.7/core' into for-next
On top of 0517 tree, reset --hard to commit cb9553d passed testing,
while reset --hard to commit d6cab70 reproduced issue.
Still working on to id which commit in this merge causes this issuer,
i noticed that lots of merge were going on there....
Meminfo, config, oom msg, bisect log are attached.
Thanks,
Xiong
[1] http://oss.sgi.com/cgi-bin/gitweb.cgi?p=xfs/cmds/xfstests.git
# Additional info
meminfo after running generic/241 7~8 rounds:
------ bad --------------
[root@host linux]# free
total used free shared buff/cache available
Mem: 32807280 422756 25782116 9456 6602408 26044756
Swap: 10485756 0 10485756
[root@host linux]# free -g
total used free shared buff/cache available
Mem: 31 0 24 0 6 24
Swap: 9 0 9
[root@host linux]# git log --oneline -1
d6cab70 Merge remote-tracking branch 'block/for-next'
[root@host linux]# echo 1 > /proc/sys/vm/drop_caches
[root@host linux]# echo 2 > /proc/sys/vm/drop_caches
[root@host linux]# echo 3 > /proc/sys/vm/drop_caches
[root@host linux]# free -g
total used free shared buff/cache available
Mem: 31 0 23 0 6 23
Swap: 9 0 9
[root@host linux]# free
total used free shared buff/cache available
Mem: 32807280 419576 25050868 9456 7336836 24938336
Swap: 10485756 0 10485756
-------- good ------------
[root@host linux]# free
total used free shared buff/cache available
Mem: 32807280 421316 30425892 9464 1960072 31884116
Swap: 10485756 0 10485756
[root@host linux]# free -g
total used free shared buff/cache available
Mem: 31 0 29 0 1 30
Swap: 9 0 9
[root@host linux]# echo 1 > /proc/sys/vm/drop_caches
[root@host linux]# echo 2 > /proc/sys/vm/drop_caches
[root@host linux]# echo 3 > /proc/sys/vm/drop_caches
[root@host linux]# free -g
total used free shared buff/cache available
Mem: 31 0 30 0 0 30
Swap: 9 0 9
[root@host linux]# free
total used free shared buff/cache available
Mem: 32807280 410820 32070196 9464 326264 31946400
Swap: 10485756 0 10485756
[root@host linux]# git log --oneline -1
cb9553d Merge remote-tracking branch 'input/next'
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: linux-next memleak after IO on dax mountpoint
2016-05-27 8:46 linux-next memleak after IO on dax mountpoint Xiong Zhou
@ 2016-05-28 4:05 ` Xiong Zhou
2016-06-01 14:37 ` Jeff Moyer
2016-06-02 15:22 ` David Drysdale
0 siblings, 2 replies; 8+ messages in thread
From: Xiong Zhou @ 2016-05-28 4:05 UTC (permalink / raw)
To: sfr, axboe, linux-next, linux-nvdimm; +Cc: linux-kernel
On Fri, May 27, 2016 at 04:46:17PM +0800, Xiong Zhou wrote:
...
> Still working on to id which commit in this merge causes this issuer,
Narrowed down to:
37e5823 block: add offset in blk_add_request_payload()
e048948 blk-mq: Export tagset iter function
58b4560 nvme: add helper nvme_map_len()
03b5929 nvme: rewrite discard support
8093f7c nvme: add helper nvme_setup_cmd()
21f033f NVMe: Skip async events for degraded controllers
82b4552 nvme: Use blk-mq helper for IO termination
93e9d8e block: add ability to flag write back caching on a device
519a7e1 dm: switch to using blk_queue_write_cache()
bb8d261 nvme: introduce a controller state machine
92911a5 nvme: tighten up state check for namespace scanning
5955be2 nvme: move namespace scanning to core
f866fc4 nvme: move AER handling to common code
0bf77e9 nvme: switch to RCU freeing the namespace
9082e87 block: remove struct bio_batch
38f2525 block: add __blkdev_issue_discard
57aac2f lightnvm: fix "warning: ‘ret’ may be used uninitialized"
ecfb40c lightnvm: handle submit_io failure
1145e63 lightnvm: implement nvm_submit_ppa_list
22e8c97 lightnvm: move block fold outside of get_bb_tbl()
7f7c5d0 lightnvm: avoid memory leak when lun_map kcalloc fails
5136061 lightnvm: introduce nvm_for_each_lun_ppa() macro
e11903f lightnvm: refactor device ops->get_bb_tbl()
5ebc7d9 lightnvm: make nvm_set_rqd_ppalist() aware of vblks
a63d5cf lightnvm: move responsibility for bad blk mgmt to target
00ee6cc lightnvm: refactor set_bb_tbl for accepting ppa list
003fad3 lightnvm: enable metadata to be sent to device
04a8aa1 lightnvm: expose gennvm_mark_blk to targets
These commits can not be reverted cleanly.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: linux-next memleak after IO on dax mountpoint
2016-05-28 4:05 ` Xiong Zhou
@ 2016-06-01 14:37 ` Jeff Moyer
2016-06-02 2:20 ` Xiong Zhou
2016-06-02 15:22 ` David Drysdale
1 sibling, 1 reply; 8+ messages in thread
From: Jeff Moyer @ 2016-06-01 14:37 UTC (permalink / raw)
To: Xiong Zhou; +Cc: sfr, axboe, linux-next, linux-nvdimm, linux-kernel
Xiong Zhou <xzhou@redhat.com> writes:
> On Fri, May 27, 2016 at 04:46:17PM +0800, Xiong Zhou wrote:
> ...
>> Still working on to id which commit in this merge causes this issuer,
>
> Narrowed down to:
Hi Xiong,
I don't see how any of those could be at all relevant to pmem. Can you
post the output from 'cat /proc/slabinfo' here?
Thanks,
Jeff
>
> 37e5823 block: add offset in blk_add_request_payload()
> e048948 blk-mq: Export tagset iter function
> 58b4560 nvme: add helper nvme_map_len()
> 03b5929 nvme: rewrite discard support
> 8093f7c nvme: add helper nvme_setup_cmd()
> 21f033f NVMe: Skip async events for degraded controllers
> 82b4552 nvme: Use blk-mq helper for IO termination
> 93e9d8e block: add ability to flag write back caching on a device
> 519a7e1 dm: switch to using blk_queue_write_cache()
> bb8d261 nvme: introduce a controller state machine
> 92911a5 nvme: tighten up state check for namespace scanning
> 5955be2 nvme: move namespace scanning to core
> f866fc4 nvme: move AER handling to common code
> 0bf77e9 nvme: switch to RCU freeing the namespace
> 9082e87 block: remove struct bio_batch
> 38f2525 block: add __blkdev_issue_discard
> 57aac2f lightnvm: fix "warning: ‘ret’ may be used uninitialized"
> ecfb40c lightnvm: handle submit_io failure
> 1145e63 lightnvm: implement nvm_submit_ppa_list
> 22e8c97 lightnvm: move block fold outside of get_bb_tbl()
> 7f7c5d0 lightnvm: avoid memory leak when lun_map kcalloc fails
> 5136061 lightnvm: introduce nvm_for_each_lun_ppa() macro
> e11903f lightnvm: refactor device ops->get_bb_tbl()
> 5ebc7d9 lightnvm: make nvm_set_rqd_ppalist() aware of vblks
> a63d5cf lightnvm: move responsibility for bad blk mgmt to target
> 00ee6cc lightnvm: refactor set_bb_tbl for accepting ppa list
> 003fad3 lightnvm: enable metadata to be sent to device
> 04a8aa1 lightnvm: expose gennvm_mark_blk to targets
>
>
> These commits can not be reverted cleanly.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: linux-next memleak after IO on dax mountpoint
2016-06-01 14:37 ` Jeff Moyer
@ 2016-06-02 2:20 ` Xiong Zhou
0 siblings, 0 replies; 8+ messages in thread
From: Xiong Zhou @ 2016-06-02 2:20 UTC (permalink / raw)
To: Jeff Moyer; +Cc: Xiong Zhou, sfr, axboe, linux-next, linux-nvdimm, linux-kernel
Hi, Jeff
On Wed, Jun 01, 2016 at 10:37:26AM -0400, Jeff Moyer wrote:
> Xiong Zhou <xzhou@redhat.com> writes:
>
> > On Fri, May 27, 2016 at 04:46:17PM +0800, Xiong Zhou wrote:
> > ...
> >> Still working on to id which commit in this merge causes this issuer,
> >
> > Narrowed down to:
>
> Hi Xiong,
>
> I don't see how any of those could be at all relevant to pmem. Can you
> post the output from 'cat /proc/slabinfo' here?
Sure. It's now reproducible in Linus tree. i'm still hunting the
bad commit.. bisecting in Linus tree point to ext4 merge commit,
but checking out to its following commit passed the test. Commits
in my last email may also not be relevant to pmem as you said.
Thanks,
Xiong
# free
total used free shared buff/cache available
Mem: 32806940 403300 18237768 9492 14165872 18062772
Swap: 10485756 0 10485756
# cat /proc/slabinfo
slabinfo - version: 2.1
# name <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
ext4_groupinfo_4k 2296 2296 144 28 1 : tunables 0 0 0 : slabdata 82 82 0
ext4_inode_cache 4154 4402 1056 31 8 : tunables 0 0 0 : slabdata 142 142 0
ext4_allocation_context 1184 1184 128 32 1 : tunables 0 0 0 : slabdata 37 37 0
ext4_io_end 5184 5184 64 64 1 : tunables 0 0 0 : slabdata 81 81 0
ext4_extent_status 23562 23562 40 102 1 : tunables 0 0 0 : slabdata 231 231 0
jbd2_journal_handle 3145 3145 48 85 1 : tunables 0 0 0 : slabdata 37 37 0
jbd2_journal_head 6290 6290 120 34 1 : tunables 0 0 0 : slabdata 185 185 0
jbd2_revoke_table_s 6144 6144 16 256 1 : tunables 0 0 0 : slabdata 24 24 0
jbd2_revoke_record_s 4096 4096 32 128 1 : tunables 0 0 0 : slabdata 32 32 0
mbcache 0 0 56 73 1 : tunables 0 0 0 : slabdata 0 0 0
nf_conntrack 918 918 320 51 4 : tunables 0 0 0 : slabdata 18 18 0
kcopyd_job 0 0 3312 9 8 : tunables 0 0 0 : slabdata 0 0 0
dm_uevent 0 0 2632 12 8 : tunables 0 0 0 : slabdata 0 0 0
kvm_async_pf 0 0 136 30 1 : tunables 0 0 0 : slabdata 0 0 0
kvm_vcpu 0 0 18880 1 8 : tunables 0 0 0 : slabdata 0 0 0
kvm_mmu_page_header 0 0 168 48 2 : tunables 0 0 0 : slabdata 0 0 0
fat_inode_cache 46 46 704 46 8 : tunables 0 0 0 : slabdata 1 1 0
fat_cache 0 0 40 102 1 : tunables 0 0 0 : slabdata 0 0 0
nfsd4_stateids 0 0 176 46 2 : tunables 0 0 0 : slabdata 0 0 0
nfsd4_openowners 0 0 440 37 4 : tunables 0 0 0 : slabdata 0 0 0
rpc_inode_cache 51 51 640 51 8 : tunables 0 0 0 : slabdata 1 1 0
xfs_dqtrx 0 0 528 31 4 : tunables 0 0 0 : slabdata 0 0 0
xfs_dquot 0 0 472 34 4 : tunables 0 0 0 : slabdata 0 0 0
xfs_icr 0 0 144 28 1 : tunables 0 0 0 : slabdata 0 0 0
xfs_ili 2809 2809 152 53 2 : tunables 0 0 0 : slabdata 53 53 0
xfs_inode 10200 10200 960 34 8 : tunables 0 0 0 : slabdata 300 300 0
xfs_efd_item 1840 1840 400 40 4 : tunables 0 0 0 : slabdata 46 46 0
xfs_da_state 1598 1598 480 34 4 : tunables 0 0 0 : slabdata 47 47 0
xfs_log_ticket 2156 2156 184 44 2 : tunables 0 0 0 : slabdata 49 49 0
bio-1 1734 1734 320 51 4 : tunables 0 0 0 : slabdata 34 34 0
RAWv6 1316 1316 1152 28 8 : tunables 0 0 0 : slabdata 47 47 0
UDPLITEv6 0 0 1152 28 8 : tunables 0 0 0 : slabdata 0 0 0
UDPv6 644 644 1152 28 8 : tunables 0 0 0 : slabdata 23 23 0
tw_sock_TCPv6 0 0 280 29 2 : tunables 0 0 0 : slabdata 0 0 0
request_sock_TCPv6 0 0 328 49 4 : tunables 0 0 0 : slabdata 0 0 0
TCPv6 300 300 2112 15 8 : tunables 0 0 0 : slabdata 20 20 0
cfq_queue 1925 1925 232 35 2 : tunables 0 0 0 : slabdata 55 55 0
bsg_cmd 0 0 312 52 4 : tunables 0 0 0 : slabdata 0 0 0
mqueue_inode_cache 36 36 896 36 8 : tunables 0 0 0 : slabdata 1 1 0
userfaultfd_ctx_cache 0 0 128 32 1 : tunables 0 0 0 : slabdata 0 0 0
dio 51 51 640 51 8 : tunables 0 0 0 : slabdata 1 1 0
pid_namespace 0 0 2224 14 8 : tunables 0 0 0 : slabdata 0 0 0
posix_timers_cache 0 0 240 34 2 : tunables 0 0 0 : slabdata 0 0 0
ip4-frags 0 0 216 37 2 : tunables 0 0 0 : slabdata 0 0 0
UDP-Lite 0 0 1024 32 8 : tunables 0 0 0 : slabdata 0 0 0
flow_cache 0 0 112 36 1 : tunables 0 0 0 : slabdata 0 0 0
RAW 1564 1564 960 34 8 : tunables 0 0 0 : slabdata 46 46 0
UDP 992 992 1024 32 8 : tunables 0 0 0 : slabdata 31 31 0
tw_sock_TCP 0 0 280 29 2 : tunables 0 0 0 : slabdata 0 0 0
request_sock_TCP 98 98 328 49 4 : tunables 0 0 0 : slabdata 2 2 0
TCP 352 352 1984 16 8 : tunables 0 0 0 : slabdata 22 22 0
hugetlbfs_inode_cache 56 56 584 28 4 : tunables 0 0 0 : slabdata 2 2 0
dquot 2784 2880 256 32 2 : tunables 0 0 0 : slabdata 90 90 0
scsi_data_buffer 128180 128180 24 170 1 : tunables 0 0 0 : slabdata 754 754 0
request_queue 85 98 2296 14 8 : tunables 0 0 0 : slabdata 7 7 0
blkdev_requests 2464 2464 368 44 4 : tunables 0 0 0 : slabdata 56 56 0
blkdev_ioc 1908 2691 104 39 1 : tunables 0 0 0 : slabdata 69 69 0
user_namespace 0 0 304 53 4 : tunables 0 0 0 : slabdata 0 0 0
dmaengine-unmap-256 15 15 2112 15 8 : tunables 0 0 0 : slabdata 1 1 0
dmaengine-unmap-128 30 30 1088 30 8 : tunables 0 0 0 : slabdata 1 1 0
sock_inode_cache 2907 2907 640 51 8 : tunables 0 0 0 : slabdata 57 57 0
file_lock_cache 1872 1872 208 39 2 : tunables 0 0 0 : slabdata 48 48 0
net_namespace 5 5 6272 5 8 : tunables 0 0 0 : slabdata 1 1 0
shmem_inode_cache 3283 3283 656 49 8 : tunables 0 0 0 : slabdata 67 67 0
taskstats 1323 1323 328 49 4 : tunables 0 0 0 : slabdata 27 27 0
proc_inode_cache 10459 10660 624 52 8 : tunables 0 0 0 : slabdata 205 205 0
sigqueue 2448 2448 160 51 2 : tunables 0 0 0 : slabdata 48 48 0
bdev_cache 507 507 832 39 8 : tunables 0 0 0 : slabdata 13 13 0
kernfs_node_cache 53652 53652 120 34 1 : tunables 0 0 0 : slabdata 1578 1578 0
mnt_cache 3990 3990 384 42 4 : tunables 0 0 0 : slabdata 95 95 0
inode_cache 25055 25144 568 28 4 : tunables 0 0 0 : slabdata 898 898 0
dentry 62696 62748 192 42 2 : tunables 0 0 0 : slabdata 1494 1494 0
iint_cache 0 0 72 56 1 : tunables 0 0 0 : slabdata 0 0 0
buffer_head 15412 15795 104 39 1 : tunables 0 0 0 : slabdata 405 405 0
mm_struct 960 960 1984 16 8 : tunables 0 0 0 : slabdata 60 60 0
files_cache 2346 2346 704 46 8 : tunables 0 0 0 : slabdata 51 51 0
signal_cache 2190 2190 1088 30 8 : tunables 0 0 0 : slabdata 73 73 0
sighand_cache 1515 1515 2112 15 8 : tunables 0 0 0 : slabdata 101 101 0
task_struct 937 970 5568 5 8 : tunables 0 0 0 : slabdata 194 194 0
cred_jar 24880 25620 192 42 2 : tunables 0 0 0 : slabdata 610 610 0
Acpi-Operand 13104 13104 72 56 1 : tunables 0 0 0 : slabdata 234 234 0
Acpi-Parse 2555 2555 56 73 1 : tunables 0 0 0 : slabdata 35 35 0
Acpi-State 57589 57681 80 51 1 : tunables 0 0 0 : slabdata 1131 1131 0
Acpi-Namespace 5202 5202 40 102 1 : tunables 0 0 0 : slabdata 51 51 0
anon_vma_chain 30664 30912 64 64 1 : tunables 0 0 0 : slabdata 483 483 0
anon_vma 30855 30855 80 51 1 : tunables 0 0 0 : slabdata 605 605 0
pid 2240 2240 128 32 1 : tunables 0 0 0 : slabdata 70 70 0
numa_policy 62 62 264 31 2 : tunables 0 0 0 : slabdata 2 2 0
radix_tree_node 7618 11704 584 28 4 : tunables 0 0 0 : slabdata 418 418 0
trace_event_file 2898 2898 88 46 1 : tunables 0 0 0 : slabdata 63 63 0
ftrace_event_field 10625 10625 48 85 1 : tunables 0 0 0 : slabdata 125 125 0
idr_layer_cache 1815 1815 2096 15 8 : tunables 0 0 0 : slabdata 121 121 0
task_group 1764 1764 768 42 8 : tunables 0 0 0 : slabdata 42 42 0
dma-kmalloc-8192 0 0 8192 4 8 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-4096 0 0 4096 8 8 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-2048 0 0 2048 16 8 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-1024 0 0 1024 32 8 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-512 32 32 512 32 4 : tunables 0 0 0 : slabdata 1 1 0
dma-kmalloc-256 0 0 256 32 2 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-128 0 0 128 32 1 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-64 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-32 0 0 32 128 1 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-16 0 0 16 256 1 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-8 0 0 8 512 1 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-192 0 0 192 42 2 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-96 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0
kmalloc-8192 1452 1488 8192 4 8 : tunables 0 0 0 : slabdata 372 372 0
kmalloc-4096 2584 2584 4096 8 8 : tunables 0 0 0 : slabdata 323 323 0
kmalloc-2048 3004480 3004496 2048 16 8 : tunables 0 0 0 : slabdata 187781 187781 0
kmalloc-1024 2079488 2079488 1024 32 8 : tunables 0 0 0 : slabdata 64984 64984 0
kmalloc-512 5550 5696 512 32 4 : tunables 0 0 0 : slabdata 178 178 0
kmalloc-256 23440835 23445568 256 32 2 : tunables 0 0 0 : slabdata 732674 732674 0
kmalloc-192 7061 7266 192 42 2 : tunables 0 0 0 : slabdata 173 173 0
kmalloc-128 6208 6208 128 32 1 : tunables 0 0 0 : slabdata 194 194 0
kmalloc-96 8606 8694 96 42 1 : tunables 0 0 0 : slabdata 207 207 0
kmalloc-64 74624 74624 64 64 1 : tunables 0 0 0 : slabdata 1166 1166 0
kmalloc-32 130277 130688 32 128 1 : tunables 0 0 0 : slabdata 1021 1021 0
kmalloc-16 64871 66304 16 256 1 : tunables 0 0 0 : slabdata 259 259 0
kmalloc-8 108567 109056 8 512 1 : tunables 0 0 0 : slabdata 213 213 0
kmem_cache_node 954 960 64 64 1 : tunables 0 0 0 : slabdata 15 15 0
kmem_cache 663 663 320 51 4 : tunables 0 0 0 : slabdata 13 13 0
# uname -r
4.7.0-rc1+
# cat /proc/meminfo
MemTotal: 32806940 kB
MemFree: 18239888 kB
MemAvailable: 18064956 kB
Buffers: 44 kB
Cached: 53740 kB
SwapCached: 0 kB
Active: 96800 kB
Inactive: 36988 kB
Active(anon): 79888 kB
Inactive(anon): 9488 kB
Active(file): 16912 kB
Inactive(file): 27500 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 10485756 kB
SwapFree: 10485756 kB
Dirty: 4 kB
Writeback: 0 kB
AnonPages: 79952 kB
Mapped: 41884 kB
Shmem: 9492 kB
Slab: 14112188 kB
SReclaimable: 61224 kB
SUnreclaim: 14050964 kB
KernelStack: 10768 kB
PageTables: 4880 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 26889224 kB
Committed_AS: 346524 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
HardwareCorrupted: 0 kB
AnonHugePages: 22528 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 317948 kB
DirectMap2M: 3743744 kB
DirectMap1G: 31457280 kB
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: linux-next memleak after IO on dax mountpoint
2016-05-28 4:05 ` Xiong Zhou
2016-06-01 14:37 ` Jeff Moyer
@ 2016-06-02 15:22 ` David Drysdale
2016-06-03 6:35 ` linux-next/linux " Xiong Zhou
2016-06-03 16:48 ` Re: linux-next " Bart Van Assche
1 sibling, 2 replies; 8+ messages in thread
From: David Drysdale @ 2016-06-02 15:22 UTC (permalink / raw)
To: Xiong Zhou
Cc: Stephen Rothwell, axboe, linux-next, linux-nvdimm, linux-kernel
On Sat, May 28, 2016 at 5:05 AM, Xiong Zhou <xzhou@redhat.com> wrote:
> On Fri, May 27, 2016 at 04:46:17PM +0800, Xiong Zhou wrote:
> ...
>> Still working on to id which commit in this merge causes this issuer,
>
> Narrowed down to:
>
> 37e5823 block: add offset in blk_add_request_payload()
> e048948 blk-mq: Export tagset iter function
> 58b4560 nvme: add helper nvme_map_len()
> 03b5929 nvme: rewrite discard support
> 8093f7c nvme: add helper nvme_setup_cmd()
> 21f033f NVMe: Skip async events for degraded controllers
> 82b4552 nvme: Use blk-mq helper for IO termination
> 93e9d8e block: add ability to flag write back caching on a device
> 519a7e1 dm: switch to using blk_queue_write_cache()
> bb8d261 nvme: introduce a controller state machine
> 92911a5 nvme: tighten up state check for namespace scanning
> 5955be2 nvme: move namespace scanning to core
> f866fc4 nvme: move AER handling to common code
> 0bf77e9 nvme: switch to RCU freeing the namespace
> 9082e87 block: remove struct bio_batch
FWIW, I'm also seeing kmemleak report a leak with v4.7-rc1, in
a different scenario (just normal desktop use). Not done much
digging so far, but this commit (9082e87bf) looks like it might be
relevant -- lots of the following:
unreferenced object 0xffff8800c288e900 (size 256):
comm "dconf-service", pid 1461, jiffies 4294895636 (age 48.028s)
hex dump (first 32 bytes):
00 00 00 00 00 00 00 00 c0 a4 c0 c6 00 88 ff ff ................
02 20 00 20 00 00 00 00 11 00 00 00 00 00 00 00 . . ............
backtrace:
[<ffffffff81955228>] kmemleak_alloc+0x28/0x50
[<ffffffff81268bdc>] kmem_cache_alloc+0xfc/0x360
[<ffffffff81203275>] mempool_alloc_slab+0x15/0x20
[<ffffffff812030de>] mempool_alloc+0x6e/0x170
[<ffffffff815014e8>] bio_alloc_bioset+0xb8/0x230
[<ffffffff81514174>] next_bio+0x24/0x50
[<ffffffff815145ef>] blkdev_issue_zeroout+0xdf/0x1d0
[<ffffffff8132ce79>] ext4_issue_zeroout+0x39/0x50
[<ffffffff81357abf>] ext4_ext_zeroout+0x2f/0x40
[<ffffffff8135ece0>] ext4_ext_map_blocks+0x1870/0x2190
[<ffffffff8132cfa1>] ext4_map_blocks+0x111/0x620
[<ffffffff81330dc8>] ext4_writepages+0x7c8/0x10a0
[<ffffffff81211851>] do_writepages+0x21/0x30
[<ffffffff812012ba>] __filemap_fdatawrite_range+0xaa/0xf0
[<ffffffff812013fd>] filemap_write_and_wait_range+0x2d/0x70
[<ffffffff81326f6d>] ext4_sync_file+0x18d/0x500
> 38f2525 block: add __blkdev_issue_discard
> 57aac2f lightnvm: fix "warning: ‘ret’ may be used uninitialized"
> ecfb40c lightnvm: handle submit_io failure
> 1145e63 lightnvm: implement nvm_submit_ppa_list
> 22e8c97 lightnvm: move block fold outside of get_bb_tbl()
> 7f7c5d0 lightnvm: avoid memory leak when lun_map kcalloc fails
> 5136061 lightnvm: introduce nvm_for_each_lun_ppa() macro
> e11903f lightnvm: refactor device ops->get_bb_tbl()
> 5ebc7d9 lightnvm: make nvm_set_rqd_ppalist() aware of vblks
> a63d5cf lightnvm: move responsibility for bad blk mgmt to target
> 00ee6cc lightnvm: refactor set_bb_tbl for accepting ppa list
> 003fad3 lightnvm: enable metadata to be sent to device
> 04a8aa1 lightnvm: expose gennvm_mark_blk to targets
>
>
> These commits can not be reverted cleanly.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: linux-next/linux memleak after IO on dax mountpoint
2016-06-02 15:22 ` David Drysdale
@ 2016-06-03 6:35 ` Xiong Zhou
2016-06-03 16:48 ` Re: linux-next " Bart Van Assche
1 sibling, 0 replies; 8+ messages in thread
From: Xiong Zhou @ 2016-06-03 6:35 UTC (permalink / raw)
To: David Drysdale, hch
Cc: Xiong Zhou, Stephen Rothwell, axboe, linux-next, linux-nvdimm,
linux-kernel
On Thu, Jun 02, 2016 at 04:22:37PM +0100, David Drysdale wrote:
> On Sat, May 28, 2016 at 5:05 AM, Xiong Zhou <xzhou@redhat.com> wrote:
> > On Fri, May 27, 2016 at 04:46:17PM +0800, Xiong Zhou wrote:
> > ...
> >> Still working on to id which commit in this merge causes this issuer,
> >
> > Narrowed down to:
> >
> > 37e5823 block: add offset in blk_add_request_payload()
...
> > 0bf77e9 nvme: switch to RCU freeing the namespace
> > 9082e87 block: remove struct bio_batch
>
> FWIW, I'm also seeing kmemleak report a leak with v4.7-rc1, in
> a different scenario (just normal desktop use). Not done much
> digging so far, but this commit (9082e87bf) looks like it might be
> relevant
<snip..>
Thanks for the tip, accelerated my searching in the block merge.
On top of v4.7-rc1 , in order to revert this commit cleanly:
9082e87 block: remove struct bio_batch
, have to revert these two:
bbd848e0f block: reinstate early return of -EOPNOTSUPP from blkdev_issue_discard
38f2525 block: add __blkdev_issue_discard
, then have to revert these two dependances:
202bae5 dm thin: unroll issue_discard() to create longer discard bio chains
3dba53a9 dm thin: use __blkdev_issue_discard for async discard support
With all these five commits reverted, NO memleak happens.
Reverting other four commits while not reverting 9082e87, memleak
happens.
Thanks,
Xiong
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Re: linux-next memleak after IO on dax mountpoint
2016-06-02 15:22 ` David Drysdale
2016-06-03 6:35 ` linux-next/linux " Xiong Zhou
@ 2016-06-03 16:48 ` Bart Van Assche
2016-06-07 10:26 ` linux-next/linux " Xiong Zhou
1 sibling, 1 reply; 8+ messages in thread
From: Bart Van Assche @ 2016-06-03 16:48 UTC (permalink / raw)
To: David Drysdale, Xiong Zhou
Cc: Stephen Rothwell, axboe, linux-next, linux-nvdimm, linux-kernel,
Christoph Hellwig
On 06/02/2016 08:22 AM, David Drysdale wrote:
> FWIW, I'm also seeing kmemleak report a leak with v4.7-rc1, in
> a different scenario (just normal desktop use). Not done much
> digging so far, but this commit (9082e87bf) looks like it might be
> relevant -- lots of the following:
>
> unreferenced object 0xffff8800c288e900 (size 256):
> comm "dconf-service", pid 1461, jiffies 4294895636 (age 48.028s)
> hex dump (first 32 bytes):
> 00 00 00 00 00 00 00 00 c0 a4 c0 c6 00 88 ff ff ................
> 02 20 00 20 00 00 00 00 11 00 00 00 00 00 00 00 . . ............
> backtrace:
> [<ffffffff81955228>] kmemleak_alloc+0x28/0x50
> [<ffffffff81268bdc>] kmem_cache_alloc+0xfc/0x360
> [<ffffffff81203275>] mempool_alloc_slab+0x15/0x20
> [<ffffffff812030de>] mempool_alloc+0x6e/0x170
> [<ffffffff815014e8>] bio_alloc_bioset+0xb8/0x230
> [<ffffffff81514174>] next_bio+0x24/0x50
> [<ffffffff815145ef>] blkdev_issue_zeroout+0xdf/0x1d0
> [<ffffffff8132ce79>] ext4_issue_zeroout+0x39/0x50
> [<ffffffff81357abf>] ext4_ext_zeroout+0x2f/0x40
> [<ffffffff8135ece0>] ext4_ext_map_blocks+0x1870/0x2190
> [<ffffffff8132cfa1>] ext4_map_blocks+0x111/0x620
> [<ffffffff81330dc8>] ext4_writepages+0x7c8/0x10a0
> [<ffffffff81211851>] do_writepages+0x21/0x30
> [<ffffffff812012ba>] __filemap_fdatawrite_range+0xaa/0xf0
> [<ffffffff812013fd>] filemap_write_and_wait_range+0x2d/0x70
> [<ffffffff81326f6d>] ext4_sync_file+0x18d/0x500
(+Christoph)
With kernel v4.7-rc1 and XFS I see that the blkdev_issue_zeroout() calls
triggered by xfstests make kmemleak complain. I hadn't seen this with
any previous kernel version.
Bart.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: linux-next/linux memleak after IO on dax mountpoint
2016-06-03 16:48 ` Re: linux-next " Bart Van Assche
@ 2016-06-07 10:26 ` Xiong Zhou
0 siblings, 0 replies; 8+ messages in thread
From: Xiong Zhou @ 2016-06-07 10:26 UTC (permalink / raw)
To: linux-block, axboe
Cc: David Drysdale, Xiong Zhou, Stephen Rothwell, axboe, linux-next,
linux-nvdimm, linux-kernel, Christoph Hellwig
adding block-list
On Fri, Jun 03, 2016 at 09:48:27AM -0700, Bart Van Assche wrote:
> On 06/02/2016 08:22 AM, David Drysdale wrote:
> > FWIW, I'm also seeing kmemleak report a leak with v4.7-rc1, in
> > a different scenario (just normal desktop use). Not done much
> > digging so far, but this commit (9082e87bf) looks like it might be
> > relevant -- lots of the following:
> >
> > unreferenced object 0xffff8800c288e900 (size 256):
> > comm "dconf-service", pid 1461, jiffies 4294895636 (age 48.028s)
> > hex dump (first 32 bytes):
> > 00 00 00 00 00 00 00 00 c0 a4 c0 c6 00 88 ff ff ................
> > 02 20 00 20 00 00 00 00 11 00 00 00 00 00 00 00 . . ............
> > backtrace:
> > [<ffffffff81955228>] kmemleak_alloc+0x28/0x50
> > [<ffffffff81268bdc>] kmem_cache_alloc+0xfc/0x360
> > [<ffffffff81203275>] mempool_alloc_slab+0x15/0x20
> > [<ffffffff812030de>] mempool_alloc+0x6e/0x170
> > [<ffffffff815014e8>] bio_alloc_bioset+0xb8/0x230
> > [<ffffffff81514174>] next_bio+0x24/0x50
> > [<ffffffff815145ef>] blkdev_issue_zeroout+0xdf/0x1d0
> > [<ffffffff8132ce79>] ext4_issue_zeroout+0x39/0x50
> > [<ffffffff81357abf>] ext4_ext_zeroout+0x2f/0x40
> > [<ffffffff8135ece0>] ext4_ext_map_blocks+0x1870/0x2190
> > [<ffffffff8132cfa1>] ext4_map_blocks+0x111/0x620
> > [<ffffffff81330dc8>] ext4_writepages+0x7c8/0x10a0
> > [<ffffffff81211851>] do_writepages+0x21/0x30
> > [<ffffffff812012ba>] __filemap_fdatawrite_range+0xaa/0xf0
> > [<ffffffff812013fd>] filemap_write_and_wait_range+0x2d/0x70
> > [<ffffffff81326f6d>] ext4_sync_file+0x18d/0x500
>
> (+Christoph)
>
> With kernel v4.7-rc1 and XFS I see that the blkdev_issue_zeroout() calls
> triggered by xfstests make kmemleak complain. I hadn't seen this with any
> previous kernel version.
>
> Bart.
unreferenced object 0xffff880c4bce2800 (size 2048):
comm "dbench", pid 11389, jiffies 4294884573 (age 193.194s)
hex dump (first 32 bytes):
c0 1d 09 00 00 ea ff ff 00 10 00 00 00 00 00 00 ................
c0 1d 09 00 00 ea ff ff 00 10 00 00 00 00 00 00 ................
backtrace:
[<ffffffff816d178e>] kmemleak_alloc+0x4e/0xb0
[<ffffffff811f6fc3>] kmem_cache_alloc+0xc3/0x200
[<ffffffff813197ec>] bvec_alloc+0x5c/0xf0
[<ffffffff81319a4e>] bio_alloc_bioset+0x1ce/0x2d0
[<ffffffff8132a574>] next_bio+0x24/0x50
[<ffffffff8132aa00>] blkdev_issue_zeroout+0xe0/0x1d0
[<ffffffffa08ebcc8>] ext4_issue_zeroout+0x38/0x50 [ext4]
[<ffffffffa08ec07e>] ext4_map_blocks+0x39e/0x640 [ext4]
[<ffffffffa08ec3b3>] _ext4_get_block+0x93/0xf0 [ext4]
[<ffffffffa08eca6f>] ext4_get_block_trans+0xbf/0x110 [ext4]
[<ffffffffa08eccc5>] ext4_dax_get_block+0x25/0x70 [ext4]
[<ffffffff81270c96>] dax_do_io+0x446/0x610
[<ffffffffa08f1d04>] ext4_direct_IO+0x494/0x750 [ext4]
[<ffffffff81195590>] generic_file_direct_write+0xb0/0x180
[<ffffffff8119571d>] __generic_file_write_iter+0xbd/0x1e0
[<ffffffffa08e5e77>] ext4_file_write_iter+0xf7/0x330 [ext4]
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2016-06-07 10:26 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-05-27 8:46 linux-next memleak after IO on dax mountpoint Xiong Zhou
2016-05-28 4:05 ` Xiong Zhou
2016-06-01 14:37 ` Jeff Moyer
2016-06-02 2:20 ` Xiong Zhou
2016-06-02 15:22 ` David Drysdale
2016-06-03 6:35 ` linux-next/linux " Xiong Zhou
2016-06-03 16:48 ` Re: linux-next " Bart Van Assche
2016-06-07 10:26 ` linux-next/linux " Xiong Zhou
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).