* memory leak bug? when do write with verify
[not found] <CADson1wk9astwdroeAtPXb4A3+yrAcDcCXaEPvEGJPhM4TBfqQ@mail.gmail.com>
@ 2011-09-14 9:53 ` Zhang Taile
2011-09-14 9:56 ` Jiri Horky
2011-09-14 17:39 ` Jens Axboe
0 siblings, 2 replies; 6+ messages in thread
From: Zhang Taile @ 2011-09-14 9:53 UTC (permalink / raw)
To: fio
HI
When I do RandWrite with verify, fio will use a lot of memory. The fio
will increase memory about 10MB in every second. And, the system will
become very slow, after fio start to use swap partition.
For example,� if use fio with following configuration,
[rand-write]
rw=randwrite
numjobs=32
loops=10
thread=1
filename=/dev/sdb
verify=md5
verify_interval=4k
after� minutes, top will Like this :
�PID USER����� PR� NI� VIRT� RES� SHR S %CPU %MEM��� TIME+� COMMAND
�3749 root����� 15�� 0 2672m 1.5g 304m S 169.4 40.4�� 4:10.04 fio
If remove the verify from the configuration, there will be no the
memory problem.
I guess it is a memory leak problem?
Thanks!
--
Taile
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: memory leak bug? when do write with verify
2011-09-14 9:53 ` memory leak bug? when do write with verify Zhang Taile
@ 2011-09-14 9:56 ` Jiri Horky
2011-09-14 13:13 ` Zhang Taile
2011-09-14 17:39 ` Jens Axboe
1 sibling, 1 reply; 6+ messages in thread
From: Jiri Horky @ 2011-09-14 9:56 UTC (permalink / raw)
To: Zhang Taile; +Cc: fio
[-- Attachment #1: Type: text/plain, Size: 1223 bytes --]
Hi,
this sounds like the problem addressed by this thread [1]. Have you
tried to use the latest (trunk) version that includes the mentioned patch?
Jiri Horky
[1] http://www.spinics.net/lists/fio/msg00843.html
On 09/14/2011 11:53 AM, Zhang Taile wrote:
> HI
> When I do RandWrite with verify, fio will use a lot of memory. The fio
> will increase memory about 10MB in every second. And, the system will
> become very slow, after fio start to use swap partition.
>
> For example, if use fio with following configuration,
> [rand-write]
> rw=randwrite
> numjobs=32
> loops=10
> thread=1
> filename=/dev/sdb
> verify=md5
> verify_interval=4k
>
> after minutes, top will Like this :
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 3749 root 15 0 2672m 1.5g 304m S 169.4 40.4 4:10.04 fio
>
>
> If remove the verify from the configuration, there will be no the
> memory problem.
>
> I guess it is a memory leak problem?
>
>
> Thanks!
> --
> Taile
> --
> To unsubscribe from this list: send the line "unsubscribe fio" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 6917 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: memory leak bug? when do write with verify
2011-09-14 9:56 ` Jiri Horky
@ 2011-09-14 13:13 ` Zhang Taile
0 siblings, 0 replies; 6+ messages in thread
From: Zhang Taile @ 2011-09-14 13:13 UTC (permalink / raw)
To: Jiri Horky; +Cc: fio
Hi,
Seems not same bug. I have tested latest (trunk), still has the memory
leak issue.
Taile
On Wed, Sep 14, 2011 at 5:56 PM, Jiri Horky <jiri.horky@cesnet.cz> wrote:
> Hi,
>
> this sounds like the problem addressed by this thread [1]. Have you tried to
> use the latest (trunk) version that includes the mentioned patch?
>
> Jiri Horky
>
> [1] http://www.spinics.net/lists/fio/msg00843.html
>
> On 09/14/2011 11:53 AM, Zhang Taile wrote:
>>
>> HI
>> When I do RandWrite with verify, fio will use a lot of memory. The fio
>> will increase memory about 10MB in every second. And, the system will
>> become very slow, after fio start to use swap partition.
>>
>> For example, if use fio with following configuration,
>> [rand-write]
>> rw=randwrite
>> numjobs=32
>> loops=10
>> thread=1
>> filename=/dev/sdb
>> verify=md5
>> verify_interval=4k
>>
>> after minutes, top will Like this :
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>> 3749 root 15 0 2672m 1.5g 304m S 169.4 40.4 4:10.04 fio
>>
>>
>> If remove the verify from the configuration, there will be no the
>> memory problem.
>>
>> I guess it is a memory leak problem?
>>
>>
>> Thanks!
>> --
>> Taile
>> --
>> To unsubscribe from this list: send the line "unsubscribe fio" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
--
Zhang, Taile 张泰乐
zhangtl@gmail.com
MP:18611745290
msn:zhangtaile@hotmail.com
skype: zhangtaile
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: memory leak bug? when do write with verify
2011-09-14 9:53 ` memory leak bug? when do write with verify Zhang Taile
2011-09-14 9:56 ` Jiri Horky
@ 2011-09-14 17:39 ` Jens Axboe
2011-09-15 2:49 ` Zhang Taile
1 sibling, 1 reply; 6+ messages in thread
From: Jens Axboe @ 2011-09-14 17:39 UTC (permalink / raw)
To: Zhang Taile; +Cc: fio
On 2011-09-14 11:53, Zhang Taile wrote:
> HI
> When I do RandWrite with verify, fio will use a lot of memory. The fio
> will increase memory about 10MB in every second. And, the system will
> become very slow, after fio start to use swap partition.
>
> For example, if use fio with following configuration,
> [rand-write]
> rw=randwrite
> numjobs=32
> loops=10
> thread=1
> filename=/dev/sdb
> verify=md5
> verify_interval=4k
>
> after minutes, top will Like this :
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 3749 root 15 0 2672m 1.5g 304m S 169.4 40.4 4:10.04 fio
>
>
> If remove the verify from the configuration, there will be no the
> memory problem.
>
> I guess it is a memory leak problem?
It's probably not a leak, fio will use a lot of memory for your job
file. Each job will maintain a full backlog of meta data for blocks
written, so it can safely verify it. And you have 32 jobs, so it'll be a
lot of memory.
--
Jens Axboe
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: memory leak bug? when do write with verify
2011-09-14 17:39 ` Jens Axboe
@ 2011-09-15 2:49 ` Zhang Taile
2011-09-15 7:24 ` Jens Axboe
0 siblings, 1 reply; 6+ messages in thread
From: Zhang Taile @ 2011-09-15 2:49 UTC (permalink / raw)
To: Jens Axboe; +Cc: fio
Hi ,
Understand now.
One more question, what backlog for? Why read only with verify do not need it?
Thanks!
On Thu, Sep 15, 2011 at 1:39 AM, Jens Axboe <axboe@kernel.dk> wrote:
> On 2011-09-14 11:53, Zhang Taile wrote:
>> HI
>> When I do RandWrite with verify, fio will use a lot of memory. The fio
>> will increase memory about 10MB in every second. And, the system will
>> become very slow, after fio start to use swap partition.
>>
>> For example, if use fio with following configuration,
>> [rand-write]
>> rw=randwrite
>> numjobs=32
>> loops=10
>> thread=1
>> filename=/dev/sdb
>> verify=md5
>> verify_interval=4k
>>
>> after minutes, top will Like this :
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>> 3749 root 15 0 2672m 1.5g 304m S 169.4 40.4 4:10.04 fio
>>
>>
>> If remove the verify from the configuration, there will be no the
>> memory problem.
>>
>> I guess it is a memory leak problem?
>
> It's probably not a leak, fio will use a lot of memory for your job
> file. Each job will maintain a full backlog of meta data for blocks
> written, so it can safely verify it. And you have 32 jobs, so it'll be a
> lot of memory.
>
> --
> Jens Axboe
>
>
--
Zhang, Taile 张泰乐
zhangtl@gmail.com
MP:18611745290
msn:zhangtaile@hotmail.com
skype: zhangtaile
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: memory leak bug? when do write with verify
2011-09-15 2:49 ` Zhang Taile
@ 2011-09-15 7:24 ` Jens Axboe
0 siblings, 0 replies; 6+ messages in thread
From: Jens Axboe @ 2011-09-15 7:24 UTC (permalink / raw)
To: Zhang Taile; +Cc: fio
On 2011-09-15 04:49, Zhang Taile wrote:
> Hi ,
> Understand now.
> One more question, what backlog for? Why read only with verify do not
> need it?
It's not strictly necessary for most cases. For random writes, we can
verify faster by doing it in disk order instead of in write order. If
you are using trim to discard parts of the ranges, then we need those
appropriately marked in memory.
But we can/could relax it a bit and save the memory. I don't think I've
had this complaint before, I guess people generally have plenty of RAM
these days :-)
--
Jens Axboe
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2011-09-15 7:24 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <CADson1wk9astwdroeAtPXb4A3+yrAcDcCXaEPvEGJPhM4TBfqQ@mail.gmail.com>
2011-09-14 9:53 ` memory leak bug? when do write with verify Zhang Taile
2011-09-14 9:56 ` Jiri Horky
2011-09-14 13:13 ` Zhang Taile
2011-09-14 17:39 ` Jens Axboe
2011-09-15 2:49 ` Zhang Taile
2011-09-15 7:24 ` Jens Axboe
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.