Running fio with offset Increment option

Running fio with offset Increment option

[Suresh Dhanarajan]

Hi,

I wanted to use the "offset increment option" and  "numjobs" so that i
can split the work load between ten jobs and reduce the run time.
But what i am seeing is that the offset counter is not getting reset
when the first job in the jobfile is completed.(is it that the offset
is global for the job file and not for the Jobs inside the jobfile?)
So  my read starts from the last offset set by the writes.

here is my job file,

[global]
bs=64k
direct=1
numjobs=10
size=1m
group_reporting

[write-phase]
offset_increment=1m
filename=/dev/sdb
rw=write
write_iolog=verfywrite2

[read-phase]
stonewall
offset_increment=1m
filename=/dev/sdb
rw=read
write_iolog=verfyread2

I tried using the offset=0 option in the read phase job but now every
time the read happens from zeroth offsite.
Now the offset increment option is not getting honored.

[write-phase]
offset_increment=1m
filename=/dev/sdb
rw=write
write_iolog=verfywrite2

[read-phase]
stonewall
offset=0
offset_increment=1m
filename=/dev/sdb
rw=read
write_iolog=verfyread2

I tried the same case with verify option.
the behavior is same.

Is there any way that i can reset the offset counters once the job is completed?

thanks,

Re: Running fio with offset Increment option

[Jens Axboe]

On 2012-04-14 01:48, Suresh Dhanarajan wrote:
> Hi,
> 
> I wanted to use the "offset increment option" and  "numjobs" so that i
> can split the work load between ten jobs and reduce the run time.
> But what i am seeing is that the offset counter is not getting reset
> when the first job in the jobfile is completed.(is it that the offset
> is global for the job file and not for the Jobs inside the jobfile?)
> So  my read starts from the last offset set by the writes.
> 
> here is my job file,
> 
> [global]
> bs=64k
> direct=1
> numjobs=10
> size=1m
> group_reporting
> 
> [write-phase]
> offset_increment=1m
> filename=/dev/sdb
> rw=write
> write_iolog=verfywrite2
> 
> [read-phase]
> stonewall
> offset_increment=1m
> filename=/dev/sdb
> rw=read
> write_iolog=verfyread2
> 
> I tried using the offset=0 option in the read phase job but now every
> time the read happens from zeroth offsite.
> Now the offset increment option is not getting honored.
> 
> [write-phase]
> offset_increment=1m
> filename=/dev/sdb
> rw=write
> write_iolog=verfywrite2
> 
> [read-phase]
> stonewall
> offset=0
> offset_increment=1m
> filename=/dev/sdb
> rw=read
> write_iolog=verfyread2
> 
> I tried the same case with verify option.
> the behavior is same.
> 
> Is there any way that i can reset the offset counters once the job is completed?

Right now there isn't, but it does make sense to reset the counter
across stonewalls. At the moment, it'll just increment for each job.
Internally, fio sets up all the jobs, even across stonewalls, when it
starts up. It just starts some of them later on, depending on those
parameters. It does seem most useful to have it reset across a hard
barrier though, I can definitely change it to do that.

Let me brew up a patch later today that you can test.

-- 
Jens Axboe

Re: Running fio with offset Increment option

[Jens Axboe]

On 2012-04-14 17:23, Jens Axboe wrote:
> On 2012-04-14 01:48, Suresh Dhanarajan wrote:
>> Hi,
>>
>> I wanted to use the "offset increment option" and  "numjobs" so that i
>> can split the work load between ten jobs and reduce the run time.
>> But what i am seeing is that the offset counter is not getting reset
>> when the first job in the jobfile is completed.(is it that the offset
>> is global for the job file and not for the Jobs inside the jobfile?)
>> So  my read starts from the last offset set by the writes.
>>
>> here is my job file,
>>
>> [global]
>> bs=64k
>> direct=1
>> numjobs=10
>> size=1m
>> group_reporting
>>
>> [write-phase]
>> offset_increment=1m
>> filename=/dev/sdb
>> rw=write
>> write_iolog=verfywrite2
>>
>> [read-phase]
>> stonewall
>> offset_increment=1m
>> filename=/dev/sdb
>> rw=read
>> write_iolog=verfyread2
>>
>> I tried using the offset=0 option in the read phase job but now every
>> time the read happens from zeroth offsite.
>> Now the offset increment option is not getting honored.
>>
>> [write-phase]
>> offset_increment=1m
>> filename=/dev/sdb
>> rw=write
>> write_iolog=verfywrite2
>>
>> [read-phase]
>> stonewall
>> offset=0
>> offset_increment=1m
>> filename=/dev/sdb
>> rw=read
>> write_iolog=verfyread2
>>
>> I tried the same case with verify option.
>> the behavior is same.
>>
>> Is there any way that i can reset the offset counters once the job is completed?
> 
> Right now there isn't, but it does make sense to reset the counter
> across stonewalls. At the moment, it'll just increment for each job.
> Internally, fio sets up all the jobs, even across stonewalls, when it
> starts up. It just starts some of them later on, depending on those
> parameters. It does seem most useful to have it reset across a hard
> barrier though, I can definitely change it to do that.
> 
> Let me brew up a patch later today that you can test.

Had a few minutes now, didn't expect that. Can you try this? Do a make
clean first.

diff --git a/filesetup.c b/filesetup.c
index 166ace8..bad5bed 100644
--- a/filesetup.c
+++ b/filesetup.c
@@ -714,7 +714,7 @@ int setup_files(struct thread_data *td)
 	need_extend = 0;
 	for_each_file(td, f, i) {
 		f->file_offset = td->o.start_offset +
-			(td->thread_number - 1) * td->o.offset_increment;
+			(td->group_thread_number - 1) * td->o.offset_increment;
 
 		if (!td->o.file_size_low) {
 			/*
diff --git a/fio.h b/fio.h
index 95d9d77..05a5572 100644
--- a/fio.h
+++ b/fio.h
@@ -67,6 +67,7 @@ struct thread_data {
 	char verror[FIO_VERROR_SIZE];
 	pthread_t thread;
 	unsigned int thread_number;
+	unsigned int group_thread_number;
 	unsigned int groupid;
 	struct thread_stat ts;
 
diff --git a/init.c b/init.c
index 7422628..46571ee 100644
--- a/init.c
+++ b/init.c
@@ -35,6 +35,7 @@ static int dump_cmdline;
 static int def_timeout;
 
 static struct thread_data def_thread;
+static int group_thread_number;
 struct thread_data *threads = NULL;
 
 int exitall_on_terminate = 0;
@@ -829,8 +830,10 @@ static int add_job(struct thread_data *td, const char *jobname, int job_add_num,
 	if ((td->o.stonewall || td->o.new_group) && prev_group_jobs) {
 		prev_group_jobs = 0;
 		groupid++;
+		group_thread_number = 0;
 	}
 
+	td->group_thread_number = ++group_thread_number;
 	td->groupid = groupid;
 	prev_group_jobs++;
 

-- 
Jens Axboe

Re: Running fio with offset Increment option

[Suresh Dhanarajan]

Hi Jens,

I am facing issues while applying the patch,
Please find the details,
===========
The patch for the file is ,

--- a/fio.h
+++ b/fio.h
@@ -67,6 +67,7 @@ struct thread_data {
       char verror[FIO_VERROR_SIZE];
       pthread_t thread;
       unsigned int thread_number;
+       unsigned int group_thread_number;
       unsigned int groupid;
       struct thread_stat ts;

There is no such line in that file.

$>grep 'unsigned int thread_number' fio.h
extern unsigned int thread_number;

The line 67 does not have any related lines,

65 enum {
 66     RW_SEQ_SEQ  = 0,
 67     RW_SEQ_IDENT,
 68 };

Let me know how to proceed further.

Thanks,
Suresh.D

On Sat, Apr 14, 2012 at 8:29 AM, Jens Axboe <axboe@kernel.dk> wrote:
> On 2012-04-14 17:23, Jens Axboe wrote:
>> On 2012-04-14 01:48, Suresh Dhanarajan wrote:
>>> Hi,
>>>
>>> I wanted to use the "offset increment option" and  "numjobs" so that i
>>> can split the work load between ten jobs and reduce the run time.
>>> But what i am seeing is that the offset counter is not getting reset
>>> when the first job in the jobfile is completed.(is it that the offset
>>> is global for the job file and not for the Jobs inside the jobfile?)
>>> So  my read starts from the last offset set by the writes.
>>>
>>> here is my job file,
>>>
>>> [global]
>>> bs=64k
>>> direct=1
>>> numjobs=10
>>> size=1m
>>> group_reporting
>>>
>>> [write-phase]
>>> offset_increment=1m
>>> filename=/dev/sdb
>>> rw=write
>>> write_iolog=verfywrite2
>>>
>>> [read-phase]
>>> stonewall
>>> offset_increment=1m
>>> filename=/dev/sdb
>>> rw=read
>>> write_iolog=verfyread2
>>>
>>> I tried using the offset=0 option in the read phase job but now every
>>> time the read happens from zeroth offsite.
>>> Now the offset increment option is not getting honored.
>>>
>>> [write-phase]
>>> offset_increment=1m
>>> filename=/dev/sdb
>>> rw=write
>>> write_iolog=verfywrite2
>>>
>>> [read-phase]
>>> stonewall
>>> offset=0
>>> offset_increment=1m
>>> filename=/dev/sdb
>>> rw=read
>>> write_iolog=verfyread2
>>>
>>> I tried the same case with verify option.
>>> the behavior is same.
>>>
>>> Is there any way that i can reset the offset counters once the job is completed?
>>
>> Right now there isn't, but it does make sense to reset the counter
>> across stonewalls. At the moment, it'll just increment for each job.
>> Internally, fio sets up all the jobs, even across stonewalls, when it
>> starts up. It just starts some of them later on, depending on those
>> parameters. It does seem most useful to have it reset across a hard
>> barrier though, I can definitely change it to do that.
>>
>> Let me brew up a patch later today that you can test.
>
> Had a few minutes now, didn't expect that. Can you try this? Do a make
> clean first.
>
> diff --git a/filesetup.c b/filesetup.c
> index 166ace8..bad5bed 100644
> --- a/filesetup.c
> +++ b/filesetup.c
> @@ -714,7 +714,7 @@ int setup_files(struct thread_data *td)
>        need_extend = 0;
>        for_each_file(td, f, i) {
>                f->file_offset = td->o.start_offset +
> -                       (td->thread_number - 1) * td->o.offset_increment;
> +                       (td->group_thread_number - 1) * td->o.offset_increment;
>
>                if (!td->o.file_size_low) {
>                        /*
> diff --git a/fio.h b/fio.h
> index 95d9d77..05a5572 100644
> --- a/fio.h
> +++ b/fio.h
> @@ -67,6 +67,7 @@ struct thread_data {
>        char verror[FIO_VERROR_SIZE];
>        pthread_t thread;
>        unsigned int thread_number;
> +       unsigned int group_thread_number;
>        unsigned int groupid;
>        struct thread_stat ts;
>
> diff --git a/init.c b/init.c
> index 7422628..46571ee 100644
> --- a/init.c
> +++ b/init.c
> @@ -35,6 +35,7 @@ static int dump_cmdline;
>  static int def_timeout;
>
>  static struct thread_data def_thread;
> +static int group_thread_number;
>  struct thread_data *threads = NULL;
>
>  int exitall_on_terminate = 0;
> @@ -829,8 +830,10 @@ static int add_job(struct thread_data *td, const char *jobname, int job_add_num,
>        if ((td->o.stonewall || td->o.new_group) && prev_group_jobs) {
>                prev_group_jobs = 0;
>                groupid++;
> +               group_thread_number = 0;
>        }
>
> +       td->group_thread_number = ++group_thread_number;
>        td->groupid = groupid;
>        prev_group_jobs++;
>
>
> --
> Jens Axboe
>

Re: Running fio with offset Increment option

[Jens Axboe]

On 04/17/2012 01:15 AM, Suresh Dhanarajan wrote:
> Hi Jens,
> 
> I am facing issues while applying the patch,
> Please find the details,
> ===========
> The patch for the file is ,
> 
> --- a/fio.h
> +++ b/fio.h
> @@ -67,6 +67,7 @@ struct thread_data {
>        char verror[FIO_VERROR_SIZE];
>        pthread_t thread;
>        unsigned int thread_number;
> +       unsigned int group_thread_number;
>        unsigned int groupid;
>        struct thread_stat ts;
> 
> There is no such line in that file.
> 
> $>grep 'unsigned int thread_number' fio.h
> extern unsigned int thread_number;
> 
> The line 67 does not have any related lines,
> 
> 65 enum {
>  66     RW_SEQ_SEQ  = 0,
>  67     RW_SEQ_IDENT,
>  68 };
> 
> Let me know how to proceed further.

It was against the gfio branch. The line is there in fio.h, it's just
further down and thread_number isn't unsigned :-)

Here's a patch against the master branch, should apply against any
recent version.

diff --git a/filesetup.c b/filesetup.c
index f3d3829..0fc0b41 100644
--- a/filesetup.c
+++ b/filesetup.c
@@ -714,7 +714,7 @@ int setup_files(struct thread_data *td)
 	need_extend = 0;
 	for_each_file(td, f, i) {
 		f->file_offset = td->o.start_offset +
-			(td->thread_number - 1) * td->o.offset_increment;
+			(td->group_thread_number - 1) * td->o.offset_increment;
 
 		if (!td->o.file_size_low) {
 			/*
diff --git a/fio.h b/fio.h
index 6da22f0..d4fb907 100644
--- a/fio.h
+++ b/fio.h
@@ -280,6 +280,7 @@ struct thread_data {
 	char verror[FIO_VERROR_SIZE];
 	pthread_t thread;
 	int thread_number;
+	unsigned int group_thread_number;
 	int groupid;
 	struct thread_stat ts;
 
diff --git a/init.c b/init.c
index 3865d09..465ffa9 100644
--- a/init.c
+++ b/init.c
@@ -33,6 +33,7 @@ static int dump_cmdline;
 static int def_timeout;
 
 static struct thread_data def_thread;
+static int group_thread_number;
 struct thread_data *threads = NULL;
 
 int exitall_on_terminate = 0;
@@ -824,8 +825,10 @@ static int add_job(struct thread_data *td, const char *jobname, int job_add_num)
 	if ((td->o.stonewall || td->o.new_group) && prev_group_jobs) {
 		prev_group_jobs = 0;
 		groupid++;
+		group_thread_number = 0;
 	}
 
+	td->group_thread_number = ++group_thread_number;
 	td->groupid = groupid;
 	prev_group_jobs++;
 

-- 
Jens Axboe

Re: Running fio with offset Increment option

[Suresh Dhanarajan]

Hi Jens,

Thanks for the patch.
I checked with the patch given you and found that the offset is
getting set to zero for every new job.

Suresh


On Mon, Apr 16, 2012 at 11:30 PM, Jens Axboe <axboe@kernel.dk> wrote:
> On 04/17/2012 01:15 AM, Suresh Dhanarajan wrote:
>> Hi Jens,
>>
>> I am facing issues while applying the patch,
>> Please find the details,
>> ===========
>> The patch for the file is ,
>>
>> --- a/fio.h
>> +++ b/fio.h
>> @@ -67,6 +67,7 @@ struct thread_data {
>>        char verror[FIO_VERROR_SIZE];
>>        pthread_t thread;
>>        unsigned int thread_number;
>> +       unsigned int group_thread_number;
>>        unsigned int groupid;
>>        struct thread_stat ts;
>>
>> There is no such line in that file.
>>
>> $>grep 'unsigned int thread_number' fio.h
>> extern unsigned int thread_number;
>>
>> The line 67 does not have any related lines,
>>
>> 65 enum {
>>  66     RW_SEQ_SEQ  = 0,
>>  67     RW_SEQ_IDENT,
>>  68 };
>>
>> Let me know how to proceed further.
>
> It was against the gfio branch. The line is there in fio.h, it's just
> further down and thread_number isn't unsigned :-)
>
> Here's a patch against the master branch, should apply against any
> recent version.
>
> diff --git a/filesetup.c b/filesetup.c
> index f3d3829..0fc0b41 100644
> --- a/filesetup.c
> +++ b/filesetup.c
> @@ -714,7 +714,7 @@ int setup_files(struct thread_data *td)
>        need_extend = 0;
>        for_each_file(td, f, i) {
>                f->file_offset = td->o.start_offset +
> -                       (td->thread_number - 1) * td->o.offset_increment;
> +                       (td->group_thread_number - 1) * td->o.offset_increment;
>
>                if (!td->o.file_size_low) {
>                        /*
> diff --git a/fio.h b/fio.h
> index 6da22f0..d4fb907 100644
> --- a/fio.h
> +++ b/fio.h
> @@ -280,6 +280,7 @@ struct thread_data {
>        char verror[FIO_VERROR_SIZE];
>        pthread_t thread;
>        int thread_number;
> +       unsigned int group_thread_number;
>        int groupid;
>        struct thread_stat ts;
>
> diff --git a/init.c b/init.c
> index 3865d09..465ffa9 100644
> --- a/init.c
> +++ b/init.c
> @@ -33,6 +33,7 @@ static int dump_cmdline;
>  static int def_timeout;
>
>  static struct thread_data def_thread;
> +static int group_thread_number;
>  struct thread_data *threads = NULL;
>
>  int exitall_on_terminate = 0;
> @@ -824,8 +825,10 @@ static int add_job(struct thread_data *td, const char *jobname, int job_add_num)
>        if ((td->o.stonewall || td->o.new_group) && prev_group_jobs) {
>                prev_group_jobs = 0;
>                groupid++;
> +               group_thread_number = 0;
>        }
>
> +       td->group_thread_number = ++group_thread_number;
>        td->groupid = groupid;
>        prev_group_jobs++;
>
>
> --
> Jens Axboe
>

Re: Running fio with offset Increment option

[Jens Axboe]

On 04/18/2012 11:37 PM, Suresh Dhanarajan wrote:
> Hi Jens,
> 
> Thanks for the patch.
> I checked with the patch given you and found that the offset is
> getting set to zero for every new job.

Are you sure? I hadn't tested the patch myself, until now. It seems to
work just fine here, group_thread_id is contigious until a stonewall is
seen. Works for new job entries and numjobs.

-- 
Jens Axboe

Re: Running fio with offset Increment option

[Suresh Dhanarajan]

Hi Jens,

I tested with and without stonewall.
with stonewall the offset is reset to zero for a newjob.
But without stonewall the offset is not reseting zero.

thanks,
Suresh.D

On Wed, Apr 18, 2012 at 10:47 PM, Jens Axboe <axboe@kernel.dk> wrote:
> On 04/18/2012 11:37 PM, Suresh Dhanarajan wrote:
>> Hi Jens,
>>
>> Thanks for the patch.
>> I checked with the patch given you and found that the offset is
>> getting set to zero for every new job.
>
> Are you sure? I hadn't tested the patch myself, until now. It seems to
> work just fine here, group_thread_id is contiguous until a stonewall is
> seen. Works for new job entries and numjobs.
>
> --
> Jens Axboe
>

Re: Running fio with offset Increment option

[Jens Axboe]

On 04/19/2012 09:32 PM, Suresh Dhanarajan wrote:
> Hi Jens,
> 
> I tested with and without stonewall.
> with stonewall the offset is reset to zero for a newjob.
> But without stonewall the offset is not reseting zero.

How are you verifying that this is the case? I added a debug printf
here, and ran the below job:

---

[global]
numjobs=4
offset_increment=100m
filename=foo
ioengine=null
size=100g
rw=randread
rate=500m

[null]

[null-a]

[null-b]
stonewall

[null-c]

[null-d]
stonewall

[null-e]

---

Which should give us 4 jobs, then 4 more jobs at continuing offsets,
then null-b at 0 again, null-c at 4*100MB and up, null-d at zero and
null-e identical to null-c.

job1: file foo offset 0                 <- null
job2: file foo offset 104857600
job3: file foo offset 209715200
job4: file foo offset 314572800
job5: file foo offset 419430400         <- null-a
job6: file foo offset 524288000
job7: file foo offset 629145600
job8: file foo offset 734003200
job9: file foo offset 0                 <- null-b
job10: file foo offset 104857600
job11: file foo offset 209715200
job12: file foo offset 314572800
job13: file foo offset 419430400        <- null-c
job14: file foo offset 524288000
job15: file foo offset 629145600
job16: file foo offset 734003200
job17: file foo offset 0                <- null-d
job18: file foo offset 104857600
job19: file foo offset 209715200
job20: file foo offset 314572800
job21: file foo offset 419430400        <- null-e
job22: file foo offset 524288000
job23: file foo offset 629145600
job24: file foo offset 734003200

So it looks like it's working here, at least offset is being calculated
and set correctly. Hence my question on why it isn't working for you. If
you are looking at the actual IO offsets generated, then it's possible
there is a bug somewhere. That bug should, however, also affect the
original patch since this is just changing how we generate them upfront.

-- 
Jens Axboe

Re: Running fio with offset Increment option

[Suresh Dhanarajan]

Hi Jens,

I verified the offsets by using the write_io_log switch.
when i use offset increment without patch the offset are incremented
and not reset to zero after stone wall.
when i use with patch the offset resets to zero after stone wall.
I hope you are testing with the patch.
If it is so this what the behavior iam  also seeing.

Thanks,
Suresh


On Thu, Apr 19, 2012 at 11:27 PM, Jens Axboe <axboe@kernel.dk> wrote:
> On 04/19/2012 09:32 PM, Suresh Dhanarajan wrote:
>> Hi Jens,
>>
>> I tested with and without stonewall.
>> with stonewall the offset is reset to zero for a newjob.
>> But without stonewall the offset is not reseting zero.
>
> How are you verifying that this is the case? I added a debug printf
> here, and ran the below job:
>
> ---
>
> [global]
> numjobs=4
> offset_increment=100m
> filename=foo
> ioengine=null
> size=100g
> rw=randread
> rate=500m
>
> [null]
>
> [null-a]
>
> [null-b]
> stonewall
>
> [null-c]
>
> [null-d]
> stonewall
>
> [null-e]
>
> ---
>
> Which should give us 4 jobs, then 4 more jobs at continuing offsets,
> then null-b at 0 again, null-c at 4*100MB and up, null-d at zero and
> null-e identical to null-c.
>
> job1: file foo offset 0                 <- null
> job2: file foo offset 104857600
> job3: file foo offset 209715200
> job4: file foo offset 314572800
> job5: file foo offset 419430400         <- null-a
> job6: file foo offset 524288000
> job7: file foo offset 629145600
> job8: file foo offset 734003200
> job9: file foo offset 0                 <- null-b
> job10: file foo offset 104857600
> job11: file foo offset 209715200
> job12: file foo offset 314572800
> job13: file foo offset 419430400        <- null-c
> job14: file foo offset 524288000
> job15: file foo offset 629145600
> job16: file foo offset 734003200
> job17: file foo offset 0                <- null-d
> job18: file foo offset 104857600
> job19: file foo offset 209715200
> job20: file foo offset 314572800
> job21: file foo offset 419430400        <- null-e
> job22: file foo offset 524288000
> job23: file foo offset 629145600
> job24: file foo offset 734003200
>
> So it looks like it's working here, at least offset is being calculated
> and set correctly. Hence my question on why it isn't working for you. If
> you are looking at the actual IO offsets generated, then it's possible
> there is a bug somewhere. That bug should, however, also affect the
> original patch since this is just changing how we generate them upfront.
>
> --
> Jens Axboe
>

Re: Running fio with offset Increment option

[Jens Axboe]

On 04/28/2012 09:09 PM, Suresh Dhanarajan wrote:
> Hi Jens,
> 
> I verified the offsets by using the write_io_log switch.
> when i use offset increment without patch the offset are incremented
> and not reset to zero after stone wall.
> when i use with patch the offset resets to zero after stone wall.
> I hope you are testing with the patch.
> If it is so this what the behavior iam  also seeing.

I'm puzzled. Did a full test here now, with the patch. Simple job:

[global]
ioengine=null
bs=1k
size=10m
offset_increment=1m
numjobs=2
write_iolog=log

[job1]
rw=read

[job2]
stonewall
rw=read

using both the write_iolog and --debug=io, I verified that job1 has two
files, one starts at 0M the next at 1M. job2 has two files, one starts
at 0M the next at 1M. If I remove the stonewall, job1 has 0M/1M and job
has 2M/3M.

That certainly looks like it's working to me. I think you need to check
and double check that you are indeed using a patched version. And note
that write_iolog will _append_ to an existing log file. Perhaps you are
looking at an earlier entry set?

-- 
Jens Axboe