Re: O_DIRECT

Re: O_DIRECT

[Stephen C. Tweedie]

Hi Alan,

On Fri, 2003-06-27 at 10:40, Stephen C. Tweedie wrote:

> On Thu, 2003-06-26 at 21:21, Alan Cox wrote:
> > So its now confirmed with 3 distros, two file systems and several 
> > compilers. It certainly seems to be the O_DIRECT patches but I'll pull
> > the back out for the next -ac and check I guess

Ouch ouch ouch, there's nasty merge conflict between the O_DIRECT patch
and an existing 64-bit rlimit chunk in -ac3.  You really, really want
the change below. :-)  Marcelo's tree appears OK, and this is a common
code path for all filesystems in -ac, so it matches the failure patterns
that far.

Cheers,
 Stephen

--- mm/filemap.c.~1~	2003-06-27 09:58:08.000000000 +0100
+++ mm/filemap.c	2003-06-27 11:13:07.000000000 +0100
@@ -2995,8 +2995,8 @@
 		}
 		/* Fix this up when we got to rlimit64 */
 		if (pos > 0xFFFFFFFFULL)
-			count = 0;
-		else if(count > limit - (u32)pos) {
+			*count = 0;
+		else if(*count > limit - (u32)pos) {
 			/* send_sig(SIGXFSZ, current, 0); */
 			*count = limit - (u32)pos;
 		}

Re: O_DIRECT

[Alan Cox]

> Ouch ouch ouch, there's nasty merge conflict between the O_DIRECT patch
> and an existing 64-bit rlimit chunk in -ac3.  You really, really want
> the change below. :-)  Marcelo's tree appears OK, and this is a common
> code path for all filesystems in -ac, so it matches the failure patterns
> that far.

Ouch indeed - ok thats good, that means its not the O_DIRECT stuff. Thanks
for figuring it out

Re: O_DIRECT

[Andrew Morton]

Rogier Wolff <R.E.Wolff@BitWizard.nl> wrote:
>
> The code
>  we use is: 
> 
>    p = malloc (blocksize + PAGE_SIZE);
>    buf = (void *) ((  ((long)p) + PAGE_SIZE -1 ) & ~(PAGE_SIZE-1));

memalign() and posix_memalign() will do this for you.

Re: O_DIRECT

[Rogier Wolff]

On Tue, Jul 20, 2004 at 08:48:24PM +0200, bert hubert wrote:
> On Tue, Jul 20, 2004 at 10:27:57AM -0700, Shesha Sreenivasamurthy wrote:
> > I am having trouble with O_DIRECT. Trying to read or write from a block 
> > device partition.
> > 
> > 1. Can O_DIRECT be used on a plain block device partition say 
> > "/dev/sda11" without having a filesystem on it.
> 
> As fas as I know, yes, but be aware that O_DIRECT requires page aligned
> addresses! (an integral of 4096 on most systems).

Agreed, and I can tell you it works with raw disk devices. The code
we use is: 

  p = malloc (blocksize + PAGE_SIZE);
  buf = (void *) ((  ((long)p) + PAGE_SIZE -1 ) & ~(PAGE_SIZE-1));

		Roger. 

-- 
** R.E.Wolff@BitWizard.nl ** http://www.BitWizard.nl/ ** +31-15-2600998 **
*-- BitWizard writes Linux device drivers for any device you may have! --*
**** "Linux is like a wigwam -  no windows, no gates, apache inside!" ****

Re: O_DIRECT

[Devel]

Thanks a lot for your answare.
So if i would write information on a partition without a fs i have to write additional information to "retrive" the first one.
Do you know some readable material about o project about that?


Il Wed, 21 Jul 2004 09:52:15 -0700
Shesha Sreenivasamurthy <shesha@inostor.com> scrisse:

> Usually what you do is, partition the disk, create filesystem and read 
> or write files.
>   Reading and writing is without a filesystem.
>    Assume you have maintained some superblock information of your own 
> driver say last 512 bytes you have designated as superblock. You can 
> seek to that position and read the last 512 bytes which is meaningful 
> only to you. Note that I am read/writing raw bytes not any file. 
> Therefore you can read/write without a filesystem and even without any 
> partition,
> 
> -Shesha
> 
> 
> Devel wrote:
> 
> >Hi all,
> >how i can write data on a partition without a filesystem??
> >
> >Saluti Carlo!
> >
> >Il Tue, 20 Jul 2004 20:48:24 +0200
> >bert hubert <ahu@ds9a.nl> scrisse:
> >
> >  
> >
> >>On Tue, Jul 20, 2004 at 10:27:57AM -0700, Shesha Sreenivasamurthy wrote:
> >>    
> >>
> >>>I am having trouble with O_DIRECT. Trying to read or write from a block 
> >>>device partition.
> >>>
> >>>1. Can O_DIRECT be used on a plain block device partition say 
> >>>"/dev/sda11" without having a filesystem on it.
> >>>      
> >>>
> >>As fas as I know, yes, but be aware that O_DIRECT requires page aligned
> >>addresses! (an integral of 4096 on most systems).
> >>
> >>    
> >>
> >>>2. If no file system is created then what should be the softblock size. 
> >>>I am using the IOCTL "BLKBSZGET". Is this correct?
> >>>      
> >>>
> >>No idea what you mean - but see above about the aligned addresses.
> >>
> >>    
> >>
> >>>3. Can we use SEEK_END with O_DIRECT on a partition without filesystem.
> >>>      
> >>>
> >>I see no reason why not. Good luck!
> >>
> >>-- 
> >>http://www.PowerDNS.com      Open source, database driven DNS Software 
> >>http://lartc.org           Linux Advanced Routing & Traffic Control HOWTO
> >>-
> >>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> >>the body of a message to majordomo@vger.kernel.org
> >>More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>Please read the FAQ at  http://www.tux.org/lkml/
> >>
> >>    
> >>
> >
> >--
> >Kernelnewbies: Help each other learn about the Linux kernel.
> >Archive:       http://mail.nl.linux.org/kernelnewbies/
> >FAQ:           http://kernelnewbies.org/faq/
> >
> >
> >.
> >
> >  
> >
> 

Re: O_DIRECT

[Shesha Sreenivasamurthy]

Ok, I got to know what exactly is the problem ,,

Basically on my partition total sectors is not a muliple of 8 (Each 
sector = 512, 4096/512=8). That is the end of the disk is not alligned 
with 4096, What I am doing is, reading/writing from/to the last 8192 
bytes by seeking (Negetive  8192) from the end. So this may not be 4096 
byte alligned. Now the start of my read/write point  WILL always be 1024 
byte alligned in case of odd and even # of sectors becasue, Even number 
of sectors is trivial/obvious, but in case of odd # of sectors, when we 
do a SEEK_END it will always seek to the last EVEN sector of the disk. 
Therefore I am always successful with 1024 byte alligned. Actually the 
last sector in my case is 490223474 :)

Thank you all, special thanks to Rob van Nieuwkerk ..
Shesha

Rob van Nieuwkerk wrote:

>On Wed, 21 Jul 2004 17:00:07 -0700
>Shesha Sreenivasamurthy <shesha@inostor.com> wrote:
>
>Hi Shesha,
>
>I've always used the default softblock sizes (eg. did not alter it)
>on the devices I've dealt with (which was 1024).
>
>Did you check what this default softblock size is for your device
>(with BLKBSZGET) ?
>
>Reading my old code I noticed that I use 3 (instead of the 2 I already
>mentioned) alignment requirements:
>
>	1.  read size 1024 byte aligned          (= softblock size)
>	2.  read buffer page aligned (4096)
>	3.  read/write offset 1024 byte aligned  (= softblock size)
>
>Rule 3. is important too for O_DIRECT !  It may be that rule 2. can be
>softblock aligned too, but page alignment certainly won't harm
>
>	greetings,
>	Rob van Nieuwkerk
>
>  
>
>>You are right I am not using any file system. Just did that experiment 
>>by creating FS too. That also did not help. I shared that information. 
>>You are absolutely right I am dealing with straight partitions.
>>
>>(1) With staraight partition even if I explicitly set the block size to 
>>4096 and adher to the rules of O_DIRECT i cannot read/write
>>
>>(2) With staraight partition if I set the block size to 1024 and adher 
>>to the rules of O_DIRECT i can read/write
>>
>>Ruls include : Alligning the buffer & read/write multiple of block size.
>>
>>-Shesha
>>
>>Rob van Nieuwkerk wrote:
>>
>>    
>>
>>>On Wed, 21 Jul 2004 16:18:37 -0700
>>>Shesha Sreenivasamurthy <shesha@inostor.com> wrote:
>>>
>>>Hi Shesha,
>>>
>>>Your original question (and my answer) was about a different situation:
>>>
>>>       - partition with no filesystem
>>>       - access on this raw partition
>>>
>>>Now you start talking about various file-systems and I get the impression
>>>that you access the device through the file-system (although that is
>>>not very clear in your text) on a mounted fs.
>>>
>>>What I told you applied to straight partitions (without mounted a mounted fs).
>>>
>>>I don't know the exact rules for the various file-systems.
>>>They might be different (and some might not support O_DIRECT at all).
>>>I expect the same "soft block size rules" to apply for the filesystems
>>>that *do* support O_DIRECT.  Note that mounting a partition might
>>>change the softblock size of that partition used by the kernel from thereon.
>>>
>>>
>>>	greetings,
>>>	Rob van Nieuwkerk
>>>
>>>PS1: what kernel are you using ?
>>>
>>>PS2: what are you trying to do ?
>>>
>>>PS3: you mention "our driver ..".  What is this driver ?
>>>
>>>
>>>
>>> 
>>>
>>>      
>>>
>>>>I could never get it working with 4096 block size.
>>>>
>>>>I created an EXT3 with blockize=4096 (mkfs.ext3 -b 4096 /dev/sda11). 
>>>>Alligned memory to 4096 boundary. Read 8192 Bytes. -> UNSUCCESSFUL :(
>>>>I created an XFS with blockize=4096 (mkfs.xfs -f  -b size=4096 
>>>>/dev/sda11). Alligned memory to 4096 boundary. Read 8192 Bytes. -> 
>>>>UNSUCCESSFUL :(
>>>>
>>>>SUCCESS HAPPEND ONLY WHEN ....
>>>>
>>>>1. I created an XFS with blockize=4096 (mkfs.xfs -f  -b size=4096 
>>>>/dev/sda11).
>>>>                                                      OR
>>>>   I created an EXT3 with blockize=4096 (mkfs.ext3 -b 4096 /dev/sda11).
>>>>
>>>>2. Changed blocksizeto 1024
>>>>3. Alligned memory to 1024 boundary.
>>>>4  Read 8192 Bytes. -> SUCCESSFUL :)
>>>>
>>>>-Shesha
>>>>
>>>>
>>>>Rob van Nieuwkerk wrote:
>>>>
>>>>   
>>>>
>>>>        
>>>>
>>>>>On Wed, 21 Jul 2004 11:01:20 -0700
>>>>>Shesha Sreenivasamurthy <shesha@inostor.com> wrote:
>>>>>
>>>>>Hi Shesha,
>>>>>
>>>>>
>>>>>
>>>>>     
>>>>>
>>>>>          
>>>>>
>>>>>>ohhh OK, if the block size is 4096, then the read/write size must be 
>>>>>>integer multiple of 4096 ??? is it ???
>>>>>>In general should the read/write length be a multiple of block size?
>>>>>>  
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>Yes, see my previous emails.
>>>>>
>>>>>	greetings,
>>>>>	Rob van Nieuwkerk
>>>>>
>>>>>
>>>>>
>>>>>     
>>>>>
>>>>>          
>>>>>
>>>>>>Rob van Nieuwkerk wrote:
>>>>>>
>>>>>>  
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>On Wed, 21 Jul 2004 10:10:26 -0700
>>>>>>>Shesha Sreenivasamurthy <shesha@inostor.com> wrote:
>>>>>>>
>>>>>>>Hi Shesha,
>>>>>>>
>>>>>>>You don't mention what the *size* of your read()/write() is.
>>>>>>>Besides a requirement on the alignment of the read/write buffer
>>>>>>>the size of the read()/write() must also be OK.
>>>>>>>
>>>>>>>	greetings,
>>>>>>>	Rob van Nieuwkerk
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>    
>>>>>>>
>>>>>>>         
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>This is what I found ....
>>>>>>>>
>>>>>>>>Our driver sets the block size to be 4096. so BLKBSZGET will return 
>>>>>>>>4096. So if I allin the memory at 4096 boundary, I cannot read using 
>>>>>>>>O_DIRECT. But, if I set the block size to 512.  I can read/write 
>>>>>>>>successfully. It also works with 1024, but no with 4096
>>>>>>>>
>>>>>>>>So the recepie what I am following is ...
>>>>>>>>
>>>>>>>>BLKBSZGET -> Get original block size
>>>>>>>>BLKBSZSET ->  Set the block size to 512
>>>>>>>>READ | WRITE Successfully ;)
>>>>>>>>BLKBSZSET ->  Set back to the original block size
>>>>>>>>
>>>>>>>>-Shesha
>>>>>>>>
>>>>>>>>Rob van Nieuwkerk wrote:
>>>>>>>>
>>>>>>>> 
>>>>>>>>
>>>>>>>>      
>>>>>>>>
>>>>>>>>           
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>>>>On Tue, 20 Jul 2004 10:27:57 -0700
>>>>>>>>>Shesha Sreenivasamurthy <shesha@inostor.com> wrote:
>>>>>>>>>
>>>>>>>>>Hi Shesha,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>   
>>>>>>>>>
>>>>>>>>>        
>>>>>>>>>
>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>I am having trouble with O_DIRECT. Trying to read or write from a block 
>>>>>>>>>>device partition.
>>>>>>>>>>
>>>>>>>>>>1. Can O_DIRECT be used on a plain block device partition say 
>>>>>>>>>>"/dev/sda11" without having a filesystem on it.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>     
>>>>>>>>>>
>>>>>>>>>>          
>>>>>>>>>>
>>>>>>>>>>               
>>>>>>>>>>
>>>>>>>>>>                    
>>>>>>>>>>
>>>>>>>>>yes.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>   
>>>>>>>>>
>>>>>>>>>        
>>>>>>>>>
>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>2. If no file system is created then what should be the softblock size. 
>>>>>>>>>>I am using the IOCTL "BLKBSZGET". Is this correct?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>     
>>>>>>>>>>
>>>>>>>>>>          
>>>>>>>>>>
>>>>>>>>>>               
>>>>>>>>>>
>>>>>>>>>>                    
>>>>>>>>>>
>>>>>>>>>yes.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>   
>>>>>>>>>
>>>>>>>>>        
>>>>>>>>>
>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>>>>3. Can we use SEEK_END with O_DIRECT on a partition without filesystem.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>     
>>>>>>>>>>
>>>>>>>>>>          
>>>>>>>>>>
>>>>>>>>>>               
>>>>>>>>>>
>>>>>>>>>>                    
>>>>>>>>>>
>>>>>>>>>yes.
>>>>>>>>>
>>>>>>>>>I'm using these exact things in an application.
>>>>>>>>>
>>>>>>>>>Note that with 2.4 kernels the "granularity" you can use for offset
>>>>>>>>>and r/w size is the softblock size (*).  For 2.6 the requirements are
>>>>>>>>>much more relaxed: it's the device blocksize (typically 512 byte).
>>>>>>>>>
>>>>>>>>>(*): actually one of offset or r/w size has a smaller minimum if
>>>>>>>>>I remember correctly.  Don't remember which one.  But if you assume
>>>>>>>>>the softblock size as a minimum for both you're allways safe.
>>>>>>>>>
>>>>>>>>>	greetings,
>>>>>>>>>	Rob van Nieuwkerk
>>>>>>>>>
>>>>>>>>>--
>>>>>>>>>Kernelnewbies: Help each other learn about the Linux kernel.
>>>>>>>>>Archive:       http://mail.nl.linux.org/kernelnewbies/
>>>>>>>>>FAQ:           http://kernelnewbies.org/faq/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>   
>>>>>>>>>
>>>>>>>>>        
>>>>>>>>>
>>>>>>>>>             
>>>>>>>>>
>>>>>>>>>                  
>>>>>>>>>
>>>>>>>--
>>>>>>>Kernelnewbies: Help each other learn about the Linux kernel.
>>>>>>>Archive:       http://mail.nl.linux.org/kernelnewbies/
>>>>>>>FAQ:           http://kernelnewbies.org/faq/
>>>>>>>
>>>>>>>
>>>>>>>.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>    
>>>>>>>
>>>>>>>         
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>--
>>>>>Kernelnewbies: Help each other learn about the Linux kernel.
>>>>>Archive:       http://mail.nl.linux.org/kernelnewbies/
>>>>>FAQ:           http://kernelnewbies.org/faq/
>>>>>
>>>>>
>>>>>.
>>>>>
>>>>>
>>>>>
>>>>>     
>>>>>
>>>>>          
>>>>>
>>>.
>>>
>>> 
>>>
>>>      
>>>
>
>.
>
>  
>

Re: O_DIRECT

[Rob van Nieuwkerk]

On Wed, 21 Jul 2004 17:00:07 -0700
Shesha Sreenivasamurthy <shesha@inostor.com> wrote:

Hi Shesha,

I've always used the default softblock sizes (eg. did not alter it)
on the devices I've dealt with (which was 1024).

Did you check what this default softblock size is for your device
(with BLKBSZGET) ?

Reading my old code I noticed that I use 3 (instead of the 2 I already
mentioned) alignment requirements:

	1.  read size 1024 byte aligned          (= softblock size)
	2.  read buffer page aligned (4096)
	3.  read/write offset 1024 byte aligned  (= softblock size)

Rule 3. is important too for O_DIRECT !  It may be that rule 2. can be
softblock aligned too, but page alignment certainly won't harm

	greetings,
	Rob van Nieuwkerk

> You are right I am not using any file system. Just did that experiment 
> by creating FS too. That also did not help. I shared that information. 
> You are absolutely right I am dealing with straight partitions.
> 
> (1) With staraight partition even if I explicitly set the block size to 
> 4096 and adher to the rules of O_DIRECT i cannot read/write
> 
> (2) With staraight partition if I set the block size to 1024 and adher 
> to the rules of O_DIRECT i can read/write
> 
> Ruls include : Alligning the buffer & read/write multiple of block size.
> 
> -Shesha
> 
> Rob van Nieuwkerk wrote:
> 
> >On Wed, 21 Jul 2004 16:18:37 -0700
> >Shesha Sreenivasamurthy <shesha@inostor.com> wrote:
> >
> >Hi Shesha,
> >
> >Your original question (and my answer) was about a different situation:
> >
> >        - partition with no filesystem
> >        - access on this raw partition
> >
> >Now you start talking about various file-systems and I get the impression
> >that you access the device through the file-system (although that is
> >not very clear in your text) on a mounted fs.
> >
> >What I told you applied to straight partitions (without mounted a mounted fs).
> >
> >I don't know the exact rules for the various file-systems.
> >They might be different (and some might not support O_DIRECT at all).
> >I expect the same "soft block size rules" to apply for the filesystems
> >that *do* support O_DIRECT.  Note that mounting a partition might
> >change the softblock size of that partition used by the kernel from thereon.
> >
> >
> >	greetings,
> >	Rob van Nieuwkerk
> >
> >PS1: what kernel are you using ?
> >
> >PS2: what are you trying to do ?
> >
> >PS3: you mention "our driver ..".  What is this driver ?
> >
> >
> >
> >  
> >
> >>I could never get it working with 4096 block size.
> >>
> >>I created an EXT3 with blockize=4096 (mkfs.ext3 -b 4096 /dev/sda11). 
> >>Alligned memory to 4096 boundary. Read 8192 Bytes. -> UNSUCCESSFUL :(
> >>I created an XFS with blockize=4096 (mkfs.xfs -f  -b size=4096 
> >>/dev/sda11). Alligned memory to 4096 boundary. Read 8192 Bytes. -> 
> >>UNSUCCESSFUL :(
> >>
> >>SUCCESS HAPPEND ONLY WHEN ....
> >>
> >>1. I created an XFS with blockize=4096 (mkfs.xfs -f  -b size=4096 
> >>/dev/sda11).
> >>                                                       OR
> >>    I created an EXT3 with blockize=4096 (mkfs.ext3 -b 4096 /dev/sda11).
> >>
> >>2. Changed blocksizeto 1024
> >>3. Alligned memory to 1024 boundary.
> >>4  Read 8192 Bytes. -> SUCCESSFUL :)
> >>
> >>-Shesha
> >>
> >>
> >>Rob van Nieuwkerk wrote:
> >>
> >>    
> >>
> >>>On Wed, 21 Jul 2004 11:01:20 -0700
> >>>Shesha Sreenivasamurthy <shesha@inostor.com> wrote:
> >>>
> >>>Hi Shesha,
> >>>
> >>> 
> >>>
> >>>      
> >>>
> >>>>ohhh OK, if the block size is 4096, then the read/write size must be 
> >>>>integer multiple of 4096 ??? is it ???
> >>>>In general should the read/write length be a multiple of block size?
> >>>>   
> >>>>
> >>>>        
> >>>>
> >>>Yes, see my previous emails.
> >>>
> >>>	greetings,
> >>>	Rob van Nieuwkerk
> >>>
> >>> 
> >>>
> >>>      
> >>>
> >>>>Rob van Nieuwkerk wrote:
> >>>>
> >>>>   
> >>>>
> >>>>        
> >>>>
> >>>>>On Wed, 21 Jul 2004 10:10:26 -0700
> >>>>>Shesha Sreenivasamurthy <shesha@inostor.com> wrote:
> >>>>>
> >>>>>Hi Shesha,
> >>>>>
> >>>>>You don't mention what the *size* of your read()/write() is.
> >>>>>Besides a requirement on the alignment of the read/write buffer
> >>>>>the size of the read()/write() must also be OK.
> >>>>>
> >>>>>	greetings,
> >>>>>	Rob van Nieuwkerk
> >>>>>
> >>>>>
> >>>>>
> >>>>>     
> >>>>>
> >>>>>          
> >>>>>
> >>>>>>This is what I found ....
> >>>>>>
> >>>>>>Our driver sets the block size to be 4096. so BLKBSZGET will return 
> >>>>>>4096. So if I allin the memory at 4096 boundary, I cannot read using 
> >>>>>>O_DIRECT. But, if I set the block size to 512.  I can read/write 
> >>>>>>successfully. It also works with 1024, but no with 4096
> >>>>>>
> >>>>>>So the recepie what I am following is ...
> >>>>>>
> >>>>>>BLKBSZGET -> Get original block size
> >>>>>>BLKBSZSET ->  Set the block size to 512
> >>>>>>READ | WRITE Successfully ;)
> >>>>>>BLKBSZSET ->  Set back to the original block size
> >>>>>>
> >>>>>>-Shesha
> >>>>>>
> >>>>>>Rob van Nieuwkerk wrote:
> >>>>>>
> >>>>>>  
> >>>>>>
> >>>>>>       
> >>>>>>
> >>>>>>            
> >>>>>>
> >>>>>>>On Tue, 20 Jul 2004 10:27:57 -0700
> >>>>>>>Shesha Sreenivasamurthy <shesha@inostor.com> wrote:
> >>>>>>>
> >>>>>>>Hi Shesha,
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>    
> >>>>>>>
> >>>>>>>         
> >>>>>>>
> >>>>>>>              
> >>>>>>>
> >>>>>>>>I am having trouble with O_DIRECT. Trying to read or write from a block 
> >>>>>>>>device partition.
> >>>>>>>>
> >>>>>>>>1. Can O_DIRECT be used on a plain block device partition say 
> >>>>>>>>"/dev/sda11" without having a filesystem on it.
> >>>>>>>> 
> >>>>>>>>
> >>>>>>>>      
> >>>>>>>>
> >>>>>>>>           
> >>>>>>>>
> >>>>>>>>                
> >>>>>>>>
> >>>>>>>yes.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>    
> >>>>>>>
> >>>>>>>         
> >>>>>>>
> >>>>>>>              
> >>>>>>>
> >>>>>>>>2. If no file system is created then what should be the softblock size. 
> >>>>>>>>I am using the IOCTL "BLKBSZGET". Is this correct?
> >>>>>>>> 
> >>>>>>>>
> >>>>>>>>      
> >>>>>>>>
> >>>>>>>>           
> >>>>>>>>
> >>>>>>>>                
> >>>>>>>>
> >>>>>>>yes.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>    
> >>>>>>>
> >>>>>>>         
> >>>>>>>
> >>>>>>>              
> >>>>>>>
> >>>>>>>>3. Can we use SEEK_END with O_DIRECT on a partition without filesystem.
> >>>>>>>> 
> >>>>>>>>
> >>>>>>>>      
> >>>>>>>>
> >>>>>>>>           
> >>>>>>>>
> >>>>>>>>                
> >>>>>>>>
> >>>>>>>yes.
> >>>>>>>
> >>>>>>>I'm using these exact things in an application.
> >>>>>>>
> >>>>>>>Note that with 2.4 kernels the "granularity" you can use for offset
> >>>>>>>and r/w size is the softblock size (*).  For 2.6 the requirements are
> >>>>>>>much more relaxed: it's the device blocksize (typically 512 byte).
> >>>>>>>
> >>>>>>>(*): actually one of offset or r/w size has a smaller minimum if
> >>>>>>>I remember correctly.  Don't remember which one.  But if you assume
> >>>>>>>the softblock size as a minimum for both you're allways safe.
> >>>>>>>
> >>>>>>>	greetings,
> >>>>>>>	Rob van Nieuwkerk
> >>>>>>>
> >>>>>>>--
> >>>>>>>Kernelnewbies: Help each other learn about the Linux kernel.
> >>>>>>>Archive:       http://mail.nl.linux.org/kernelnewbies/
> >>>>>>>FAQ:           http://kernelnewbies.org/faq/
> >>>>>>>
> >>>>>>>
> >>>>>>>.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>    
> >>>>>>>
> >>>>>>>         
> >>>>>>>
> >>>>>>>              
> >>>>>>>
> >>>>>--
> >>>>>Kernelnewbies: Help each other learn about the Linux kernel.
> >>>>>Archive:       http://mail.nl.linux.org/kernelnewbies/
> >>>>>FAQ:           http://kernelnewbies.org/faq/
> >>>>>
> >>>>>
> >>>>>.
> >>>>>
> >>>>>
> >>>>>
> >>>>>     
> >>>>>
> >>>>>          
> >>>>>
> >>>--
> >>>Kernelnewbies: Help each other learn about the Linux kernel.
> >>>Archive:       http://mail.nl.linux.org/kernelnewbies/
> >>>FAQ:           http://kernelnewbies.org/faq/
> >>>
> >>>
> >>>.
> >>>
> >>> 
> >>>
> >>>      
> >>>
> >
> >.
> >
> >  
> >
> 

Re: O_DIRECT

[Shesha Sreenivasamurthy]

You are right I am not using any file system. Just did that experiment 
by creating FS too. That also did not help. I shared that information. 
You are absolutely right I am dealing with straight partitions.

(1) With staraight partition even if I explicitly set the block size to 
4096 and adher to the rules of O_DIRECT i cannot read/write

(2) With staraight partition if I set the block size to 1024 and adher 
to the rules of O_DIRECT i can read/write

Ruls include : Alligning the buffer & read/write multiple of block size.

-Shesha

Rob van Nieuwkerk wrote:

>On Wed, 21 Jul 2004 16:18:37 -0700
>Shesha Sreenivasamurthy <shesha@inostor.com> wrote:
>
>Hi Shesha,
>
>Your original question (and my answer) was about a different situation:
>
>        - partition with no filesystem
>        - access on this raw partition
>
>Now you start talking about various file-systems and I get the impression
>that you access the device through the file-system (although that is
>not very clear in your text) on a mounted fs.
>
>What I told you applied to straight partitions (without mounted a mounted fs).
>
>I don't know the exact rules for the various file-systems.
>They might be different (and some might not support O_DIRECT at all).
>I expect the same "soft block size rules" to apply for the filesystems
>that *do* support O_DIRECT.  Note that mounting a partition might
>change the softblock size of that partition used by the kernel from thereon.
>
>
>	greetings,
>	Rob van Nieuwkerk
>
>PS1: what kernel are you using ?
>
>PS2: what are you trying to do ?
>
>PS3: you mention "our driver ..".  What is this driver ?
>
>
>
>  
>
>>I could never get it working with 4096 block size.
>>
>>I created an EXT3 with blockize=4096 (mkfs.ext3 -b 4096 /dev/sda11). 
>>Alligned memory to 4096 boundary. Read 8192 Bytes. -> UNSUCCESSFUL :(
>>I created an XFS with blockize=4096 (mkfs.xfs -f  -b size=4096 
>>/dev/sda11). Alligned memory to 4096 boundary. Read 8192 Bytes. -> 
>>UNSUCCESSFUL :(
>>
>>SUCCESS HAPPEND ONLY WHEN ....
>>
>>1. I created an XFS with blockize=4096 (mkfs.xfs -f  -b size=4096 
>>/dev/sda11).
>>                                                       OR
>>    I created an EXT3 with blockize=4096 (mkfs.ext3 -b 4096 /dev/sda11).
>>
>>2. Changed blocksizeto 1024
>>3. Alligned memory to 1024 boundary.
>>4  Read 8192 Bytes. -> SUCCESSFUL :)
>>
>>-Shesha
>>
>>
>>Rob van Nieuwkerk wrote:
>>
>>    
>>
>>>On Wed, 21 Jul 2004 11:01:20 -0700
>>>Shesha Sreenivasamurthy <shesha@inostor.com> wrote:
>>>
>>>Hi Shesha,
>>>
>>> 
>>>
>>>      
>>>
>>>>ohhh OK, if the block size is 4096, then the read/write size must be 
>>>>integer multiple of 4096 ??? is it ???
>>>>In general should the read/write length be a multiple of block size?
>>>>   
>>>>
>>>>        
>>>>
>>>Yes, see my previous emails.
>>>
>>>	greetings,
>>>	Rob van Nieuwkerk
>>>
>>> 
>>>
>>>      
>>>
>>>>Rob van Nieuwkerk wrote:
>>>>
>>>>   
>>>>
>>>>        
>>>>
>>>>>On Wed, 21 Jul 2004 10:10:26 -0700
>>>>>Shesha Sreenivasamurthy <shesha@inostor.com> wrote:
>>>>>
>>>>>Hi Shesha,
>>>>>
>>>>>You don't mention what the *size* of your read()/write() is.
>>>>>Besides a requirement on the alignment of the read/write buffer
>>>>>the size of the read()/write() must also be OK.
>>>>>
>>>>>	greetings,
>>>>>	Rob van Nieuwkerk
>>>>>
>>>>>
>>>>>
>>>>>     
>>>>>
>>>>>          
>>>>>
>>>>>>This is what I found ....
>>>>>>
>>>>>>Our driver sets the block size to be 4096. so BLKBSZGET will return 
>>>>>>4096. So if I allin the memory at 4096 boundary, I cannot read using 
>>>>>>O_DIRECT. But, if I set the block size to 512.  I can read/write 
>>>>>>successfully. It also works with 1024, but no with 4096
>>>>>>
>>>>>>So the recepie what I am following is ...
>>>>>>
>>>>>>BLKBSZGET -> Get original block size
>>>>>>BLKBSZSET ->  Set the block size to 512
>>>>>>READ | WRITE Successfully ;)
>>>>>>BLKBSZSET ->  Set back to the original block size
>>>>>>
>>>>>>-Shesha
>>>>>>
>>>>>>Rob van Nieuwkerk wrote:
>>>>>>
>>>>>>  
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>>>On Tue, 20 Jul 2004 10:27:57 -0700
>>>>>>>Shesha Sreenivasamurthy <shesha@inostor.com> wrote:
>>>>>>>
>>>>>>>Hi Shesha,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>    
>>>>>>>
>>>>>>>         
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>I am having trouble with O_DIRECT. Trying to read or write from a block 
>>>>>>>>device partition.
>>>>>>>>
>>>>>>>>1. Can O_DIRECT be used on a plain block device partition say 
>>>>>>>>"/dev/sda11" without having a filesystem on it.
>>>>>>>> 
>>>>>>>>
>>>>>>>>      
>>>>>>>>
>>>>>>>>           
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>>yes.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>    
>>>>>>>
>>>>>>>         
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>2. If no file system is created then what should be the softblock size. 
>>>>>>>>I am using the IOCTL "BLKBSZGET". Is this correct?
>>>>>>>> 
>>>>>>>>
>>>>>>>>      
>>>>>>>>
>>>>>>>>           
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>>yes.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>    
>>>>>>>
>>>>>>>         
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>>>>3. Can we use SEEK_END with O_DIRECT on a partition without filesystem.
>>>>>>>> 
>>>>>>>>
>>>>>>>>      
>>>>>>>>
>>>>>>>>           
>>>>>>>>
>>>>>>>>                
>>>>>>>>
>>>>>>>yes.
>>>>>>>
>>>>>>>I'm using these exact things in an application.
>>>>>>>
>>>>>>>Note that with 2.4 kernels the "granularity" you can use for offset
>>>>>>>and r/w size is the softblock size (*).  For 2.6 the requirements are
>>>>>>>much more relaxed: it's the device blocksize (typically 512 byte).
>>>>>>>
>>>>>>>(*): actually one of offset or r/w size has a smaller minimum if
>>>>>>>I remember correctly.  Don't remember which one.  But if you assume
>>>>>>>the softblock size as a minimum for both you're allways safe.
>>>>>>>
>>>>>>>	greetings,
>>>>>>>	Rob van Nieuwkerk
>>>>>>>
>>>>>>>--
>>>>>>>Kernelnewbies: Help each other learn about the Linux kernel.
>>>>>>>Archive:       http://mail.nl.linux.org/kernelnewbies/
>>>>>>>FAQ:           http://kernelnewbies.org/faq/
>>>>>>>
>>>>>>>
>>>>>>>.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>    
>>>>>>>
>>>>>>>         
>>>>>>>
>>>>>>>              
>>>>>>>
>>>>>--
>>>>>Kernelnewbies: Help each other learn about the Linux kernel.
>>>>>Archive:       http://mail.nl.linux.org/kernelnewbies/
>>>>>FAQ:           http://kernelnewbies.org/faq/
>>>>>
>>>>>
>>>>>.
>>>>>
>>>>>
>>>>>
>>>>>     
>>>>>
>>>>>          
>>>>>
>>>--
>>>Kernelnewbies: Help each other learn about the Linux kernel.
>>>Archive:       http://mail.nl.linux.org/kernelnewbies/
>>>FAQ:           http://kernelnewbies.org/faq/
>>>
>>>
>>>.
>>>
>>> 
>>>
>>>      
>>>
>
>.
>
>  
>

Re: O_DIRECT

[Rob van Nieuwkerk]

On Wed, 21 Jul 2004 16:18:37 -0700
Shesha Sreenivasamurthy <shesha@inostor.com> wrote:

Hi Shesha,

Your original question (and my answer) was about a different situation:

        - partition with no filesystem
        - access on this raw partition

Now you start talking about various file-systems and I get the impression
that you access the device through the file-system (although that is
not very clear in your text) on a mounted fs.

What I told you applied to straight partitions (without mounted a mounted fs).

I don't know the exact rules for the various file-systems.
They might be different (and some might not support O_DIRECT at all).
I expect the same "soft block size rules" to apply for the filesystems
that *do* support O_DIRECT.  Note that mounting a partition might
change the softblock size of that partition used by the kernel from thereon.


	greetings,
	Rob van Nieuwkerk

PS1: what kernel are you using ?

PS2: what are you trying to do ?

PS3: you mention "our driver ..".  What is this driver ?



> I could never get it working with 4096 block size.
> 
> I created an EXT3 with blockize=4096 (mkfs.ext3 -b 4096 /dev/sda11). 
> Alligned memory to 4096 boundary. Read 8192 Bytes. -> UNSUCCESSFUL :(
> I created an XFS with blockize=4096 (mkfs.xfs -f  -b size=4096 
> /dev/sda11). Alligned memory to 4096 boundary. Read 8192 Bytes. -> 
> UNSUCCESSFUL :(
> 
> SUCCESS HAPPEND ONLY WHEN ....
> 
> 1. I created an XFS with blockize=4096 (mkfs.xfs -f  -b size=4096 
> /dev/sda11).
>                                                        OR
>     I created an EXT3 with blockize=4096 (mkfs.ext3 -b 4096 /dev/sda11).
> 
> 2. Changed blocksizeto 1024
> 3. Alligned memory to 1024 boundary.
> 4  Read 8192 Bytes. -> SUCCESSFUL :)
> 
> -Shesha
> 
> 
> Rob van Nieuwkerk wrote:
> 
> >On Wed, 21 Jul 2004 11:01:20 -0700
> >Shesha Sreenivasamurthy <shesha@inostor.com> wrote:
> >
> >Hi Shesha,
> >
> >  
> >
> >>ohhh OK, if the block size is 4096, then the read/write size must be 
> >>integer multiple of 4096 ??? is it ???
> >>In general should the read/write length be a multiple of block size?
> >>    
> >>
> >
> >Yes, see my previous emails.
> >
> >	greetings,
> >	Rob van Nieuwkerk
> >
> >  
> >
> >>Rob van Nieuwkerk wrote:
> >>
> >>    
> >>
> >>>On Wed, 21 Jul 2004 10:10:26 -0700
> >>>Shesha Sreenivasamurthy <shesha@inostor.com> wrote:
> >>>
> >>>Hi Shesha,
> >>>
> >>>You don't mention what the *size* of your read()/write() is.
> >>>Besides a requirement on the alignment of the read/write buffer
> >>>the size of the read()/write() must also be OK.
> >>>
> >>>	greetings,
> >>>	Rob van Nieuwkerk
> >>>
> >>> 
> >>>
> >>>      
> >>>
> >>>>This is what I found ....
> >>>>
> >>>>Our driver sets the block size to be 4096. so BLKBSZGET will return 
> >>>>4096. So if I allin the memory at 4096 boundary, I cannot read using 
> >>>>O_DIRECT. But, if I set the block size to 512.  I can read/write 
> >>>>successfully. It also works with 1024, but no with 4096
> >>>>
> >>>>So the recepie what I am following is ...
> >>>>
> >>>>BLKBSZGET -> Get original block size
> >>>>BLKBSZSET ->  Set the block size to 512
> >>>>READ | WRITE Successfully ;)
> >>>>BLKBSZSET ->  Set back to the original block size
> >>>>
> >>>>-Shesha
> >>>>
> >>>>Rob van Nieuwkerk wrote:
> >>>>
> >>>>   
> >>>>
> >>>>        
> >>>>
> >>>>>On Tue, 20 Jul 2004 10:27:57 -0700
> >>>>>Shesha Sreenivasamurthy <shesha@inostor.com> wrote:
> >>>>>
> >>>>>Hi Shesha,
> >>>>>
> >>>>>
> >>>>>
> >>>>>     
> >>>>>
> >>>>>          
> >>>>>
> >>>>>>I am having trouble with O_DIRECT. Trying to read or write from a block 
> >>>>>>device partition.
> >>>>>>
> >>>>>>1. Can O_DIRECT be used on a plain block device partition say 
> >>>>>>"/dev/sda11" without having a filesystem on it.
> >>>>>>  
> >>>>>>
> >>>>>>       
> >>>>>>
> >>>>>>            
> >>>>>>
> >>>>>yes.
> >>>>>
> >>>>>
> >>>>>
> >>>>>     
> >>>>>
> >>>>>          
> >>>>>
> >>>>>>2. If no file system is created then what should be the softblock size. 
> >>>>>>I am using the IOCTL "BLKBSZGET". Is this correct?
> >>>>>>  
> >>>>>>
> >>>>>>       
> >>>>>>
> >>>>>>            
> >>>>>>
> >>>>>yes.
> >>>>>
> >>>>>
> >>>>>
> >>>>>     
> >>>>>
> >>>>>          
> >>>>>
> >>>>>>3. Can we use SEEK_END with O_DIRECT on a partition without filesystem.
> >>>>>>  
> >>>>>>
> >>>>>>       
> >>>>>>
> >>>>>>            
> >>>>>>
> >>>>>yes.
> >>>>>
> >>>>>I'm using these exact things in an application.
> >>>>>
> >>>>>Note that with 2.4 kernels the "granularity" you can use for offset
> >>>>>and r/w size is the softblock size (*).  For 2.6 the requirements are
> >>>>>much more relaxed: it's the device blocksize (typically 512 byte).
> >>>>>
> >>>>>(*): actually one of offset or r/w size has a smaller minimum if
> >>>>>I remember correctly.  Don't remember which one.  But if you assume
> >>>>>the softblock size as a minimum for both you're allways safe.
> >>>>>
> >>>>>	greetings,
> >>>>>	Rob van Nieuwkerk
> >>>>>
> >>>>>--
> >>>>>Kernelnewbies: Help each other learn about the Linux kernel.
> >>>>>Archive:       http://mail.nl.linux.org/kernelnewbies/
> >>>>>FAQ:           http://kernelnewbies.org/faq/
> >>>>>
> >>>>>
> >>>>>.
> >>>>>
> >>>>>
> >>>>>
> >>>>>     
> >>>>>
> >>>>>          
> >>>>>
> >>>--
> >>>Kernelnewbies: Help each other learn about the Linux kernel.
> >>>Archive:       http://mail.nl.linux.org/kernelnewbies/
> >>>FAQ:           http://kernelnewbies.org/faq/
> >>>
> >>>
> >>>.
> >>>
> >>> 
> >>>
> >>>      
> >>>
> >
> >--
> >Kernelnewbies: Help each other learn about the Linux kernel.
> >Archive:       http://mail.nl.linux.org/kernelnewbies/
> >FAQ:           http://kernelnewbies.org/faq/
> >
> >
> >.
> >
> >  
> >
> 

Re: O_DIRECT

[Shesha Sreenivasamurthy]

I could never get it working with 4096 block size.

I created an EXT3 with blockize=4096 (mkfs.ext3 -b 4096 /dev/sda11). 
Alligned memory to 4096 boundary. Read 8192 Bytes. -> UNSUCCESSFUL :(
I created an XFS with blockize=4096 (mkfs.xfs -f  -b size=4096 
/dev/sda11). Alligned memory to 4096 boundary. Read 8192 Bytes. -> 
UNSUCCESSFUL :(

SUCCESS HAPPEND ONLY WHEN ....

1. I created an XFS with blockize=4096 (mkfs.xfs -f  -b size=4096 
/dev/sda11).
                                                       OR
    I created an EXT3 with blockize=4096 (mkfs.ext3 -b 4096 /dev/sda11).

2. Changed blocksizeto 1024
3. Alligned memory to 1024 boundary.
4  Read 8192 Bytes. -> SUCCESSFUL :)

-Shesha


Rob van Nieuwkerk wrote:

>On Wed, 21 Jul 2004 11:01:20 -0700
>Shesha Sreenivasamurthy <shesha@inostor.com> wrote:
>
>Hi Shesha,
>
>  
>
>>ohhh OK, if the block size is 4096, then the read/write size must be 
>>integer multiple of 4096 ??? is it ???
>>In general should the read/write length be a multiple of block size?
>>    
>>
>
>Yes, see my previous emails.
>
>	greetings,
>	Rob van Nieuwkerk
>
>  
>
>>Rob van Nieuwkerk wrote:
>>
>>    
>>
>>>On Wed, 21 Jul 2004 10:10:26 -0700
>>>Shesha Sreenivasamurthy <shesha@inostor.com> wrote:
>>>
>>>Hi Shesha,
>>>
>>>You don't mention what the *size* of your read()/write() is.
>>>Besides a requirement on the alignment of the read/write buffer
>>>the size of the read()/write() must also be OK.
>>>
>>>	greetings,
>>>	Rob van Nieuwkerk
>>>
>>> 
>>>
>>>      
>>>
>>>>This is what I found ....
>>>>
>>>>Our driver sets the block size to be 4096. so BLKBSZGET will return 
>>>>4096. So if I allin the memory at 4096 boundary, I cannot read using 
>>>>O_DIRECT. But, if I set the block size to 512.  I can read/write 
>>>>successfully. It also works with 1024, but no with 4096
>>>>
>>>>So the recepie what I am following is ...
>>>>
>>>>BLKBSZGET -> Get original block size
>>>>BLKBSZSET ->  Set the block size to 512
>>>>READ | WRITE Successfully ;)
>>>>BLKBSZSET ->  Set back to the original block size
>>>>
>>>>-Shesha
>>>>
>>>>Rob van Nieuwkerk wrote:
>>>>
>>>>   
>>>>
>>>>        
>>>>
>>>>>On Tue, 20 Jul 2004 10:27:57 -0700
>>>>>Shesha Sreenivasamurthy <shesha@inostor.com> wrote:
>>>>>
>>>>>Hi Shesha,
>>>>>
>>>>>
>>>>>
>>>>>     
>>>>>
>>>>>          
>>>>>
>>>>>>I am having trouble with O_DIRECT. Trying to read or write from a block 
>>>>>>device partition.
>>>>>>
>>>>>>1. Can O_DIRECT be used on a plain block device partition say 
>>>>>>"/dev/sda11" without having a filesystem on it.
>>>>>>  
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>yes.
>>>>>
>>>>>
>>>>>
>>>>>     
>>>>>
>>>>>          
>>>>>
>>>>>>2. If no file system is created then what should be the softblock size. 
>>>>>>I am using the IOCTL "BLKBSZGET". Is this correct?
>>>>>>  
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>yes.
>>>>>
>>>>>
>>>>>
>>>>>     
>>>>>
>>>>>          
>>>>>
>>>>>>3. Can we use SEEK_END with O_DIRECT on a partition without filesystem.
>>>>>>  
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>yes.
>>>>>
>>>>>I'm using these exact things in an application.
>>>>>
>>>>>Note that with 2.4 kernels the "granularity" you can use for offset
>>>>>and r/w size is the softblock size (*).  For 2.6 the requirements are
>>>>>much more relaxed: it's the device blocksize (typically 512 byte).
>>>>>
>>>>>(*): actually one of offset or r/w size has a smaller minimum if
>>>>>I remember correctly.  Don't remember which one.  But if you assume
>>>>>the softblock size as a minimum for both you're allways safe.
>>>>>
>>>>>	greetings,
>>>>>	Rob van Nieuwkerk
>>>>>
>>>>>--
>>>>>Kernelnewbies: Help each other learn about the Linux kernel.
>>>>>Archive:       http://mail.nl.linux.org/kernelnewbies/
>>>>>FAQ:           http://kernelnewbies.org/faq/
>>>>>
>>>>>
>>>>>.
>>>>>
>>>>>
>>>>>
>>>>>     
>>>>>
>>>>>          
>>>>>
>>>--
>>>Kernelnewbies: Help each other learn about the Linux kernel.
>>>Archive:       http://mail.nl.linux.org/kernelnewbies/
>>>FAQ:           http://kernelnewbies.org/faq/
>>>
>>>
>>>.
>>>
>>> 
>>>
>>>      
>>>
>
>--
>Kernelnewbies: Help each other learn about the Linux kernel.
>Archive:       http://mail.nl.linux.org/kernelnewbies/
>FAQ:           http://kernelnewbies.org/faq/
>
>
>.
>
>  
>

Re: O_DIRECT

[Shesha Sreenivasamurthy]

Thank you very much ....
-Shesha

Rob van Nieuwkerk wrote:

>On Wed, 21 Jul 2004 11:01:20 -0700
>Shesha Sreenivasamurthy <shesha@inostor.com> wrote:
>
>Hi Shesha,
>
>  
>
>>ohhh OK, if the block size is 4096, then the read/write size must be 
>>integer multiple of 4096 ??? is it ???
>>In general should the read/write length be a multiple of block size?
>>    
>>
>
>Yes, see my previous emails.
>
>	greetings,
>	Rob van Nieuwkerk
>
>  
>
>>Rob van Nieuwkerk wrote:
>>
>>    
>>
>>>On Wed, 21 Jul 2004 10:10:26 -0700
>>>Shesha Sreenivasamurthy <shesha@inostor.com> wrote:
>>>
>>>Hi Shesha,
>>>
>>>You don't mention what the *size* of your read()/write() is.
>>>Besides a requirement on the alignment of the read/write buffer
>>>the size of the read()/write() must also be OK.
>>>
>>>	greetings,
>>>	Rob van Nieuwkerk
>>>
>>> 
>>>
>>>      
>>>
>>>>This is what I found ....
>>>>
>>>>Our driver sets the block size to be 4096. so BLKBSZGET will return 
>>>>4096. So if I allin the memory at 4096 boundary, I cannot read using 
>>>>O_DIRECT. But, if I set the block size to 512.  I can read/write 
>>>>successfully. It also works with 1024, but no with 4096
>>>>
>>>>So the recepie what I am following is ...
>>>>
>>>>BLKBSZGET -> Get original block size
>>>>BLKBSZSET ->  Set the block size to 512
>>>>READ | WRITE Successfully ;)
>>>>BLKBSZSET ->  Set back to the original block size
>>>>
>>>>-Shesha
>>>>
>>>>Rob van Nieuwkerk wrote:
>>>>
>>>>   
>>>>
>>>>        
>>>>
>>>>>On Tue, 20 Jul 2004 10:27:57 -0700
>>>>>Shesha Sreenivasamurthy <shesha@inostor.com> wrote:
>>>>>
>>>>>Hi Shesha,
>>>>>
>>>>>
>>>>>
>>>>>     
>>>>>
>>>>>          
>>>>>
>>>>>>I am having trouble with O_DIRECT. Trying to read or write from a block 
>>>>>>device partition.
>>>>>>
>>>>>>1. Can O_DIRECT be used on a plain block device partition say 
>>>>>>"/dev/sda11" without having a filesystem on it.
>>>>>>  
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>yes.
>>>>>
>>>>>
>>>>>
>>>>>     
>>>>>
>>>>>          
>>>>>
>>>>>>2. If no file system is created then what should be the softblock size. 
>>>>>>I am using the IOCTL "BLKBSZGET". Is this correct?
>>>>>>  
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>yes.
>>>>>
>>>>>
>>>>>
>>>>>     
>>>>>
>>>>>          
>>>>>
>>>>>>3. Can we use SEEK_END with O_DIRECT on a partition without filesystem.
>>>>>>  
>>>>>>
>>>>>>       
>>>>>>
>>>>>>            
>>>>>>
>>>>>yes.
>>>>>
>>>>>I'm using these exact things in an application.
>>>>>
>>>>>Note that with 2.4 kernels the "granularity" you can use for offset
>>>>>and r/w size is the softblock size (*).  For 2.6 the requirements are
>>>>>much more relaxed: it's the device blocksize (typically 512 byte).
>>>>>
>>>>>(*): actually one of offset or r/w size has a smaller minimum if
>>>>>I remember correctly.  Don't remember which one.  But if you assume
>>>>>the softblock size as a minimum for both you're allways safe.
>>>>>
>>>>>	greetings,
>>>>>	Rob van Nieuwkerk
>>>>>
>>>>>--
>>>>>Kernelnewbies: Help each other learn about the Linux kernel.
>>>>>Archive:       http://mail.nl.linux.org/kernelnewbies/
>>>>>FAQ:           http://kernelnewbies.org/faq/
>>>>>
>>>>>
>>>>>.
>>>>>
>>>>>
>>>>>
>>>>>     
>>>>>
>>>>>          
>>>>>
>>>--
>>>Kernelnewbies: Help each other learn about the Linux kernel.
>>>Archive:       http://mail.nl.linux.org/kernelnewbies/
>>>FAQ:           http://kernelnewbies.org/faq/
>>>
>>>
>>>.
>>>
>>> 
>>>
>>>      
>>>
>
>--
>Kernelnewbies: Help each other learn about the Linux kernel.
>Archive:       http://mail.nl.linux.org/kernelnewbies/
>FAQ:           http://kernelnewbies.org/faq/
>
>
>.
>
>  
>

Re: O_DIRECT

[Rob van Nieuwkerk]

On Wed, 21 Jul 2004 11:01:20 -0700
Shesha Sreenivasamurthy <shesha@inostor.com> wrote:

Hi Shesha,

> ohhh OK, if the block size is 4096, then the read/write size must be 
> integer multiple of 4096 ??? is it ???
> In general should the read/write length be a multiple of block size?

Yes, see my previous emails.

	greetings,
	Rob van Nieuwkerk

> Rob van Nieuwkerk wrote:
> 
> >On Wed, 21 Jul 2004 10:10:26 -0700
> >Shesha Sreenivasamurthy <shesha@inostor.com> wrote:
> >
> >Hi Shesha,
> >
> >You don't mention what the *size* of your read()/write() is.
> >Besides a requirement on the alignment of the read/write buffer
> >the size of the read()/write() must also be OK.
> >
> >	greetings,
> >	Rob van Nieuwkerk
> >
> >  
> >
> >>This is what I found ....
> >>
> >>Our driver sets the block size to be 4096. so BLKBSZGET will return 
> >>4096. So if I allin the memory at 4096 boundary, I cannot read using 
> >>O_DIRECT. But, if I set the block size to 512.  I can read/write 
> >>successfully. It also works with 1024, but no with 4096
> >>
> >>So the recepie what I am following is ...
> >>
> >>BLKBSZGET -> Get original block size
> >>BLKBSZSET ->  Set the block size to 512
> >>READ | WRITE Successfully ;)
> >>BLKBSZSET ->  Set back to the original block size
> >>
> >>-Shesha
> >>
> >>Rob van Nieuwkerk wrote:
> >>
> >>    
> >>
> >>>On Tue, 20 Jul 2004 10:27:57 -0700
> >>>Shesha Sreenivasamurthy <shesha@inostor.com> wrote:
> >>>
> >>>Hi Shesha,
> >>>
> >>> 
> >>>
> >>>      
> >>>
> >>>>I am having trouble with O_DIRECT. Trying to read or write from a block 
> >>>>device partition.
> >>>>
> >>>>1. Can O_DIRECT be used on a plain block device partition say 
> >>>>"/dev/sda11" without having a filesystem on it.
> >>>>   
> >>>>
> >>>>        
> >>>>
> >>>yes.
> >>>
> >>> 
> >>>
> >>>      
> >>>
> >>>>2. If no file system is created then what should be the softblock size. 
> >>>>I am using the IOCTL "BLKBSZGET". Is this correct?
> >>>>   
> >>>>
> >>>>        
> >>>>
> >>>yes.
> >>>
> >>> 
> >>>
> >>>      
> >>>
> >>>>3. Can we use SEEK_END with O_DIRECT on a partition without filesystem.
> >>>>   
> >>>>
> >>>>        
> >>>>
> >>>yes.
> >>>
> >>>I'm using these exact things in an application.
> >>>
> >>>Note that with 2.4 kernels the "granularity" you can use for offset
> >>>and r/w size is the softblock size (*).  For 2.6 the requirements are
> >>>much more relaxed: it's the device blocksize (typically 512 byte).
> >>>
> >>>(*): actually one of offset or r/w size has a smaller minimum if
> >>>I remember correctly.  Don't remember which one.  But if you assume
> >>>the softblock size as a minimum for both you're allways safe.
> >>>
> >>>	greetings,
> >>>	Rob van Nieuwkerk
> >>>
> >>>--
> >>>Kernelnewbies: Help each other learn about the Linux kernel.
> >>>Archive:       http://mail.nl.linux.org/kernelnewbies/
> >>>FAQ:           http://kernelnewbies.org/faq/
> >>>
> >>>
> >>>.
> >>>
> >>> 
> >>>
> >>>      
> >>>
> >
> >--
> >Kernelnewbies: Help each other learn about the Linux kernel.
> >Archive:       http://mail.nl.linux.org/kernelnewbies/
> >FAQ:           http://kernelnewbies.org/faq/
> >
> >
> >.
> >
> >  
> >
> 

Re: O_DIRECT

[Rob van Nieuwkerk]

On Wed, 21 Jul 2004 10:10:26 -0700
Shesha Sreenivasamurthy <shesha@inostor.com> wrote:

Hi Shesha,

You don't mention what the *size* of your read()/write() is.
Besides a requirement on the alignment of the read/write buffer
the size of the read()/write() must also be OK.

	greetings,
	Rob van Nieuwkerk

> This is what I found ....
> 
> Our driver sets the block size to be 4096. so BLKBSZGET will return 
> 4096. So if I allin the memory at 4096 boundary, I cannot read using 
> O_DIRECT. But, if I set the block size to 512.  I can read/write 
> successfully. It also works with 1024, but no with 4096
> 
> So the recepie what I am following is ...
> 
> BLKBSZGET -> Get original block size
> BLKBSZSET ->  Set the block size to 512
> READ | WRITE Successfully ;)
> BLKBSZSET ->  Set back to the original block size
> 
> -Shesha
> 
> Rob van Nieuwkerk wrote:
> 
> >On Tue, 20 Jul 2004 10:27:57 -0700
> >Shesha Sreenivasamurthy <shesha@inostor.com> wrote:
> >
> >Hi Shesha,
> >
> >  
> >
> >>I am having trouble with O_DIRECT. Trying to read or write from a block 
> >>device partition.
> >>
> >>1. Can O_DIRECT be used on a plain block device partition say 
> >>"/dev/sda11" without having a filesystem on it.
> >>    
> >>
> >
> >yes.
> >
> >  
> >
> >>2. If no file system is created then what should be the softblock size. 
> >>I am using the IOCTL "BLKBSZGET". Is this correct?
> >>    
> >>
> >
> >yes.
> >
> >  
> >
> >>3. Can we use SEEK_END with O_DIRECT on a partition without filesystem.
> >>    
> >>
> >
> >yes.
> >
> >I'm using these exact things in an application.
> >
> >Note that with 2.4 kernels the "granularity" you can use for offset
> >and r/w size is the softblock size (*).  For 2.6 the requirements are
> >much more relaxed: it's the device blocksize (typically 512 byte).
> >
> >(*): actually one of offset or r/w size has a smaller minimum if
> >I remember correctly.  Don't remember which one.  But if you assume
> >the softblock size as a minimum for both you're allways safe.
> >
> >	greetings,
> >	Rob van Nieuwkerk
> >
> >--
> >Kernelnewbies: Help each other learn about the Linux kernel.
> >Archive:       http://mail.nl.linux.org/kernelnewbies/
> >FAQ:           http://kernelnewbies.org/faq/
> >
> >
> >.
> >
> >  
> >
> 

Re: O_DIRECT

[Devel]

Hi all,
how i can write data on a partition without a filesystem??

Saluti Carlo!

Il Tue, 20 Jul 2004 20:48:24 +0200
bert hubert <ahu@ds9a.nl> scrisse:

> On Tue, Jul 20, 2004 at 10:27:57AM -0700, Shesha Sreenivasamurthy wrote:
> > I am having trouble with O_DIRECT. Trying to read or write from a block 
> > device partition.
> > 
> > 1. Can O_DIRECT be used on a plain block device partition say 
> > "/dev/sda11" without having a filesystem on it.
> 
> As fas as I know, yes, but be aware that O_DIRECT requires page aligned
> addresses! (an integral of 4096 on most systems).
> 
> > 2. If no file system is created then what should be the softblock size. 
> > I am using the IOCTL "BLKBSZGET". Is this correct?
> 
> No idea what you mean - but see above about the aligned addresses.
> 
> > 3. Can we use SEEK_END with O_DIRECT on a partition without filesystem.
> 
> I see no reason why not. Good luck!
> 
> -- 
> http://www.PowerDNS.com      Open source, database driven DNS Software 
> http://lartc.org           Linux Advanced Routing & Traffic Control HOWTO
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: O_DIRECT

[Rob van Nieuwkerk]

On Tue, 20 Jul 2004 20:48:24 +0200
bert hubert <ahu@ds9a.nl> wrote:

Hi Bert,

> On Tue, Jul 20, 2004 at 10:27:57AM -0700, Shesha Sreenivasamurthy wrote:
> > I am having trouble with O_DIRECT. Trying to read or write from a block 
> > device partition.
> > 
> > 1. Can O_DIRECT be used on a plain block device partition say 
> > "/dev/sda11" without having a filesystem on it.
> 
> As fas as I know, yes, but be aware that O_DIRECT requires page aligned
> addresses! (an integral of 4096 on most systems).

"soft block size" aligned.  The soft block size can be smaller than
a page (but not bigger).  There's a requirement for:

	- r/w offset
	- r/w size

	greetings,
	Rob van Nieuwkerk

Re: O_DIRECT

[Rob van Nieuwkerk]

On Tue, 20 Jul 2004 10:27:57 -0700
Shesha Sreenivasamurthy <shesha@inostor.com> wrote:

Hi Shesha,

> I am having trouble with O_DIRECT. Trying to read or write from a block 
> device partition.
> 
> 1. Can O_DIRECT be used on a plain block device partition say 
> "/dev/sda11" without having a filesystem on it.

yes.

> 2. If no file system is created then what should be the softblock size. 
> I am using the IOCTL "BLKBSZGET". Is this correct?

yes.

> 3. Can we use SEEK_END with O_DIRECT on a partition without filesystem.

yes.

I'm using these exact things in an application.

Note that with 2.4 kernels the "granularity" you can use for offset
and r/w size is the softblock size (*).  For 2.6 the requirements are
much more relaxed: it's the device blocksize (typically 512 byte).

(*): actually one of offset or r/w size has a smaller minimum if
I remember correctly.  Don't remember which one.  But if you assume
the softblock size as a minimum for both you're allways safe.

	greetings,
	Rob van Nieuwkerk

Re: O_DIRECT

[bert hubert]

On Tue, Jul 20, 2004 at 12:17:31PM -0700, Shesha Sreenivasamurthy wrote:
> Thank you very much Hubert.
> 
> Regarding my second question, what I mean is, to calculate the soft 
> block size of a partition on which there is no filesystem created, I am 
> using the IOCTL "BLKBSZGET".

Sure - but what do you need this blocksize for?

-- 
http://www.PowerDNS.com      Open source, database driven DNS Software 
http://lartc.org           Linux Advanced Routing & Traffic Control HOWTO

Re: O_DIRECT

[bert hubert]

On Tue, Jul 20, 2004 at 10:27:57AM -0700, Shesha Sreenivasamurthy wrote:
> I am having trouble with O_DIRECT. Trying to read or write from a block 
> device partition.
> 
> 1. Can O_DIRECT be used on a plain block device partition say 
> "/dev/sda11" without having a filesystem on it.

As fas as I know, yes, but be aware that O_DIRECT requires page aligned
addresses! (an integral of 4096 on most systems).

> 2. If no file system is created then what should be the softblock size. 
> I am using the IOCTL "BLKBSZGET". Is this correct?

No idea what you mean - but see above about the aligned addresses.

> 3. Can we use SEEK_END with O_DIRECT on a partition without filesystem.

I see no reason why not. Good luck!

-- 
http://www.PowerDNS.com      Open source, database driven DNS Software 
http://lartc.org           Linux Advanced Routing & Traffic Control HOWTO

O_DIRECT

[Shesha Sreenivasamurthy]

I am having trouble with O_DIRECT. Trying to read or write from a block 
device partition.

1. Can O_DIRECT be used on a plain block device partition say 
"/dev/sda11" without having a filesystem on it.

2. If no file system is created then what should be the softblock size. 
I am using the IOCTL "BLKBSZGET". Is this correct?

3. Can we use SEEK_END with O_DIRECT on a partition without filesystem.

Any help will be highly regarded.

-Shesha

Re: O_DIRECT

[Peter T. Breuer]

"A month of sundays ago kernel@street-vision.com wrote:"
> > 
> > What functions does a block driver have to implement in order to
> > support read/write when it has been opened with O_DIRECT from user
> > space.

> Remeber that the memory you read/write from must be page aligned (ie
> mmap /dev/zero not malloc) and reads and writes must be multiples of the
> page size. I think block devices work on 2.4 too, but I forget (otherwise
> you can use raw devices).

I do believe you are right. Multiples of 4096 seem to work fine. No
support needed in the block device driver.

Peter

Re: O_DIRECT

[Peter T. Breuer]

"A month of sundays ago kernel@street-vision.com wrote:"
> > What functions does a block driver have to implement in order to
> > support read/write when it has been opened with O_DIRECT from user
> > space.
> > 
> > I have made some experiments with plain read/write after opening with 
> > O_DIRECT:
> > 
> > 2.5.31:
> >    /dev/ram0      open fails
> >    file on ext2   r/w gives EINVAL
> >    /dev/hdaN      works
> > 
> > 2.4.19:
> >    /dev/ram0      r/w gives EINVAL
> >    file on ext2   r/w gives EINVAL
> >    /dev/hdaN      r/w gives EINVAL
> > 
> > WTF? It's not a library issue - strace shows the syscalls happening
> > with the right flag set on the open.
> 
> You should be able to get it to work on ext2. It works fine for me.
> Remeber that the memory you read/write from must be page aligned (ie
> mmap /dev/zero not malloc) and reads and writes must be multiples of the
> page size. I think block devices work on 2.4 too, but I forget (otherwise
> you can use raw devices).

Thanks for the input. Well, I used the same test program on all, and
the buffer was aligned at 512 bytes (because I intended it to work
with raw character devices too). You are saying that it was luck .. OK,
I'll retest in a little while.

So I simply have to do "nothing" in the driver?

Peter

O_DIRECT

[Peter T. Breuer]

What functions does a block driver have to implement in order to
support read/write when it has been opened with O_DIRECT from user
space.

I have made some experiments with plain read/write after opening with 
O_DIRECT:

2.5.31:
   /dev/ram0      open fails
   file on ext2   r/w gives EINVAL
   /dev/hdaN      works

2.4.19:
   /dev/ram0      r/w gives EINVAL
   file on ext2   r/w gives EINVAL
   /dev/hdaN      r/w gives EINVAL

WTF? It's not a library issue - strace shows the syscalls happening
with the right flag set on the open.

Can someone put me out of my misery? Where the heck is this implemented
in the 2.5.31 ide code? If there? There's no mention of direct_IO.
Clues?

What I ultimately want is to know what code I have to put into a block
device driver in order to support O_DIRECT on the device. 

Peter

O_DIRECT

[Andrea Arcangeli]

[-- Attachment #1: Type: text/plain, Size: 8437 bytes --]

I wrote the O_DIRECT zerocopy raw I/O support (dma from disk to the userspace
memory through the filesystem). The patch against 2.4.4pre2 + rawio-3 is here:

	ftp://ftp.us.kernel.org/pub/linux/kernel/people/andrea/patches/v2.4/2.4.4pre2/o_direct-1

Only ext2 is supported at the moment, but extending it to the other fses that
use the common code helper functions is trivial (I guess Chris will take care
of reiserfs, it may be an option to not do tail packing for files opened with
O_DIRECT so you can dma from userspace the tail as well).

The above patch depends on the rawio performance improvement patch posted to
the list a few days ago here (latest version here):

	ftp://ftp.us.kernel.org/pub/linux/kernel/people/andrea/patches/v2.4/2.4.4pre2/rawio-3

The rawio-3 patch is suggested for integration, certainly it's saner and faster
than mainline 2.4 (a note of credit: part of the fixes in the rawio-3 patch
are merged from SCT's patch).

To benchmark the improvement given by O_DIRECT, I hacked bonnie to open the
file with O_DIRECT in the "block" tests and I changed the chunk size to 1MB (so
that the blkdev layer will send large requests the hardware). Then I made a
comparision between the bonnie numbers w/o and w/ O_DIRECT (the -o_direct param
to bonnie now selects the O_DIRECT or standard behaviour, I also added
a -fast param to skip the slow seek test).  I cutted out the numbers that
aren't using O_DIRECT to make the report more readable.

Those are still preliminary results on a mid machine: 2-way SMP PII 450mhz with
128mbyte of ram on a lvm volume (physical volume is a single IDE disk, so not
striped and all in the same harddisk) using a working set of 400mbytes.

without using o_direct:
              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
           MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
          400  xxxx xxxx 12999 12.1  5918 10.8  xxxx xxxx 13412 12.1   xxx  xxx
          400  xxxx xxxx 12960 12.3  5896 11.1  xxxx xxxx 13520 13.3   xxx  xxx

with o_direct:
              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
           MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
          400  xxxx xxxx 12810  1.8  5855  1.6  xxxx xxxx 13529  1.2   xxx  xxx
          400  xxxx xxxx 12814  1.8  5866  1.7  xxxx xxxx 13519  1.3   xxx  xxx

As you can see there's a small performance drop in writes and I guess it's
beause we are some more synchronous and it takes a bit more of time between the
completion of one I/O request and the the sumbission of a new one and that's ok.
(if you really care that litte difference can be covered by using async-io)

The most interesting part is the cpu load that just decreases down to 1/2%
during all the I/O and this is ""only"" a 13mbyte/sec harddisk and on
top of lvm. I didn't run any benchmark on any faster disk and without lvm but
O_DIRECT is the obvious way to go for streaming endless data to disk (possibly
at 100mbyte/sec or more) like in multimedia or scientific apps and of course
with DBMS that do their own userspace management of the I/O cache in shm.

>From a DBMS point of view the only downside of the O_DIRECT compared to the
rawio device is: 1) walking of the meatadata in the fs [but that is in turn
_the_ feature that gives more flexibility to the administrator] and 2) O_DIRECT
cannot be done on a shared disk without also using a filesystem like GFS
because the regular fs doesn't know how to keep the metadata coherent across
multiple hosts.

Programming-wise the coherency of the cache under the direct I/O is the only
non obvious issue, what I did is been simply to flush the data (nothing of the
metadata!) before starting the direct I/O and to discard all the unmapped
pagecache from the inode after any direct write or to invalidate them (clear
all dirty bits in the pagecache and on the overlapped buffers if the page was
mapped so the next read through the cache will hit the disk again). This seems
safe even if it somehow breaks the semantics of mmaps (for example if the file
is mmaped during the o_direct the mmap view won't be updated, but o_direct
after all is magic anyways in the way it requires alignment of address and size
of the user buffer so this doesn't seem to be a problem... and not keeping
perfect coherency of the not useful cases increases performance of the useful
cases. I didn't see a value in updating the cache, if you want to update the
cache in a mapping simply don't use O_DIRECT ;).

To invalidate the cache I couldn't use the invalidate_inode_pages and
truncate_inode_pages, the former was too weak (I cannot simply skip the
stuff that is not just clean and unmapped or the user could never see
the updates afterwards when he stops using O_DIRECT), and the latter was too
aggressive (it can be used only once the inode is getting released and we know
there are no active mmaps on the file, the set_page_dirty was oopsing on me
because the page->i_mapping become null in a shared mapping for example). So I
wrote a third invalidate_inode_pages2 that also seems attractive from a NFS
point of view though I didn't changed nfs to use it, if we change the VM to be
robust about mapped pages getting removed by the pagecache I could use
truncate_inode_pages() in the future.

One important detail is that the invalidate_inode_pages2 doesn't get
a "range" of addresses in the address_space but it invalidates the whole
address_space instead. At first I was invalidating only the modified range and
after the putc test (that loads the cache) the rewrite tests was overloading
the cpu (around 30% of cpu usage). After I started to cut all the address space
the cpu load during rewrite test returned to the early numbers without cache
coherency in O_DIRECT.  (I was also browsing the list in the less optimal order
but I preferred to drop the range option to not force people to always
read physically consecutive to avoid the quadratic behaviour :)

As mentioned above if somebody keeps mmaps on the file the
invalidate_inode_pages2 won't be able to drop the pages from the list and it
will waste some CPU in kernel space. Only thing I did is to put the reschedule
points in the right places so the machine will remain perfectly responsive if
somebody tries to exploit it.

Some other explanation on the i_dirty_data_buffers list (probably you are
wondering why I added it): in the early stage I didn't flush the cache before
starting the direct I/O and I was getting performance like now in the final
patch. When the thing started working and I started to care about having
something that works with mixed O_DIRECT and non direct I/O I added a
generic_osync_inode (ala generic_file_write) to flush the cache to disk before
starting the direct I/O. This degraded significantly the performance because
the generic_osync_inode goes down and writes synchronously all the metadata as
well. More precisely I was getting those numbers:

              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
           MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
          400  xxxx xxxx 11138  1.7  5880  3.2  xxxx xxxx 13519  1.7  xxxx xxxx
			 ^^^^^		   ^^^
(I am writing only one line but the numbers were very stable and it was really a
noticeable slowdown)

To resurrect the performance of the original patch so I had to split the inode buffer
list in two, one for the data and one for the metadata, and now I only flush and
wait synchronously I/O completion of the i_dirty_data_buffers list before
starting the direct I/O and this fixed the performance problem completly.

If O_SYNC is used in combination of O_DIRECT I in turn skip the i_dirty_data_buffers
list and I only flush the metadata (i_dirty_buffers list) and the inode if
there was metadata dirty (this is why I introduced a third case in the
generic_osync_inode, this in practice probably doesn't make any difference though).

Another change I did in the brw_kiovec is that in the block[] array the
blocknumber -1UL is reserved and it's used during reads from disk to indicate
that the destination memory should be cleared (it's used to handle holes in the
files).

Andrea

[-- Attachment #2: bonnie-o_direct-1 --]
[-- Type: text/plain, Size: 6200 bytes --]

--- bonnie-1.orig/bonnie.c	Tue Sep  3 00:59:11 1996
+++ bonnie/bonnie-o_direct.c	Thu Apr 12 20:26:14 2001
@@ -95,6 +95,7 @@
 #else
 #include <sys/resource.h>
 #endif
+#include <asm/page.h>
 
 #define IntSize (4)
 
@@ -109,7 +110,11 @@
 #define Seeks (4000)
 #define UpdateSeek (10)
 #define SeekProcCount (3)
-#define Chunk (8192)
+#define Chunk (1<<20)
+
+#if defined(__alpha__) || defined(__i386__)
+#define O_DIRECT	040000	/* direct disk access - should check with OSF/1 */
+#endif
 
 static double cpu_so_far();
 static void   doseek();
@@ -146,7 +151,8 @@
   int    argc;
   char * argv[];
 {
-  int    buf[Chunk / IntSize];
+  char   __buf[Chunk+~PAGE_MASK];
+  int    *buf = (int *) ((unsigned long) (__buf + ~PAGE_MASK) & PAGE_MASK);
   int    bufindex;
   int    chars[256];
   int    child;
@@ -164,24 +170,28 @@
   int    size;
   FILE * stream;
   int    words;
+  int    o_direct = 0, fast = 0;
 
   fd = -1;
   basetime = (int) time((time_t *) NULL);
   size = 100;
   dir = ".";
 
-  for (next = 1; next < argc - 1; next++)
+   for (next = 1; next < argc; next++)
     if (argv[next][0] == '-')
     { /* option? */
       if (strcmp(argv[next] + 1, "d") == 0)
-        dir = argv[next + 1];
+	      dir = argv[next + 1], next++;
       else if (strcmp(argv[next] + 1, "s") == 0)
-        size = atoi(argv[next + 1]);
+	      size = atoi(argv[next + 1]), next++;
       else if (strcmp(argv[next] + 1, "m") == 0)
-        machine = argv[next + 1];
+	      machine = argv[next + 1], next++;
+      else if (strcmp(argv[next] + 1, "o_direct") == 0)
+	o_direct = 1, printf("Using O_DIRECT for block based I/O\n");
+      else if (strcmp(argv[next] + 1, "fast") == 0)
+	fast = 1, printf("Will skip the seek test\n");
       else
         usage();
-      next++;
     } /* option? */
     else
       usage();
@@ -194,12 +204,13 @@
 
   /* Fill up a file, writing it a char at a time with the stdio putc() call */
   fprintf(stderr, "Writing with putc()...");
-  newfile(name, &fd, &stream, 1);
+  newfile(name, &fd, &stream, 1, 0);
   timestamp();
   for (words = 0; words < size; words++)
     if (putc(words & 0x7f, stream) == EOF)
       io_error("putc");
   
+  fsync(fd);
   /*
    * note that we always close the file before measuring time, in an
    *  effort to force as much of the I/O out as we can
@@ -210,13 +221,13 @@
   fprintf(stderr, "done\n");
 
   /* Now read & rewrite it using block I/O.  Dirty one word in each block */
-  newfile(name, &fd, &stream, 0);
+  newfile(name, &fd, &stream, 0, o_direct);
   if (lseek(fd, (off_t) 0, 0) == (off_t) -1)
     io_error("lseek(2) before rewrite");
   fprintf(stderr, "Rewriting...");
   timestamp();
   bufindex = 0;
-  if ((words = read(fd, (char *) buf, Chunk)) == -1)
+  if ((words = read(fd, buf, Chunk)) == -1)
     io_error("rewrite read");
   while (words == Chunk)
   { /* while we can read a block */
@@ -230,13 +241,14 @@
     if ((words = read(fd, (char *) buf, Chunk)) == -1)
       io_error("rwrite read");
   } /* while we can read a block */
+  fsync(fd);
   if (close(fd) == -1)
     io_error("close after rewrite");
   get_delta_t(ReWrite);
   fprintf(stderr, "done\n");
 
   /* Write the whole file from scratch, again, with block I/O */
-  newfile(name, &fd, &stream, 1);
+  newfile(name, &fd, &stream, 1, o_direct);
   fprintf(stderr, "Writing intelligently...");
   for (words = 0; words < Chunk / IntSize; words++)
     buf[words] = 0;
@@ -249,13 +261,14 @@
     if (write(fd, (char *) buf, Chunk) == -1)
       io_error("write(2)");
   } /* for each word */
+  fsync(fd);
   if (close(fd) == -1)
     io_error("close after fast write");
   get_delta_t(FastWrite);
   fprintf(stderr, "done\n");
 
   /* read them all back with getc() */
-  newfile(name, &fd, &stream, 0);
+  newfile(name, &fd, &stream, 0, 0);
   for (words = 0; words < 256; words++)
     chars[words] = 0;
   fprintf(stderr, "Reading with getc()...");
@@ -278,7 +291,7 @@
     sprintf((char *) buf, "%d", chars[words]);
 
   /* Now suck it in, Chunk at a time, as fast as we can */
-  newfile(name, &fd, &stream, 0);
+  newfile(name, &fd, &stream, 0, o_direct);
   if (lseek(fd, (off_t) 0, 0) == -1)
     io_error("lseek before read");
   fprintf(stderr, "Reading intelligently...");
@@ -295,6 +308,11 @@
   get_delta_t(FastRead);
   fprintf(stderr, "done\n");
 
+  if (fast) {
+	  delta[Lseek][Elapsed] = 1;
+	  delta[Lseek][CPU] = 0;
+	  goto finish;
+  }
   /* use the frequency count */
   for (words = 0; words < 256; words++)
     sprintf((char *) buf, "%d", chars[words]);
@@ -334,8 +352,8 @@
       /* set up and wait for the go-ahead */
       close(seek_feedback[0]);
       close(seek_control[1]);
-      newfile(name, &fd, &stream, 0);
-      srandom(getpid());
+      newfile(name, &fd, &stream, 0, 0);
+      srandom(next); /* must always generate the same sequence or you'll compare orange to apples */
       fprintf(stderr, "Seeker %d...", next + 1);
 
       /* wait for the go-ahead */
@@ -411,6 +429,7 @@
   fprintf(stderr, "\n");
   delta[(int) Lseek][Elapsed] = last_stop - first_start;
 
+ finish:
   report(size);
   unlink(name);
 }
@@ -448,20 +467,21 @@
 }
 
 static void
-newfile(name, fd, stream, create)
+newfile(name, fd, stream, create, o_direct)
   char *   name;
   int *    fd;
   FILE * * stream;
   int      create;
+  int      o_direct;
 {
   if (create)
   { /* create from scratch */
     if (unlink(name) == -1 && *fd != -1)
       io_error("unlink");
-    *fd = open(name, O_RDWR | O_CREAT | O_EXCL, 0777);
+    *fd = open(name, O_RDWR | O_CREAT | O_EXCL | (o_direct ? O_DIRECT : 0), 0777);
   } /* create from scratch */
   else
-    *fd = open(name, O_RDWR, 0777);
+    *fd = open(name, O_RDWR | (o_direct ? O_DIRECT : 0), 0777);
 
   if (*fd == -1)
     io_error(name);
@@ -474,7 +494,7 @@
 usage()
 {
   fprintf(stderr,
-    "usage: Bonnie [-d scratch-dir] [-s size-in-Mb] [-m machine-label]\n");
+    "usage: Bonnie [-d scratch-dir] [-s size-in-Mb] [-m machine-label] [-o_direct]\n");
   exit(1);
 }
 
@@ -544,8 +564,7 @@
 io_error(message)
   char * message;
 {
-  char buf[Chunk];
-
+  char buf[8192];
   sprintf(buf, "Bonnie: drastic I/O error (%s)", message);
   perror(buf);
   exit(1);