How can I force a read to hit the disk?

How can I force a read to hit the disk?

[Alan Stern]

My kernel module for Linux-2.6 needs to be able to verify that the media 
on which a file resides actually is readable.  How can I do that?

It would certainly suffice to use the normal VFS read routines, if there
was some way to force the system to actually read from the device rather
than just returning data already in the cache.  So I guess it would be 
enough to perform an fdatasync for the file and then invalidate the file's 
cache entries.  How does one invalidate a file's cache entries?  Does 
filemap_flush() perform both these operations for you?

TIA,

Alan Stern

Re: How can I force a read to hit the disk?

[Andreas Dilger]

On Sep 05, 2003  13:46 -0400, Alan Stern wrote:
> My kernel module for Linux-2.6 needs to be able to verify that the media 
> on which a file resides actually is readable.  How can I do that?
> 
> It would certainly suffice to use the normal VFS read routines, if there
> was some way to force the system to actually read from the device rather
> than just returning data already in the cache.  So I guess it would be 
> enough to perform an fdatasync for the file and then invalidate the file's 
> cache entries.  How does one invalidate a file's cache entries?  Does 
> filemap_flush() perform both these operations for you?

If you open the file with O_DIRECT, it should read/write directly on the
disk, and it will also invalidate any existing cache for the read/written
area.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: How can I force a read to hit the disk?

[Richard B. Johnson]

On Fri, 5 Sep 2003, Alan Stern wrote:

> My kernel module for Linux-2.6 needs to be able to verify that the media
> on which a file resides actually is readable.  How can I do that?
>
> It would certainly suffice to use the normal VFS read routines, if there
> was some way to force the system to actually read from the device rather
> than just returning data already in the cache.  So I guess it would be
> enough to perform an fdatasync for the file and then invalidate the file's
> cache entries.  How does one invalidate a file's cache entries?  Does
> filemap_flush() perform both these operations for you?
>
> TIA,
>
> Alan Stern

Force a read() before some write data was flushed to the disk??
Well, if you insist, just do a raw read of the device after
you find the inode, then offset, that your data is on. You
can walk the same fs structure(s) as the active file-system
to get all meta-data information.

...but... The most current data is probably in the kernel
buffers. I don't think you really want what you are asking
for. If you did a fdatasync(), you get the current data your
process wrote to go to the disk. However, there may be
other writers. This, too, may not be what you want. Also,
Just because the data was queued to go to a (SCSI) disk,
doesn't mean it actually got to the platters.

Cheers,
Dick Johnson
Penguin : Linux version 2.4.22 on an i686 machine (794.73 BogoMips).
            Note 96.31% of all statistics are fiction.

Re: How can I force a read to hit the disk?

[Alan Stern]

On Fri, 5 Sep 2003, Andreas Dilger wrote:

> On Sep 05, 2003  13:46 -0400, Alan Stern wrote:
> > My kernel module for Linux-2.6 needs to be able to verify that the media 
> > on which a file resides actually is readable.  How can I do that?
> > 
> > It would certainly suffice to use the normal VFS read routines, if there
> > was some way to force the system to actually read from the device rather
> > than just returning data already in the cache.  So I guess it would be 
> > enough to perform an fdatasync for the file and then invalidate the file's 
> > cache entries.  How does one invalidate a file's cache entries?  Does 
> > filemap_flush() perform both these operations for you?
> 
> If you open the file with O_DIRECT, it should read/write directly on the
> disk, and it will also invalidate any existing cache for the read/written
> area.

Unfortunately that's not a good solution for me.  The file has already 
been opened without O_DIRECT, and O_DIRECT wouldn't be appropriate because 
most of the time I do want I/O to go through the cache.  It's just on a 
few rare occasions that I need direct access to the disk.

Maybe simply opening a new struct file using O_DIRECT, for purposes of 
the verification, while keeping the old struct file around for other uses 
later, will work?  That would be awkward though -- and there's no 
guarantee that the original filename would still exist.  It would be a lot 
nicer to do everything using the original file reference.

Alan Stern

Re: How can I force a read to hit the disk?

[Jörn Engel]

On Fri, 5 September 2003 14:49:15 -0400, Alan Stern wrote:
> On Fri, 5 Sep 2003, Andreas Dilger wrote:
> 
> > If you open the file with O_DIRECT, it should read/write directly on the
> > disk, and it will also invalidate any existing cache for the read/written
> > area.
> 
> Unfortunately that's not a good solution for me.  The file has already 
> been opened without O_DIRECT, and O_DIRECT wouldn't be appropriate because 
> most of the time I do want I/O to go through the cache.  It's just on a 
> few rare occasions that I need direct access to the disk.
> 
> Maybe simply opening a new struct file using O_DIRECT, for purposes of 
> the verification, while keeping the old struct file around for other uses 
> later, will work?  That would be awkward though -- and there's no 
> guarantee that the original filename would still exist.  It would be a lot 
> nicer to do everything using the original file reference.

Maybe these help you here:
man 3 fdopen
man 3 fileno

No filename needed for the second open.

Jörn

-- 
"Error protection by error detection and correction."
-- from a university class

Re: How can I force a read to hit the disk?

[Alan Stern]

On Fri, 5 Sep 2003, Richard B. Johnson wrote:

> On Fri, 5 Sep 2003, Alan Stern wrote:
> 
> > My kernel module for Linux-2.6 needs to be able to verify that the media
> > on which a file resides actually is readable.  How can I do that?
> >
> > It would certainly suffice to use the normal VFS read routines, if there
> > was some way to force the system to actually read from the device rather
> > than just returning data already in the cache.  So I guess it would be
> > enough to perform an fdatasync for the file and then invalidate the file's
> > cache entries.  How does one invalidate a file's cache entries?  Does
> > filemap_flush() perform both these operations for you?
> >
> > TIA,
> >
> > Alan Stern
> 
> Force a read() before some write data was flushed to the disk??

No, no!  Force a read() _after_ all the dirty buffers are flushed to the 
disk.

> Well, if you insist, just do a raw read of the device after
> you find the inode, then offset, that your data is on. You
> can walk the same fs structure(s) as the active file-system
> to get all meta-data information.

I assume you're kidding... :-)

> ...but... The most current data is probably in the kernel
> buffers. I don't think you really want what you are asking
> for.

No, I _do_ want exactly what I asked for.  Actually, I would really like a 
bit more: to ask the disk drive to read from its media rather than its 
cache.  But I'll settle for bypassing the O/S's cache.

>  If you did a fdatasync(), you get the current data your
> process wrote to go to the disk.

Not if there is a bad sector that causes a read error.

>  However, there may be
> other writers. This, too, may not be what you want.

I'm assuming there aren't other writers.  It doesn't really matter if 
there are.  I don't care _what_ data is on the disk; I just want to know 
that it is _readable_ without getting hardware errors.

>  Also,
> Just because the data was queued to go to a (SCSI) disk,
> doesn't mean it actually got to the platters.

Yes.  It's an imperfect world.

Alan Stern

Re: How can I force a read to hit the disk?

[Alan Stern]

On Fri, 5 Sep 2003, Jörn Engel wrote:

> On Fri, 5 September 2003 14:49:15 -0400, Alan Stern wrote:
> > On Fri, 5 Sep 2003, Andreas Dilger wrote:
> > 
> > > If you open the file with O_DIRECT, it should read/write directly on the
> > > disk, and it will also invalidate any existing cache for the read/written
> > > area.
> > 
> > Unfortunately that's not a good solution for me.  The file has already 
> > been opened without O_DIRECT, and O_DIRECT wouldn't be appropriate because 
> > most of the time I do want I/O to go through the cache.  It's just on a 
> > few rare occasions that I need direct access to the disk.
> > 
> > Maybe simply opening a new struct file using O_DIRECT, for purposes of 
> > the verification, while keeping the old struct file around for other uses 
> > later, will work?  That would be awkward though -- and there's no 
> > guarantee that the original filename would still exist.  It would be a lot 
> > nicer to do everything using the original file reference.
> 
> Maybe these help you here:
> man 3 fdopen
> man 3 fileno
> 
> No filename needed for the second open.

I don't think they will help.  Apart from the fact that I'm working on a 
kernel module, not a user program, neither of these accepts options like 
O_DIRECT.

Alan Stern

Re: How can I force a read to hit the disk?

[Alan Stern]

On Fri, 5 Sep 2003, Andreas Dilger wrote:

> On Sep 05, 2003  13:46 -0400, Alan Stern wrote:
> > My kernel module for Linux-2.6 needs to be able to verify that the media 
> > on which a file resides actually is readable.  How can I do that?
> > 
> > It would certainly suffice to use the normal VFS read routines, if there
> > was some way to force the system to actually read from the device rather
> > than just returning data already in the cache.  So I guess it would be 
> > enough to perform an fdatasync for the file and then invalidate the file's 
> > cache entries.  How does one invalidate a file's cache entries?  Does 
> > filemap_flush() perform both these operations for you?
> 
> If you open the file with O_DIRECT, it should read/write directly on the
> disk, and it will also invalidate any existing cache for the read/written
> area.

If I set the O_DIRECT bit in filp->flags just for the duration of a call
to vfs_read(), will that accomplish what I want?

Alan Stern

Re: How can I force a read to hit the disk?

[Alan Stern]

On Fri, 5 Sep 2003, Andreas Dilger wrote:

> On Sep 05, 2003  13:46 -0400, Alan Stern wrote:
> > My kernel module for Linux-2.6 needs to be able to verify that the media 
> > on which a file resides actually is readable.  How can I do that?
> > 
> > It would certainly suffice to use the normal VFS read routines, if there
> > was some way to force the system to actually read from the device rather
> > than just returning data already in the cache.  So I guess it would be 
> > enough to perform an fdatasync for the file and then invalidate the file's 
> > cache entries.  How does one invalidate a file's cache entries?  Does 
> > filemap_flush() perform both these operations for you?
> 
> If you open the file with O_DIRECT, it should read/write directly on the
> disk, and it will also invalidate any existing cache for the read/written
> area.

I tried doing that, but it caused a segmentation violation.  The trouble 
was this line near the start of fs/direct_io.c:dio_refill_pages()

	down_read(&current->mm->mmap_sem);

Unfortunately, in a kernel thread (which is where my code runs)  
current->mm is NULL.

Can anybody offer additional advice?  How about a way to invalidate all 
the page cache entries that contain a page from the file?

Alan Stern

Re: How can I force a read to hit the disk?

[Richard B. Johnson]

On Mon, 8 Sep 2003, Alan Stern wrote:

> On Fri, 5 Sep 2003, Andreas Dilger wrote:
>
> > On Sep 05, 2003  13:46 -0400, Alan Stern wrote:
> > > My kernel module for Linux-2.6 needs to be able to verify that the media
> > > on which a file resides actually is readable.  How can I do that?
> > >
> > > It would certainly suffice to use the normal VFS read routines, if there
> > > was some way to force the system to actually read from the device rather
> > > than just returning data already in the cache.  So I guess it would be
> > > enough to perform an fdatasync for the file and then invalidate the file's
> > > cache entries.  How does one invalidate a file's cache entries?  Does
> > > filemap_flush() perform both these operations for you?
> >
> > If you open the file with O_DIRECT, it should read/write directly on the
> > disk, and it will also invalidate any existing cache for the read/written
> > area.
>
> I tried doing that, but it caused a segmentation violation.  The trouble
> was this line near the start of fs/direct_io.c:dio_refill_pages()
>
> 	down_read(&current->mm->mmap_sem);
>
> Unfortunately, in a kernel thread (which is where my code runs)
> current->mm is NULL.
>
> Can anybody offer additional advice?  How about a way to invalidate all
> the page cache entries that contain a page from the file?
>
> Alan Stern

When your thread code starts up, execute
                init_rwsem(&current->mm->mmap_sem);

... This is in the thread's code, not the module init code.

Cheers,
Dick Johnson
Penguin : Linux version 2.4.22 on an i686 machine (794.73 BogoMips).
            Note 96.31% of all statistics are fiction.

Re: How can I force a read to hit the disk?

[Alan Stern]

On Mon, 8 Sep 2003, Richard B. Johnson wrote:

> On Mon, 8 Sep 2003, Alan Stern wrote:
> 
> > I tried doing that, but it caused a segmentation violation.  The trouble
> > was this line near the start of fs/direct_io.c:dio_refill_pages()
> >
> > 	down_read(&current->mm->mmap_sem);
> >
> > Unfortunately, in a kernel thread (which is where my code runs)
> > current->mm is NULL.
> >
> > Can anybody offer additional advice?  How about a way to invalidate all
> > the page cache entries that contain a page from the file?
> >
> > Alan Stern
> 
> When your thread code starts up, execute
>                 init_rwsem(&current->mm->mmap_sem);
> 
> ... This is in the thread's code, not the module init code.

That doesn't work either; it also causes a segmentation violation.  As I 
said before, current->mm is NULL.  It gets set that way by exit_mm() which 
is called from daemonize().

Alan Stern

Re: How can I force a read to hit the disk?

[Richard B. Johnson]

On Mon, 8 Sep 2003, Alan Stern wrote:

> On Mon, 8 Sep 2003, Richard B. Johnson wrote:
>
> > On Mon, 8 Sep 2003, Alan Stern wrote:
> >
> > > I tried doing that, but it caused a segmentation violation.  The trouble
> > > was this line near the start of fs/direct_io.c:dio_refill_pages()
> > >
> > > 	down_read(&current->mm->mmap_sem);
> > >
> > > Unfortunately, in a kernel thread (which is where my code runs)
> > > current->mm is NULL.
> > >
> > > Can anybody offer additional advice?  How about a way to invalidate all
> > > the page cache entries that contain a page from the file?
> > >
> > > Alan Stern
> >
> > When your thread code starts up, execute
> >                 init_rwsem(&current->mm->mmap_sem);
> >
> > ... This is in the thread's code, not the module init code.
>
> That doesn't work either; it also causes a segmentation violation.  As I
> said before, current->mm is NULL.  It gets set that way by exit_mm() which
> is called from daemonize().
>
> Alan Stern
>

Gawd. I assumed you knew how to initialize a pointer. I just
located the procedure used to initialize the semaphore.


Cheers,
Dick Johnson
Penguin : Linux version 2.4.22 on an i686 machine (794.73 BogoMips).
            Note 96.31% of all statistics are fiction.

Re: How can I force a read to hit the disk?

[Alan Stern]

On Tue, 9 Sep 2003, Richard B. Johnson wrote:

> On Mon, 8 Sep 2003, Alan Stern wrote:
> 
> > > When your thread code starts up, execute
> > >                 init_rwsem(&current->mm->mmap_sem);
> > >
> > > ... This is in the thread's code, not the module init code.
> >
> > That doesn't work either; it also causes a segmentation violation.  As I
> > said before, current->mm is NULL.  It gets set that way by exit_mm() which
> > is called from daemonize().
> >
> > Alan Stern
> >
> 
> Gawd. I assumed you knew how to initialize a pointer. I just
> located the procedure used to initialize the semaphore.

Shucks, I know how to initialize a pointer.  I even know how to initialize
a read-write semaphore.  The problem here is not _doing_ the
initialization; it's what _value_ to use.  Kernel threads don't have a
userpace memory component, so naturally current->mm is NULL -- there's no
memory map for it to point to.  Without having a memory map, of course
there's no semaphore to initialize, since the semaphore is _part_ of the 
memory map structure.

Alan Stern

Re: How can I force a read to hit the disk?

[Richard B. Johnson]

On Tue, 9 Sep 2003, Alan Stern wrote:

> On Tue, 9 Sep 2003, Richard B. Johnson wrote:
>
> > On Mon, 8 Sep 2003, Alan Stern wrote:
> >
> > > > When your thread code starts up, execute
> > > >                 init_rwsem(&current->mm->mmap_sem);
> > > >
> > > > ... This is in the thread's code, not the module init code.
> > >
> > > That doesn't work either; it also causes a segmentation violation.  As I
> > > said before, current->mm is NULL.  It gets set that way by exit_mm() which
> > > is called from daemonize().
> > >
> > > Alan Stern
> > >
> >
> > Gawd. I assumed you knew how to initialize a pointer. I just
> > located the procedure used to initialize the semaphore.
>
> Shucks, I know how to initialize a pointer.  I even know how to initialize
> a read-write semaphore.  The problem here is not _doing_ the
> initialization; it's what _value_ to use.  Kernel threads don't have a
> userpace memory component, so naturally current->mm is NULL -- there's no
> memory map for it to point to.  Without having a memory map, of course
> there's no semaphore to initialize, since the semaphore is _part_ of the
> memory map structure.
>
> Alan Stern
>

The semaphore location exists in the memory map structure, mm_struct.
You only need to allocate some memory the size of that structure to
contain the semaphore. You can readily see what happens in
../kernel/fork.c ...:

    Something like:

          if((current->mm = mm_alloc()) == NULL)
              printk("Some problem with mm_alloc()\n");
          else
              init_rwsem(&current->mm->mmap_sem);

   ... should get you going.

    If, in all your code, the only problem you had was an allocation
problem with your semaphore, and not a problem with actual memory
access, then you are golden. If you are even going to shut down your
kernel thread, you probably need to add your allocation to the
linked list. If not, don't bother.

Cheers,
Dick Johnson
Penguin : Linux version 2.4.22 on an i686 machine (794.73 BogoMips).
            Note 96.31% of all statistics are fiction.

Re: How can I force a read to hit the disk?

[Alan Stern]

On Tue, 9 Sep 2003, Richard B. Johnson wrote:

> The semaphore location exists in the memory map structure, mm_struct.
> You only need to allocate some memory the size of that structure to
> contain the semaphore. You can readily see what happens in
> ../kernel/fork.c ...:
> 
>     Something like:
> 
>           if((current->mm = mm_alloc()) == NULL)
>               printk("Some problem with mm_alloc()\n");
>           else
>               init_rwsem(&current->mm->mmap_sem);
> 
>    ... should get you going.
> 
>     If, in all your code, the only problem you had was an allocation
> problem with your semaphore, and not a problem with actual memory
> access, then you are golden.

I'm afraid it's not as simple as that.  Yes, my driver is working just 
fine, apart from these conceptual difficulties in getting O_DIRECT to 
perform.  (Without the O_DIRECT, it works perfectly -- other than not 
bypassing the page cache.)

I did what you suggested: current->mm = mm_alloc().  (I had to change
kernel/fork.c since mm_alloc() isn't EXPORTed.)  The call to init_rwsem()  
isn't needed because mm_alloc() calls mm_init(), which does it.  That much
succeeded.  But what about setting current->active_mm?  What about undoing 
the effect of enter_lazy_tlb(), which is called indirectly by daemonize()?  
Or does it not need to be undone?

In the end, your suggestion didn't work.  Subsequent calls to vfs_read()
returned -EFAULT, which obviously isn't useful.  I know very little about
the intimate details of Linux's memory management; is it possible that the
EFAULT occurred because my I/O buffer is in kernel space rather than user
space?  My kernel thread does execute set_fs(get_ds()) before doing much
else.  But the direct_io path seems to do some involved memory-address
calculations; is it only meant to work with user-space buffers?

>  If you are even going to shut down your
> kernel thread, you probably need to add your allocation to the
> linked list. If not, don't bother.

Yes, my thread does need to exit.  To what linked list do you refer -- one
based on init_mm?  Is the mm->mmlist member of my structure the part that
should be linked in, as dup_mmap() does?  What about all the other stuff
dup_mmap() does, that my driver doesn't do?

Simply exiting without doing anything else was disastrous.  The system got
caught in an uninterruptible error loop.

Alan Stern