Re: block device/VM question

Re: block device/VM question

[Peter T. Breuer]

"A month of sundays ago Thunder from the hill wrote:"
> On Tue, 27 Aug 2002, Peter T. Breuer wrote:
> > You mean do a syscall with flags O_DIRECT from within the driver?
> > I don't like syscalsl from kernel code, but if you say so, I'll think
> > about it.
> 
> Not exactly, but it's a starting point. See how.

I think you're directing my attention to how to get to the inode of the
special device file. That's OK. I can get to that with no hassle. Once
there I can do whatever the sysopen with O_DIRECT does.

> <URL:ftp://ftp.es.kernel.org/pub/linux/docs/manpages/man-pages-1.53.tar.bz2>
> 
> Contains lots of the useful stuff.

I'm sure :-). But until it gets packaged for debian I don't touch :-/.

> > suggesting that it depends on the mode of access to the device? That I
> > can't control - I simply want ALL accesses to not be cached.
> 
> That's a word, and confirms the O_DIRECT interface. However, that still 
> doesn't enable me to tell you what exact implementation you'll need. Do 
> you understand my point?

Evidently not.  You must be saying that there are two possible
implementations, and that it is up to me to choose one that I like.
That's nice, but I would like to have the possibilities to choose from
in front of me! While we know schoedinger's cat may only exist or not
exist, it is only possible to say which of the two I prefer once I
have seen at least one of them :-).

Peter

Re: block device/VM question

[Peter T. Breuer]

"A month of sundays ago ptb wrote:"
> "A month of sundays ago ptb wrote:"
> > "A month of sundays ago Thunder from the hill wrote:"
> > > > And for the O_DIRECT flag we seem to do alloc_kiovec(1, &f->f_iobuf).
> > > 
> > > Perhaps we should go biovec here?

I've now tried opening the ram disk device O_DIRECT in 2.4.19. Same result
as on my driver: every write returns EINVAL.

How does one use O_DIRECT? It needs driver support?

Peter

Re: block device/VM question

[Peter T. Breuer]

"A month of sundays ago ptb wrote:"
> "A month of sundays ago Thunder from the hill wrote:"
> > > And for the O_DIRECT flag we seem to do alloc_kiovec(1, &f->f_iobuf).
> > 
> > Perhaps we should go biovec here?

State of play:

1) I abandoned doing the dentry_open with O_DIRECT flags bit
from within the driver, because every external access then errored
with EINVAL without getting to my driver functions.

2) I then just tried opening the device normally from userspace
but with O_DIRECT set. This is harder than it should have been - I had
to set nearly everything in features.h on before I got the desired
compilation (was it _XOPEN_SOURCE that did it? I forget), and surprise,
surprise, I get exactly the same effects as in (1). Every access
errors with EINVAL and my driver code never gets called. An open
with code compiled without O_DIRECT works fine, as usual.

I put a little debugging in the driver open call. With O_DIRECT
in the userspace open:

#881[12]: _open have O_DIRECT iobuf 0xcf935000 for part 0
#883[12]: _open have i_mapping 0xcce7f874 for part 0
#885[12]: _open have a_ops 0xc029ef40 for part 0
#889[12]: _open have direct_IO 0xc013edc0 for part 0

(I presume the latter is generic_file_direct_IO - it's not in my
map, but it's static so shouldn't be).

and without:

#881[14]: _open have ~O_DIRECT iobuf 0x0 for part 0
#883[14]: _open have i_mapping 0xcce7f874 for part 0
#885[14]: _open have a_ops 0xc029ef40 for part 0
#889[14]: _open have direct_IO 0xc013edc0 for part 0

So it looks as though some driver cooperation is required in
order for direct_IO/O_DIRECT to work.

What? Anyone know?


Peter

Re: block device/VM question

[Peter T. Breuer]

"A month of sundays ago Thunder from the hill wrote:"
> > And for the O_DIRECT flag we seem to do alloc_kiovec(1, &f->f_iobuf).
> 
> Perhaps we should go biovec here?
> 
> For you, if you can stand it you can even go directly into the dio stuff 
> from direct-io.c. You'll just need to know what to do. Or you fill your 
> information into some underway function.

Yes, the 2.5.31 code looks much much simpler. But you encouraged me to
look at 2.4.19 by pointing out that there has been support since 2.4.10,
so that's what I'm looking at!

I'm sure I'm just missing a couple of methods in some struct.  I'll test
a few on the way home in the train.  Things look good, just no methods to
do the work when the O_DIRECT flag is set. So I get EINVAL on every
read/write access.

I'm getting the hang of it. At every open of the device I look at the
file pointer that they're opening and check to see if it has an iobuff
field set. If not, I set it (and the O_DIRECT flag if necessary).
At every release of the device, I look at the file they're releasing
and if it has an iobuff field, I free it. I guess I should set a
flag to say "I did it", but for the moment, I guess it's only me
in the driver doing it. Kernel programming would be so much easier
if there were an explanation of what things were for :-).

I'll trace what happens in the generic_read() /write() stuff. They'll
be usiing the iobuf there.

Thanks.

Peter

Re: block device/VM question

[Thunder from the hill]

Hi,

On Tue, 27 Aug 2002, Peter T. Breuer wrote:
> Yecch. I have the inode of the sepecial device file. I don't want to
> know the name. I even have a file pointer.

Then you're obviously for the wrong stuff.

> In dentry_open(), we get a struct file f = get_empty_filp(), and then
> fill out various of its fields with enormously obscure things.

For a good reason: we always need to fill in the values somewhere, they 
don't quite come from heaven.

> And for the O_DIRECT flag we seem to do alloc_kiovec(1, &f->f_iobuf).

Perhaps we should go biovec here?

For you, if you can stand it you can even go directly into the dio stuff 
from direct-io.c. You'll just need to know what to do. Or you fill your 
information into some underway function.

			Thunder
-- 
--./../...-/. -.--/---/..-/.-./..././.-../..-. .---/..-/.../- .-
--/../-./..-/-/./--..-- ../.----./.-../.-.. --./../...-/. -.--/---/..-
.- -/---/--/---/.-./.-./---/.--/.-.-.-
--./.-/-.../.-./.././.-../.-.-.-

Re: block device/VM question

[Peter T. Breuer]

"ago ptb wrote:"
> In dentry_open(), we get a struct file f = get_empty_filp(), and then
> fill out various of its fields with enormously obscure things.  And for
> the O_DIRECT flag we seem to do alloc_kiovec(1, &f->f_iobuf).
> 
> I feel that the latter is all I want to do, and the question is to what,
> where (I'll clean up on release). Do I do this every time the devices
> _open() function is called? Or just once, and what do I do it to? I
> should do it to the struct file that gets passed into to the driver
> open()? I'll try that. And set the flag.

Well, that was fun! I checked that on entry into the devices
open function, the file->f_iobuf field was null, and then called
alloc_kiovec on it while I set the O_DIRECT flag on file-_f_flags.

The result was that all read/write calls on the device failed
with EINVAL! Whee!

But ioctls worked. Apparently I am supposed to fill out
some more fields of something else with some methods. Hmm. OK.
I'll look. I guess this will be the a_ops field of i_mapping.


Peter

Re: block device/VM question

[Peter T. Breuer]

"A month of sundays ago Thunder from the hill wrote:"
> > there I can do whatever the sysopen with O_DIRECT does.
> 
> I try to say, see how sys_open() handles O_DIRECT. You should be able to 
> do just the same.

Well, how it handles it is manifestly unclear. It seems to trace down to
the dentry and call dentry_open(), but with a struct vfsmount as
another arg. I don't know what that's for. It got the vfsmount by aother
esoteric lookup, getting a struct nameidata from the filename via
open_namei.

Yecch. I have the inode of the sepecial device file. I don't want to
know the name. I even have a file pointer.

In dentry_open(), we get a struct file f = get_empty_filp(), and then
fill out various of its fields with enormously obscure things.  And for
the O_DIRECT flag we seem to do alloc_kiovec(1, &f->f_iobuf).

I feel that the latter is all I want to do, and the question is to what,
where (I'll clean up on release). Do I do this every time the devices
_open() function is called? Or just once, and what do I do it to? I
should do it to the struct file that gets passed into to the driver
open()? I'll try that. And set the flag.

Peter

Re: block device/VM question

[Peter T. Breuer]

"A month of sundays ago Thunder from the hill wrote:"
> On Tue, 27 Aug 2002, Peter T. Breuer wrote:
> > > Why so rough?
> > 
> > Rough? You mean "approximate"? I was working from my vague mamory of
> > what I read in the code on the train this morning.
> 
> No, rough. I meant, why not use sys_open()?

You mean do a syscall with flags O_DIRECT from within the driver?
I don't like syscalsl from kernel code, but if you say so, I'll think
about it.

> Certainly not, as I'm not using libc6. But my manpages are newer, the 
> latest version is appended. It might clear this issue to you.

Thanks.

> > Oh, well, thanks.I suppose you won't give me a recipe saying "do this
> > and your device won't be cached", but I'll follow the lead.
> 
> I can't, because I don't have your code here and can't tell your needs. I 
> just try to serve it all. (I mean it all depends not just on where you go, 
> but also on what you have.)

Mmm ..  I must say I don't see how.  Making all requests on a block
device not be cached by VMS seems to me to have a globally defined
semantics, not confined to my particular use or context! Are you
suggesting that it depends on the mode of access to the device? That I
can't control - I simply want ALL accesses to not be cached.

Peter

Re: block device/VM question

[Thunder from the hill]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1018 bytes --]

Hi,

On Tue, 27 Aug 2002, Peter T. Breuer wrote:
> > Why so rough?
> 
> Rough? You mean "approximate"? I was working from my vague mamory of
> what I read in the code on the train this morning.

No, rough. I meant, why not use sys_open()?

> My manpage for open(2) claims no such thing. It doesn't mention
> O_DIRECT. Maybe your libc6 is newer.

Certainly not, as I'm not using libc6. But my manpages are newer, the 
latest version is appended. It might clear this issue to you.

> Oh, well, thanks.I suppose you won't give me a recipe saying "do this
> and your device won't be cached", but I'll follow the lead.

I can't, because I don't have your code here and can't tell your needs. I 
just try to serve it all. (I mean it all depends not just on where you go, 
but also on what you have.)

			Thunder
-- 
--./../...-/. -.--/---/..-/.-./..././.-../..-. .---/..-/.../- .-
--/../-./..-/-/./--..-- ../.----./.-../.-.. --./../...-/. -.--/---/..-
.- -/---/--/---/.-./.-./---/.--/.-.-.-
--./.-/-.../.-./.././.-../.-.-.-

[-- Attachment #2: Type: APPLICATION/octet-stream, Size: 4972 bytes --]

Re: block device/VM question

[Peter T. Breuer]

"A month of sundays ago Thunder from the hill wrote:"
> On Tue, 27 Aug 2002, Peter T. Breuer wrote:
> > Yes, I've noticed this in the 2.5.31 kernel. This is something
> > to be done on open by the overlying fs or userspace utility, or
> > should I set the flag on the inode->filp (or whatever) myself in
> > the drivers open function? And do I need to define
> > inode->mapping->a_ops->direct_IO()? (sorry - but I haven't had time to
> > experiment yet today!).
> 
> Why so rough?

Rough? You mean "approximate"? I was working from my vague mamory of
what I read in the code on the train this morning.

> > It looks like the 2.5 kernel has this pathway, but what about the 2.4
> > kernel? Nothing.
> 
> The manpage claims:
> 
>     O_DIRECT
> 	This flag is supported on a number of Unix-like systems;
> 	support was added under Linux in kernel version 2.4.10.

My manpage for open(2) claims no such thing. It doesn't mention
O_DIRECT. Maybe your libc6 is newer.

In any case, it doesn't make it clear to me if I have to set the flag
in the driver. Well, anyway, I'll set it.

> Also:
> 
> 	A semantically similar interface for block devices is 
> 	described in raw(8).

That's nothing like, since it's a character device that they were
describing. But yes, reading drivers/char/raw.c led me to see the
pathway via direct_IO() in 2.5.31, but in the 2.4.19 kernel the path is
obscure. In 2.5.31 it's clear that mm/filemap.c does pay attention to the
flag, and YES, you are right, in 2.4.19, generic_file_read() does
look at the flag. However, I have not much idea where that filp
comes from or what it represents.  Oh, well, thanks.I suppose
you won't give me a recipe saying "do this and your device won't be
cached", but I'll follow the lead.

Thanks again.


Peter

Re: block device/VM question

[Peter T. Breuer]

"A month of sundays ago Thunder from the hill wrote:"
> Hi,
> On Tue, 27 Aug 2002, Peter T. Breuer wrote:
> > Is there any way of turning off VMS caching for a block device?

> O_DIRECT, or easily set the buffer to zero...

Thanks!

Yes, I've noticed this in the 2.5.31 kernel. This is something
to be done on open by the overlying fs or userspace utility, or
should I set the flag on the inode->filp (or whatever) myself in
the drivers open function? And do I need to define
inode->mapping->a_ops->direct_IO()? (sorry - but I haven't had time to
experiment yet today!).

It looks like the 2.5 kernel has this pathway, but what about the 2.4
kernel? Nothing.

Peter

Re: block device/VM question

[Thunder from the hill]

Hi,

On Tue, 27 Aug 2002, Peter T. Breuer wrote:
> Is there any way of turning off VMS caching for a block device?
> 
> I want all reads to come down to the driver, where I decide what to do
> about them.  I don't want reads to read locally cached buffers in VMS
> unless I say so.  The reason is that the device might have a remote
> writer.
> 
> I'll have a look at the raw character device later (but I recall
> having looked before without it telling me anything - probably
> they make a fake request and transfer it to the device queue
> directly and treat the return with their own substituted end_req).
> I need a block device - I can't mount a character device. Now
> there's an idea! A mouse represented as a file system ..

O_DIRECT, or easily set the buffer to zero...

			Thunder
-- 
--./../...-/. -.--/---/..-/.-./..././.-../..-. .---/..-/.../- .-
--/../-./..-/-/./--..-- ../.----./.-../.-.. --./../...-/. -.--/---/..-
.- -/---/--/---/.-./.-./---/.--/.-.-.-
--./.-/-.../.-./.././.-../.-.-.-

Re: block device/VM question

[Peter T. Breuer]

PTB  wrote:
> Is there any way of turning off VMS caching for a block device?

Well, I've been reading the 2.5.31 code, and it looks like if
the block device node is opened O_DIRECT and it has an
i_mapping->a_ops->direct_IO method, then file systems will use it
for read/write and won't go thorgh VMS.  I'll investigate.

Does anyone have a pointer for where to go in the 2.4 code?

Peter

block device/VM question

[Peter T. Breuer]

Hi

Is there any way of turning off VMS caching for a block device?

I want all reads to come down to the driver, where I decide what to do
about them.  I don't want reads to read locally cached buffers in VMS
unless I say so.  The reason is that the device might have a remote
writer.

I'll have a look at the raw character device later (but I recall
having looked before without it telling me anything - probably
they make a fake request and transfer it to the device queue
directly and treat the return with their own substituted end_req).
I need a block device - I can't mount a character device. Now
there's an idea! A mouse represented as a file system ..

Peter