Vramfs: filesystem driver to utilize extra RAM on VGA devices

Vramfs: filesystem driver to utilize extra RAM on VGA devices

[Jonathan Campbell]

Hey guys.
About a month ago while covered in the Seattle snowstorm I hacked 
together this pseudofilesystem that might be of interest.

I thought that this driver could solve two issues that I have:

one, that today's graphics cards have relatively obscene amounts of RAM 
on them even if you're not using it. If you're running it as a server 
and not using it for 3D graphics, why not mount the VRAM on the graphics 
card as a filesystem and store things there to get some extra space?

two, if 3D hardware acceleration and access to GPU or texture memory 
could be provided to user-space, one way to do it would be to provide 
sections of VRAM as a filesystem that most languages (yes---even Perl!) 
could use to work with todays graphics cards. They could treat the 
texture memory the way they treat files in /dev/shm: read/write it for 
general access or mmap it for direct manipulation. At least, it makes 
far more sense to me from a programming point of view than to abstract 
it using specialized ioctls through the DRI. It might make writing an 
OpenGL driver for this kind of arrangement cleaner, too.

http://www.nerdgrounds.com/vramfs-20090126-1458.tar.gz

So far I've tested it against 2.6.25.17 and 2.6.28 on both x86 and 
x86_64 with reads, writes, directory creation, symlink creation, and 
mmap() and it seems to work fine.
Just give it a range of memory on the bus, or the 
domain:bus:device:function numbers of a VGA PCI device, and it will 
mount the VGA video RAM and allow files to exist there.
As a special hack: you can also specify the size of the active 
framebuffer console so that fbcon doesn't collide with this driver 
(unless you want to see what your files look like splattered across your 
screen, ha). The active VRAM area becomes a "sentinel" file named 
"framebuffer".

What do you guys think?

Jonathan Campbell
Impact Studio Pro

Re: Vramfs: filesystem driver to utilize extra RAM on VGA devices

[H. Peter Anvin]

Jonathan Campbell wrote:
> 
> So far I've tested it against 2.6.25.17 and 2.6.28 on both x86 and 
> x86_64 with reads, writes, directory creation, symlink creation, and 
> mmap() and it seems to work fine.
> Just give it a range of memory on the bus, or the 
> domain:bus:device:function numbers of a VGA PCI device, and it will 
> mount the VGA video RAM and allow files to exist there.
> As a special hack: you can also specify the size of the active 
> framebuffer console so that fbcon doesn't collide with this driver 
> (unless you want to see what your files look like splattered across your 
> screen, ha). The active VRAM area becomes a "sentinel" file named 
> "framebuffer".
> 
> What do you guys think?
> 

How is this different from the MTD driver we already have?

	-hpa

Re: Vramfs: filesystem driver to utilize extra RAM on VGA devices

[Jonathan Campbell]

vramfs is designed to take the memory range and directly turn it into a 
usable filesystem.
The structures are not actually in VRAM, but the file contents are.
Vramfs has a builtin mechanism as described to avoid conflicting with 
the region in use by the framebuffer console.

I don't really know about the mtd device, but I thought it would be good 
kernel coding practice to write a filesystem driver to pull off a stunt 
like that.

I also wrote this in consideration of the GPU which probably wouldn't 
know how to handle the fragmentation that would inevitably happen if 
ext3 needed to write blocks in a non-contiguous manner, this fs enforces 
the rule that files are always unbroken with only a start and length.

Also, doesn't mtd come in as a block device?
So you'd have to format the memory region using a filesystem like ext3, 
right?
And as a block device you can't use mmap() to map that region directly 
into your process space, right?
> Jonathan Campbell wrote:
>>
>> So far I've tested it against 2.6.25.17 and 2.6.28 on both x86 and 
>> x86_64 with reads, writes, directory creation, symlink creation, and 
>> mmap() and it seems to work fine.
>> Just give it a range of memory on the bus, or the 
>> domain:bus:device:function numbers of a VGA PCI device, and it will 
>> mount the VGA video RAM and allow files to exist there.
>> As a special hack: you can also specify the size of the active 
>> framebuffer console so that fbcon doesn't collide with this driver 
>> (unless you want to see what your files look like splattered across 
>> your screen, ha). The active VRAM area becomes a "sentinel" file 
>> named "framebuffer".
>>
>> What do you guys think?
>>
>
> How is this different from the MTD driver we already have?
>
>     -hpa
> -- 
> To unsubscribe from this list: send the line "unsubscribe 
> linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>
>

Re: Vramfs: filesystem driver to utilize extra RAM on VGA devices

[Peter W. Morreale]

On Mon, 2009-01-26 at 15:50 -0800, Jonathan Campbell wrote:
> vramfs is designed to take the memory range and directly turn it into a 
> usable filesystem.
> The structures are not actually in VRAM, but the file contents are.
> Vramfs has a builtin mechanism as described to avoid conflicting with 
> the region in use by the framebuffer console.
> 
> I don't really know about the mtd device, but I thought it would be good 
> kernel coding practice to write a filesystem driver to pull off a stunt 
> like that.
> 
> I also wrote this in consideration of the GPU which probably wouldn't 
> know how to handle the fragmentation that would inevitably happen if 
> ext3 needed to write blocks in a non-contiguous manner, this fs enforces 
> the rule that files are always unbroken with only a start and length.
> 

How do you accomplish that? 

(I haven't yet looked at your code...)

-PWM


> Also, doesn't mtd come in as a block device?
> So you'd have to format the memory region using a filesystem like ext3, 
> right?
> And as a block device you can't use mmap() to map that region directly 
> into your process space, right?
> > Jonathan Campbell wrote:
> >>
> >> So far I've tested it against 2.6.25.17 and 2.6.28 on both x86 and 
> >> x86_64 with reads, writes, directory creation, symlink creation, and 
> >> mmap() and it seems to work fine.
> >> Just give it a range of memory on the bus, or the 
> >> domain:bus:device:function numbers of a VGA PCI device, and it will 
> >> mount the VGA video RAM and allow files to exist there.
> >> As a special hack: you can also specify the size of the active 
> >> framebuffer console so that fbcon doesn't collide with this driver 
> >> (unless you want to see what your files look like splattered across 
> >> your screen, ha). The active VRAM area becomes a "sentinel" file 
> >> named "framebuffer".
> >>
> >> What do you guys think?
> >>
> >
> > How is this different from the MTD driver we already have?
> >
> >     -hpa
> > -- 
> > To unsubscribe from this list: send the line "unsubscribe 
> > linux-kernel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > Please read the FAQ at  http://www.tux.org/lkml/
> >
> >
> 
> _______________________________________________
> devel mailing list
> devel@linuxdriverproject.org
> http://driverdev.linuxdriverproject.org/mailman/listinfo/devel

Re: Vramfs: filesystem driver to utilize extra RAM on VGA devices

[Jonathan Campbell]

Every VGA device connected to the PCI bus has resources that are known 
to the Linux kernel. One of the memory resources is the video RAM 
(always the largest prefetchable resource listed), with the base and 
length known to the PCI subsystem. So, all vramfs needs on mount is the 
PCI device, and the mountpoint, and it can dole it out to userspace. 
Reading and writing is handled with ioremap(). mmap() is handled by 
directly mmaping() that VRAM into user-space (just like /dev/fb0). Files 
are recorded in system memory as a list of entities with a starting 
block and length, so no fragmentation is possible.

No communication with the GPU is needed to do this, it's possible to use 
VRAM this way even on devices that nobody in the system knows how to 
talk to.
> How do you accomplish that? 
>
> (I haven't yet looked at your code...)
>
> -PWM
>
>   

Re: Vramfs: filesystem driver to utilize extra RAM on VGA devices

[H. Peter Anvin]

Jonathan Campbell wrote:
> Every VGA device connected to the PCI bus has resources that are known
> to the Linux kernel. One of the memory resources is the video RAM
> (always the largest prefetchable resource listed), with the base and
> length known to the PCI subsystem. So, all vramfs needs on mount is the
> PCI device, and the mountpoint, and it can dole it out to userspace.
> Reading and writing is handled with ioremap(). mmap() is handled by
> directly mmaping() that VRAM into user-space (just like /dev/fb0). Files
> are recorded in system memory as a list of entities with a starting
> block and length, so no fragmentation is possible.
> 
> No communication with the GPU is needed to do this, it's possible to use
> VRAM this way even on devices that nobody in the system knows how to
> talk to.

There still seems to be quite a bit of overlap between this and the MTD
subsystem...

	-hpa

-- 
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.

Re: Vramfs: filesystem driver to utilize extra RAM on VGA devices

[Jonathan Campbell]

I don't really see the similarity between the MTD subsystem and dividing 
vram up by files. Video cards don't have "erase blocks". And MTD is not 
a filesystem. And the onboard memory mtd driver (map system RAM) only 
handles one fixed region determined at load time.

Vramfs on the other hand determines what resources to use at mount time. 
It supports multiple mounts, one per PCI device, if you want the 
combined VRAM of two VGA cards in your system.

I don't think vramfs would fit well into the MTD subsystem.
> Jonathan Campbell wrote:
>   
>> Every VGA device connected to the PCI bus has resources that are known
>> to the Linux kernel. One of the memory resources is the video RAM
>> (always the largest prefetchable resource listed), with the base and
>> length known to the PCI subsystem. So, all vramfs needs on mount is the
>> PCI device, and the mountpoint, and it can dole it out to userspace.
>> Reading and writing is handled with ioremap(). mmap() is handled by
>> directly mmaping() that VRAM into user-space (just like /dev/fb0). Files
>> are recorded in system memory as a list of entities with a starting
>> block and length, so no fragmentation is possible.
>>
>> No communication with the GPU is needed to do this, it's possible to use
>> VRAM this way even on devices that nobody in the system knows how to
>> talk to.
>>     
>
> There still seems to be quite a bit of overlap between this and the MTD
> subsystem...
>
> 	-hpa
>
>   

Re: Vramfs: filesystem driver to utilize extra RAM on VGA devices

[H. Peter Anvin]

Jonathan Campbell wrote:
> I don't really see the similarity between the MTD subsystem and dividing
> vram up by files. Video cards don't have "erase blocks". And MTD is not
> a filesystem. And the onboard memory mtd driver (map system RAM) only
> handles one fixed region determined at load time.
> 
> Vramfs on the other hand determines what resources to use at mount time.
> It supports multiple mounts, one per PCI device, if you want the
> combined VRAM of two VGA cards in your system.
> 
> I don't think vramfs would fit well into the MTD subsystem.

Think about it this way: it is a continuum of facilities.  Why is VRAM
different than, say, external DRAM that can only be accessed via a DMA
engine?  Now swap the DRAM with NAND flash and perform the same exercise.

Perhaps the most important reason is that you want to be able to use
this as backing store for swap.  Currently the mm doesn't handle swap
which is much faster than filesystems very well, but that should be
possible to address.

	-hpa

-- 
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.

Re: Vramfs: filesystem driver to utilize extra RAM on VGA devices

[Willy Tarreau]

On Thu, Jan 29, 2009 at 08:46:06PM -0800, H. Peter Anvin wrote:
> Jonathan Campbell wrote:
> > I don't really see the similarity between the MTD subsystem and dividing
> > vram up by files. Video cards don't have "erase blocks". And MTD is not
> > a filesystem. And the onboard memory mtd driver (map system RAM) only
> > handles one fixed region determined at load time.
> > 
> > Vramfs on the other hand determines what resources to use at mount time.
> > It supports multiple mounts, one per PCI device, if you want the
> > combined VRAM of two VGA cards in your system.
> > 
> > I don't think vramfs would fit well into the MTD subsystem.
> 
> Think about it this way: it is a continuum of facilities.  Why is VRAM
> different than, say, external DRAM that can only be accessed via a DMA
> engine?  Now swap the DRAM with NAND flash and perform the same exercise.
> 
> Perhaps the most important reason is that you want to be able to use
> this as backing store for swap.  Currently the mm doesn't handle swap
> which is much faster than filesystems very well, but that should be
> possible to address.

Also, MTD allows you to map *part* of the VRAM. On my laptop, I only
map 56 MB out of the 64 MB of VRAM, which still allows me to use the
VGA normally. And BTW, I map this VRAM as a fast swap memory.

In fact, I see little use to the VRAMFS for the average user. Still
it might have been a nice coding exercice for Jonathan.

Willy

Re: Vramfs: filesystem driver to utilize extra RAM on VGA devices

[H. Peter Anvin]

Willy Tarreau wrote:
> 
> Also, MTD allows you to map *part* of the VRAM. On my laptop, I only
> map 56 MB out of the 64 MB of VRAM, which still allows me to use the
> VGA normally. And BTW, I map this VRAM as a fast swap memory.
> 
> In fact, I see little use to the VRAMFS for the average user. Still
> it might have been a nice coding exercice for Jonathan.
> 

Also, the idea of using the DMA engine on the video card rather than
just doing PIO references is an excellent one.

	-hpa

-- 
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.

Re: Vramfs: filesystem driver to utilize extra RAM on VGA devices

[Mark Knecht]

On Mon, Jan 26, 2009 at 3:20 PM, Jonathan Campbell <jon@nerdgrounds.com> wrote:
> Hey guys.
> About a month ago while covered in the Seattle snowstorm I hacked together
> this pseudofilesystem that might be of interest.
>
> I thought that this driver could solve two issues that I have:
>
> one, that today's graphics cards have relatively obscene amounts of RAM on
> them even if you're not using it. If you're running it as a server and not
> using it for 3D graphics, why not mount the VRAM on the graphics card as a
> filesystem and store things there to get some extra space?
>
> two, if 3D hardware acceleration and access to GPU or texture memory could
> be provided to user-space, one way to do it would be to provide sections of
> VRAM as a filesystem that most languages (yes---even Perl!) could use to
> work with todays graphics cards. They could treat the texture memory the way
> they treat files in /dev/shm: read/write it for general access or mmap it
> for direct manipulation. At least, it makes far more sense to me from a
> programming point of view than to abstract it using specialized ioctls
> through the DRI. It might make writing an OpenGL driver for this kind of
> arrangement cleaner, too.
>
> http://www.nerdgrounds.com/vramfs-20090126-1458.tar.gz
>
> So far I've tested it against 2.6.25.17 and 2.6.28 on both x86 and x86_64
> with reads, writes, directory creation, symlink creation, and mmap() and it
> seems to work fine.
> Just give it a range of memory on the bus, or the domain:bus:device:function
> numbers of a VGA PCI device, and it will mount the VGA video RAM and allow
> files to exist there.
> As a special hack: you can also specify the size of the active framebuffer
> console so that fbcon doesn't collide with this driver (unless you want to
> see what your files look like splattered across your screen, ha). The active
> VRAM area becomes a "sentinel" file named "framebuffer".
>
> What do you guys think?
>
> Jonathan Campbell
> Impact Studio Pro
>

Can the GPU use the data placed in your file system? Do you have
strong control as to exactly how the data is mapped into VRAM? I'm
thinking about parallel processing  - Linux puts data there and then
the GPU works on it to produce a result which Linux can eventually
fetch.

- Mark

Re: Vramfs: filesystem driver to utilize extra RAM on VGA devices

[Eric Anholt]

[-- Attachment #1: Type: text/plain, Size: 2766 bytes --]

On Mon, 2009-01-26 at 18:59 -0800, Mark Knecht wrote:
> On Mon, Jan 26, 2009 at 3:20 PM, Jonathan Campbell <jon@nerdgrounds.com> wrote:
> > Hey guys.
> > About a month ago while covered in the Seattle snowstorm I hacked together
> > this pseudofilesystem that might be of interest.
> >
> > I thought that this driver could solve two issues that I have:
> >
> > one, that today's graphics cards have relatively obscene amounts of RAM on
> > them even if you're not using it. If you're running it as a server and not
> > using it for 3D graphics, why not mount the VRAM on the graphics card as a
> > filesystem and store things there to get some extra space?
> >
> > two, if 3D hardware acceleration and access to GPU or texture memory could
> > be provided to user-space, one way to do it would be to provide sections of
> > VRAM as a filesystem that most languages (yes---even Perl!) could use to
> > work with todays graphics cards. They could treat the texture memory the way
> > they treat files in /dev/shm: read/write it for general access or mmap it
> > for direct manipulation. At least, it makes far more sense to me from a
> > programming point of view than to abstract it using specialized ioctls
> > through the DRI. It might make writing an OpenGL driver for this kind of
> > arrangement cleaner, too.
> >
> > http://www.nerdgrounds.com/vramfs-20090126-1458.tar.gz
> >
> > So far I've tested it against 2.6.25.17 and 2.6.28 on both x86 and x86_64
> > with reads, writes, directory creation, symlink creation, and mmap() and it
> > seems to work fine.
> > Just give it a range of memory on the bus, or the domain:bus:device:function
> > numbers of a VGA PCI device, and it will mount the VGA video RAM and allow
> > files to exist there.
> > As a special hack: you can also specify the size of the active framebuffer
> > console so that fbcon doesn't collide with this driver (unless you want to
> > see what your files look like splattered across your screen, ha). The active
> > VRAM area becomes a "sentinel" file named "framebuffer".
> >
> > What do you guys think?
> >
> > Jonathan Campbell
> > Impact Studio Pro
> >
> 
> Can the GPU use the data placed in your file system? Do you have
> strong control as to exactly how the data is mapped into VRAM? I'm
> thinking about parallel processing  - Linux puts data there and then
> the GPU works on it to produce a result which Linux can eventually
> fetch.

For that you want something like GEM, which is aware of the graphics
pipeline and the cache management necessary.  Wrapping a filesystem
around it shouldn't be hard, and would be of some use for debugging.

-- 
Eric Anholt
eric@anholt.net                         eric.anholt@intel.com



[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Vramfs: filesystem driver to utilize extra RAM on VGA devices

[Mark Knecht]

On Mon, Jan 26, 2009 at 8:44 PM, Eric Anholt <eric@anholt.net> wrote:
> On Mon, 2009-01-26 at 18:59 -0800, Mark Knecht wrote:
<SNIP>
>> >
>>
>> Can the GPU use the data placed in your file system? Do you have
>> strong control as to exactly how the data is mapped into VRAM? I'm
>> thinking about parallel processing  - Linux puts data there and then
>> the GPU works on it to produce a result which Linux can eventually
>> fetch.
>
> For that you want something like GEM, which is aware of the graphics
> pipeline and the cache management necessary.  Wrapping a filesystem
> around it shouldn't be hard, and would be of some use for debugging.
>
> --
> Eric Anholt

Right. I agree. However over time a number of us in the pro-audio area
have thought about using the GPU for things like building complex
reverb convolutions in real-time. Lots of parallelism in the math.
These machines are not typically very graphically intensive and the
VRAM on the cards isn't required but it's there. Somehow being able to
get data in and out of the VRAM using standard file commands, letting
something like GEM do the work, and then getting the data back seems
appealing.

I hope Jonathan keeps up the efforts. Interesting stuff.

- Mark

Re: Vramfs: filesystem driver to utilize extra RAM on VGA devices

[Eric Anholt]

[-- Attachment #1: Type: text/plain, Size: 2119 bytes --]

On Tue, 2009-01-27 at 09:37 -0800, Mark Knecht wrote:
> On Mon, Jan 26, 2009 at 8:44 PM, Eric Anholt <eric@anholt.net> wrote:
> > On Mon, 2009-01-26 at 18:59 -0800, Mark Knecht wrote:
> <SNIP>
> >> >
> >>
> >> Can the GPU use the data placed in your file system? Do you have
> >> strong control as to exactly how the data is mapped into VRAM? I'm
> >> thinking about parallel processing  - Linux puts data there and then
> >> the GPU works on it to produce a result which Linux can eventually
> >> fetch.
> >
> > For that you want something like GEM, which is aware of the graphics
> > pipeline and the cache management necessary.  Wrapping a filesystem
> > around it shouldn't be hard, and would be of some use for debugging.
> >
> > --
> > Eric Anholt
> 
> Right. I agree. However over time a number of us in the pro-audio area
> have thought about using the GPU for things like building complex
> reverb convolutions in real-time. Lots of parallelism in the math.
> These machines are not typically very graphically intensive and the
> VRAM on the cards isn't required but it's there. Somehow being able to
> get data in and out of the VRAM using standard file commands, letting
> something like GEM do the work, and then getting the data back seems
> appealing.

The way you want do that is using OpenGL to put your data in textures
and framebuffer objects, and render them.  With KMS, we'll be able to
support EGL even on the console so you can do the work without having an
X environment set up.

The problem with vramfs as a basis for GPU offload is that most GPU
tasks end up at some point exceeding the size of available
aperture/VRAM.  So you need code that manages loading buffer objects in
and out on demand, managing the execution pipeline and GPU and CPU
caches as required.  We have that with GEM already.

Remember, writing data to an aperture isn't the hard part of offloading
to the GPU, programming the GPU is.  That's why you use OpenGL or
another abstraction to do it.

-- 
Eric Anholt
eric@anholt.net                         eric.anholt@intel.com



[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Vramfs: filesystem driver to utilize extra RAM on VGA devices

[Jonathan Campbell]

Well my expectation of vramfs is that it's not meant to be used for the 
heavy-duty 3D gaming-style rendering that GEM is built to handle. It's 
meant for lighter tasks, like simple 2D/3D compositing or GPU work where 
you know what your resources need, you don't need to take that much, and 
you need direct mmap() access because some of it involves video that you 
will handle later. Situations like this can work perfectly fine without 
the use of a swapping system.

My other concern is that GEM with a filesystem might be the best option 
for 3D gaming, but that it wouldn't work if the driver doesn't know the 
card. The DRI drivers, as far as I know, are tied to the GPU and chipset 
of the device (because they have to manage it, after all!). How exactly 
would GEM work for cards that it doesn't recognize, like one machine of 
mine with a weird ATI chipset nobody knows how to talk to? If GEM 
doesn't recognize it, it won't provide VRAM resources to use it, right?

This is where vramfs has it's advantage: it's not the absolute best 
solution for 3D graphics, but it's simple, it can serve as a starting 
point for GPU experiments from userspace, or if nothing else allows the 
use of the onboard video RAM on an otherwise unused and unrecognized 
video device. It's device-agnostic by design.
> The way you want do that is using OpenGL to put your data in textures
> and framebuffer objects, and render them.  With KMS, we'll be able to
> support EGL even on the console so you can do the work without having an
> X environment set up.
>
> The problem with vramfs as a basis for GPU offload is that most GPU
> tasks end up at some point exceeding the size of available
> aperture/VRAM.  So you need code that manages loading buffer objects in
> and out on demand, managing the execution pipeline and GPU and CPU
> caches as required.  We have that with GEM already.
>
> Remember, writing data to an aperture isn't the hard part of offloading
> to the GPU, programming the GPU is.  That's why you use OpenGL or
> another abstraction to do it.
>
>   

Re: Vramfs: filesystem driver to utilize extra RAM on VGA devices

[Dave Airlie]

On Wed, Jan 28, 2009 at 4:36 PM, Jonathan Campbell <jon@nerdgrounds.com> wrote:
> Well my expectation of vramfs is that it's not meant to be used for the
> heavy-duty 3D gaming-style rendering that GEM is built to handle. It's meant
> for lighter tasks, like simple 2D/3D compositing or GPU work where you know
> what your resources need, you don't need to take that much, and you need
> direct mmap() access because some of it involves video that you will handle
> later. Situations like this can work perfectly fine without the use of a
> swapping system.

I can't think of a nice way to say this, but this idea is crack.

I can see one use for vramfs which is to use VRAM as a file backing
store, we won't mention that VRAM is slow. Some caveats being
the PCI aperture only exposes 1/2 the VRAM on some cards, there is
more hidden away where the CPU can't find it.
 and that some cards have made up VRAM (read IGPs).

It doesn't in any way serve as a basis for talking to the GPU behind
said VRAM or as a basis for any
sort of backend to talk to the GPU.

Dave.

>
> My other concern is that GEM with a filesystem might be the best option for
> 3D gaming, but that it wouldn't work if the driver doesn't know the card.
> The DRI drivers, as far as I know, are tied to the GPU and chipset of the
> device (because they have to manage it, after all!). How exactly would GEM
> work for cards that it doesn't recognize, like one machine of mine with a
> weird ATI chipset nobody knows how to talk to? If GEM doesn't recognize it,
> it won't provide VRAM resources to use it, right?
>
> This is where vramfs has it's advantage: it's not the absolute best solution
> for 3D graphics, but it's simple, it can serve as a starting point for GPU
> experiments from userspace, or if nothing else allows the use of the onboard
> video RAM on an otherwise unused and unrecognized video device. It's
> device-agnostic by design.
>>
>> The way you want do that is using OpenGL to put your data in textures
>> and framebuffer objects, and render them.  With KMS, we'll be able to
>> support EGL even on the console so you can do the work without having an
>> X environment set up.
>>
>> The problem with vramfs as a basis for GPU offload is that most GPU
>> tasks end up at some point exceeding the size of available
>> aperture/VRAM.  So you need code that manages loading buffer objects in
>> and out on demand, managing the execution pipeline and GPU and CPU
>> caches as required.  We have that with GEM already.
>>
>> Remember, writing data to an aperture isn't the hard part of offloading
>> to the GPU, programming the GPU is.  That's why you use OpenGL or
>> another abstraction to do it.
>>
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: Vramfs: filesystem driver to utilize extra RAM on VGA devices

[Jonathan Campbell]

I can't think of a nice way to say this, but this idea is crack.

Hey, thanks :)
> I can see one use for vramfs which is to use VRAM as a file backing
> store, we won't mention that VRAM is slow. Some caveats being
> the PCI aperture only exposes 1/2 the VRAM on some cards, there is
> more hidden away where the CPU can't find it.
>  and that some cards have made up VRAM (read IGPs).
>   
I actually have in my possession PCI devices that do just that. The 
second half of VRAM can always be found as the "Display controller", 
function #1, where the main VGA device is function #0. A PCI Radeon card 
I have does that, the Intel graphics chipset in my laptop does that... 
vramfs is able to mount that VRAM and use it just fine.

Use lspci -v and look for your VGA card, then at the "Display 
controller" part (PCI class 0x038000), you'll see where that other half is.

And VRAM isn't always slow. It usually is slower than host memory, yes, 
but not slow enough to be useless.
> It doesn't in any way serve as a basis for talking to the GPU behind
> said VRAM or as a basis for any
> sort of backend to talk to the GPU.
>   
It's not supposed to. By design. Someone else runs the GPU. A root-level 
userspace daemon controlling the MMIO registers directly through 
/dev/mem if you like. This might be useful for the XFree86 or X.org guys 
for example.
> Dave.
>
>   
>> My other concern is that GEM with a filesystem might be the best option for
>> 3D gaming, but that it wouldn't work if the driver doesn't know the card.
>> The DRI drivers, as far as I know, are tied to the GPU and chipset of the
>> device (because they have to manage it, after all!). How exactly would GEM
>> work for cards that it doesn't recognize, like one machine of mine with a
>> weird ATI chipset nobody knows how to talk to? If GEM doesn't recognize it,
>> it won't provide VRAM resources to use it, right?
>>
>> This is where vramfs has it's advantage: it's not the absolute best solution
>> for 3D graphics, but it's simple, it can serve as a starting point for GPU
>> experiments from userspace, or if nothing else allows the use of the onboard
>> video RAM on an otherwise unused and unrecognized video device. It's
>> device-agnostic by design.
>>     
>>> The way you want do that is using OpenGL to put your data in textures
>>> and framebuffer objects, and render them.  With KMS, we'll be able to
>>> support EGL even on the console so you can do the work without having an
>>> X environment set up.
>>>
>>> The problem with vramfs as a basis for GPU offload is that most GPU
>>> tasks end up at some point exceeding the size of available
>>> aperture/VRAM.  So you need code that manages loading buffer objects in
>>> and out on demand, managing the execution pipeline and GPU and CPU
>>> caches as required.  We have that with GEM already.
>>>
>>> Remember, writing data to an aperture isn't the hard part of offloading
>>> to the GPU, programming the GPU is.  That's why you use OpenGL or
>>> another abstraction to do it.
>>>
>>>
>>>       
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> Please read the FAQ at  http://www.tux.org/lkml/
>>
>>     
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>
>
>   

Re: Vramfs: filesystem driver to utilize extra RAM on VGA devices

[Dave Airlie]

On Wed, Jan 28, 2009 at 7:03 PM, Jonathan Campbell <jon@nerdgrounds.com> wrote:
> I can't think of a nice way to say this, but this idea is crack.
>
> Hey, thanks :)
>>
>> I can see one use for vramfs which is to use VRAM as a file backing
>> store, we won't mention that VRAM is slow. Some caveats being
>> the PCI aperture only exposes 1/2 the VRAM on some cards, there is
>> more hidden away where the CPU can't find it.
>>  and that some cards have made up VRAM (read IGPs).
>>
>
> I actually have in my possession PCI devices that do just that. The second
> half of VRAM can always be found as the "Display controller", function #1,
> where the main VGA device is function #0. A PCI Radeon card I have does
> that, the Intel graphics chipset in my laptop does that... vramfs is able to
> mount that VRAM and use it just fine.

one swallow does not a summer make.

I have lots of other gpus that don't do this sitting on my desk. some GPUs
don't expose a second head. Intel GPU in laptops don't actually have any real
VRAM behind them, they just have some small amount allocated by the BIOS.

>
> Use lspci -v and look for your VGA card, then at the "Display controller"
> part (PCI class 0x038000), you'll see where that other half is.
>
> And VRAM isn't always slow. It usually is slower than host memory, yes, but
> not slow enough to be useless.
>>
>> It doesn't in any way serve as a basis for talking to the GPU behind
>> said VRAM or as a basis for any
>> sort of backend to talk to the GPU.
>>
>
> It's not supposed to. By design. Someone else runs the GPU. A root-level
> userspace daemon controlling the MMIO registers directly through /dev/mem if
> you like. This might be useful for the XFree86 or X.org guys for example.

I am along with Eric one of the X.org people, hence my crack comment. We are
also trying to kill userspace drivers that control the MMIO regs.

TBH if you wanted to make this faster, you really would need some sort
of CPU offload
using the GPU blit engine which is a lot faster than doing CPU access
to the uncached
VRAM.


Dave.

>>
>> Dave.
>>
>>
>>>
>>> My other concern is that GEM with a filesystem might be the best option
>>> for
>>> 3D gaming, but that it wouldn't work if the driver doesn't know the card.
>>> The DRI drivers, as far as I know, are tied to the GPU and chipset of the
>>> device (because they have to manage it, after all!). How exactly would
>>> GEM
>>> work for cards that it doesn't recognize, like one machine of mine with a
>>> weird ATI chipset nobody knows how to talk to? If GEM doesn't recognize
>>> it,
>>> it won't provide VRAM resources to use it, right?
>>>
>>> This is where vramfs has it's advantage: it's not the absolute best
>>> solution
>>> for 3D graphics, but it's simple, it can serve as a starting point for
>>> GPU
>>> experiments from userspace, or if nothing else allows the use of the
>>> onboard
>>> video RAM on an otherwise unused and unrecognized video device. It's
>>> device-agnostic by design.
>>>
>>>>
>>>> The way you want do that is using OpenGL to put your data in textures
>>>> and framebuffer objects, and render them.  With KMS, we'll be able to
>>>> support EGL even on the console so you can do the work without having an
>>>> X environment set up.
>>>>
>>>> The problem with vramfs as a basis for GPU offload is that most GPU
>>>> tasks end up at some point exceeding the size of available
>>>> aperture/VRAM.  So you need code that manages loading buffer objects in
>>>> and out on demand, managing the execution pipeline and GPU and CPU
>>>> caches as required.  We have that with GEM already.
>>>>
>>>> Remember, writing data to an aperture isn't the hard part of offloading
>>>> to the GPU, programming the GPU is.  That's why you use OpenGL or
>>>> another abstraction to do it.
>>>>
>>>>
>>>>
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-kernel"
>>> in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>> Please read the FAQ at  http://www.tux.org/lkml/
>>>
>>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> Please read the FAQ at  http://www.tux.org/lkml/
>>
>>
>>
>
>

Re: Vramfs: filesystem driver to utilize extra RAM on VGA devices

[Jonathan Campbell]

> Can the GPU use the data placed in your file system? 
Assuming the GPU can access any part of VRAM, yes. Files created in 
vramfs will always have content that exists somewhere in video ram. A 
file you create never moves, and is always contiguous in memory. All 
your program needs to direct the GPU to it is the block offset from the 
start of VRAM, which can be obtained by an ioctl() or bmap().

I thought the best possible design would be for any process to create a 
file in vramfs, and then direct the attention of whoever's managing the 
GPU to that file (perhaps a user-space daemon), who could open the file 
and use bmap or an ioctl to locate it and direct the GPU to operate on it.

I'd also like to point out the filesystem structures themselves are 
never placed in VRAM on purpose. I'd hate for files to suddenly 
disappear from the filesystem because of an errant GPU bug or pixel 
shader gone amok; the worst that can happen by doing it that way is that 
a bunch of files go blank or get filled with garbage and no harm done.
> Do you have strong control as to exactly how the data is mapped into VRAM? 
Not exactly. When you create a file you get whatever free space is 
available. However, vramfs does guarantee that your file is never 
fragmented, never sparse, and will always exist for the life of the file 
from it's offset in video ram to the offset plus the file size. I 
believe I've written the code to be flexible enough however to allow 
stronger control if needed.
> I'm thinking about parallel processing  - Linux puts data there and then
> the GPU works on it to produce a result which Linux can eventually
> fetch.
>   
Or multimedia uses too. I'd like to see this used by MPlayer or Xine 
someday for example to make use of the MPEG-2 or H.264 hardware GPU 
decoding that today's graphics cards support: Just feed in the H.264 in 
one file and direct the GPU to spit out decoded video to another file, 
which MPlayer would have memory-mapped and could directly do something 
with it, -vf filters and all.
> - Mark
>
>
>   

Re: Vramfs: filesystem driver to utilize extra RAM on VGA devices

[Mark Knecht]

On Mon, Jan 26, 2009 at 9:07 PM, Jonathan Campbell <jon@nerdgrounds.com> wrote:
>
>> Can the GPU use the data placed in your file system?
>
> Assuming the GPU can access any part of VRAM, yes. Files created in vramfs
> will always have content that exists somewhere in video ram. A file you
> create never moves, and is always contiguous in memory. All your program
> needs to direct the GPU to it is the block offset from the start of VRAM,
> which can be obtained by an ioctl() or bmap().
>
> I thought the best possible design would be for any process to create a file
> in vramfs, and then direct the attention of whoever's managing the GPU to
> that file (perhaps a user-space daemon), who could open the file and use
> bmap or an ioctl to locate it and direct the GPU to operate on it.
>

Yep, as I said in the response to Eric a number of us had talked in
the past about using the GPU for some audio operations that are just
too CPU intensive - reverb, multi-band compressors, etc. See Jamin as
an example of a great app that brings most DAW type boxes to their
knees.

http://jamin.sourceforge.net/en/about.html

> I'd also like to point out the filesystem structures themselves are never
> placed in VRAM on purpose. I'd hate for files to suddenly disappear from the
> filesystem because of an errant GPU bug or pixel shader gone amok; the worst
> that can happen by doing it that way is that a bunch of files go blank or
> get filled with garbage and no harm done.
>>
>> Do you have strong control as to exactly how the data is mapped into VRAM?
>
> Not exactly. When you create a file you get whatever free space is
> available. However, vramfs does guarantee that your file is never
> fragmented, never sparse, and will always exist for the life of the file
> from it's offset in video ram to the offset plus the file size. I believe
> I've written the code to be flexible enough however to allow stronger
> control if needed.

So what happens to the file system if the user changes screen
resolution, color depth, etc.? There must be features to manage stuff
like that?

- Mark

Re: Vramfs: filesystem driver to utilize extra RAM on VGA devices

[Mark Knecht]

On Tue, Jan 27, 2009 at 11:29 AM, Jonathan Campbell <jon@nerdgrounds.com> wrote:
>
> On Mon, Jan 26, 2009 at 9:07 PM, Jonathan Campbell <jon@nerdgrounds.com>
> wrote:
>
>
> Can the GPU use the data placed in your file system?
>
>
> Assuming the GPU can access any part of VRAM, yes. Files created in vramfs
> will always have content that exists somewhere in video ram. A file you
> create never moves, and is always contiguous in memory. All your program
> needs to direct the GPU to it is the block offset from the start of VRAM,
> which can be obtained by an ioctl() or bmap().
>
> I thought the best possible design would be for any process to create a file
> in vramfs, and then direct the attention of whoever's managing the GPU to
> that file (perhaps a user-space daemon), who could open the file and use
> bmap or an ioctl to locate it and direct the GPU to operate on it.
>
>
>
> Yep, as I said in the response to Eric a number of us had talked in
> the past about using the GPU for some audio operations that are just
> too CPU intensive - reverb, multi-band compressors, etc. See Jamin as
> an example of a great app that brings most DAW type boxes to their
> knees.
>
> http://jamin.sourceforge.net/en/about.html
>
>
>
> I'd also like to point out the filesystem structures themselves are never
> placed in VRAM on purpose. I'd hate for files to suddenly disappear from the
> filesystem because of an errant GPU bug or pixel shader gone amok; the worst
> that can happen by doing it that way is that a bunch of files go blank or
> get filled with garbage and no harm done.
>
>
> Do you have strong control as to exactly how the data is mapped into VRAM?
>
>
> Not exactly. When you create a file you get whatever free space is
> available. However, vramfs does guarantee that your file is never
> fragmented, never sparse, and will always exist for the life of the file
> from it's offset in video ram to the offset plus the file size. I believe
> I've written the code to be flexible enough however to allow stronger
> control if needed.
>
>
> So what happens to the file system if the user changes screen
> resolution, color depth, etc.? There must be features to manage stuff
> like that?
>
>
> That depends on the framebuffer driver. Some drivers like radeonfb don't
> clear the VRAM on mode change, so they remain intact until stomped on by
> fbcon or anyone else using /dev/fb0. However, vramfs creates a default
> "framebuffer" file to occupy that region of VRAM for this reason, so as long
> as the default file is large enough to cover any possible framebuffer you
> could set up, and nobody reads/writes /dev/fb0 that far, the files will be
> fine. Vramfs doesn't really need to interact much with the framebuffer
> system.
>
> Again, filesystem structures are never stored in VRAM for this reason: so
> that in the worst case scenario files are overwritten with bitmaps, garbage,
> or cleared. No additional harm done. I wouldn't want the kernel oopses or
> hangs associated with storing kernel structures there.
>
> I suppose vramfs could ask the kernel which framebuffer is attached to the
> PCI device, and ask it the highest possible resolution and bits per pixel
> count (not necessarily supported by the monitor, I mean highest possible
> supported by the CRTC and DACs)

I think it unlikely that any audio engineer is going to purposely
change screen resolutions while doing a mix so I'd not want you to
complicate things for situations like that.

What would happen a lot I think is that while X is running someone
might drop into the console to do something or switch to another
desktop in Gnome or KDE. I suspect those aren't issues but it would be
important not to corrupt any file system data when those happen. For
instance, someone might run a GPU program on one desktop and then move
to their main desktop for mixing, etc. None of that effects the size
of framebuffers as far as I know.

If I was to build this and try it our (not saying I have time right
now though) then as root how do I determine what size file system I
can build? Do I run fdisk against /dev/fb0 and look at what portion is
unallocated?

Thanks,
Mark

Re: Vramfs: filesystem driver to utilize extra RAM on VGA devices

[Jonathan Campbell]

> If I was to build this and try it our (not saying I have time right
> now though) then as root how do I determine what size file system I
> can build? Do I run fdisk against /dev/fb0 and look at what portion is
> unallocated?
>
>   
The capacity is automatically set to the amount of VRAM out there, based 
on what the PCI subsystem says the resource length is. So if you mount 
your ATI Radeon's VRAM and it has 512MB on it, you'll get a filesystem 
with a capacity of 512MB.

Fdisk is not necessary, the filesystem structures are maintained in 
system memory not VRAM. Like tmpfs/shmemfs the filesystem structures are 
discarded on unmount, though the contents are not cleared from VRAM.

Adding a mount parameter to override the PCI system reported length 
wouldn't be hard either.

Re: Vramfs: filesystem driver to utilize extra RAM on VGA devices

[Trent Piepho]

On Mon, 26 Jan 2009, Jonathan Campbell wrote:
> one, that today's graphics cards have relatively obscene amounts of RAM on 
> them even if you're not using it. If you're running it as a server and not 
> using it for 3D graphics, why not mount the VRAM on the graphics card as a 
> filesystem and store things there to get some extra space?

How fast is it?  For instance, what are your bonnie++ scores for vramfs vs
tmpfs vs something on disk?

It seems like you could at the very least put a swap file on it that's
faster than one on disk.