Re: [PATCH 2/2] vfio/pci: Remove console drivers

Re: [PATCH 2/2] vfio/pci: Remove console drivers

[mb]

Hi,

I hope it is ok to reply to this old thread. Unfortunately, I found a
problem only now after upgrading to 6.0.

My setup has multiple GPUs (2), and I depend on EFIFB to have a working console.
pre-patch behavior, when I bind the vfio-pci to my secondary GPU both
the passthrough and the EFIFB keep working fine.
post-patch behavior, when I bind the vfio-pci to the secondary GPU,
the EFIFB disappears from the system, binding the console to the
"dummy console".
Whenever you try to access the terminal, you have the screen stuck in
whatever was the last buffer content, which gives the impression of
"freezing," but I can still type.
Everything else works, including the passthrough.

I can only think about a few options:

- Is there a way to have EFIFB show up again? After all it looks like
the kernel has just abandoned it, but the buffer is still there. I
can't find a single message about the secondary card and EFIFB in
dmesg, but there's a message for the primary card and EFIFB.
- Can we have a boolean controlling the behavior of vfio-pci
altogether or at least controlling the behavior of vfio-pci for that
specific ID? I know there's already some option for vfio-pci and VGA
cards, would it be appropriate to attach this behavior to that option?

Thanks,

Carlos Augusto

Re: [PATCH 2/2] vfio/pci: Remove console drivers

[mb]

Hi,

I hope it is ok to reply to this old thread. Unfortunately, I found a
problem only now after upgrading to 6.0.

My setup has multiple GPUs (2), and I depend on EFIFB to have a working console.
pre-patch behavior, when I bind the vfio-pci to my secondary GPU both
the passthrough and the EFIFB keep working fine.
post-patch behavior, when I bind the vfio-pci to the secondary GPU,
the EFIFB disappears from the system, binding the console to the
"dummy console".
Whenever you try to access the terminal, you have the screen stuck in
whatever was the last buffer content, which gives the impression of
"freezing," but I can still type.
Everything else works, including the passthrough.

I can only think about a few options:

- Is there a way to have EFIFB show up again? After all it looks like
the kernel has just abandoned it, but the buffer is still there. I
can't find a single message about the secondary card and EFIFB in
dmesg, but there's a message for the primary card and EFIFB.
- Can we have a boolean controlling the behavior of vfio-pci
altogether or at least controlling the behavior of vfio-pci for that
specific ID? I know there's already some option for vfio-pci and VGA
cards, would it be appropriate to attach this behavior to that option?

Thanks,

Carlos Augusto

Re: [PATCH 2/2] vfio/pci: Remove console drivers

[Alex Williamson]

On Sat, 3 Dec 2022 17:12:38 -0700
"mb@lab.how" <mb@lab.how> wrote:

> Hi,
> 
> I hope it is ok to reply to this old thread.

It is, but the only relic of the thread is the subject.  For reference,
the latest version of this posted is here:

https://lore.kernel.org/all/20220622140134.12763-4-tzimmermann@suse.de/

Which is committed as:

d17378062079 ("vfio/pci: Remove console drivers")

> Unfortunately, I found a
> problem only now after upgrading to 6.0.
> 
> My setup has multiple GPUs (2), and I depend on EFIFB to have a working console.
> pre-patch behavior, when I bind the vfio-pci to my secondary GPU both
> the passthrough and the EFIFB keep working fine.
> post-patch behavior, when I bind the vfio-pci to the secondary GPU,
> the EFIFB disappears from the system, binding the console to the
> "dummy console".
> Whenever you try to access the terminal, you have the screen stuck in
> whatever was the last buffer content, which gives the impression of
> "freezing," but I can still type.
> Everything else works, including the passthrough.

This sounds like the call to aperture_remove_conflicting_pci_devices()
is removing the conflicting driver itself rather than removing the
device from the driver.  Is it not possible to unbind the GPU from
efifb before binding the GPU to vfio-pci to effectively nullify the
added call?
 
> I can only think about a few options:
> 
> - Is there a way to have EFIFB show up again? After all it looks like
> the kernel has just abandoned it, but the buffer is still there. I
> can't find a single message about the secondary card and EFIFB in
> dmesg, but there's a message for the primary card and EFIFB.
> - Can we have a boolean controlling the behavior of vfio-pci
> altogether or at least controlling the behavior of vfio-pci for that
> specific ID? I know there's already some option for vfio-pci and VGA
> cards, would it be appropriate to attach this behavior to that option?

I suppose we could have an opt-out module option on vfio-pci to skip
the above call, but clearly it would be better if things worked by
default.  We cannot make full use of GPUs with vfio-pci if they're
still in use by host console drivers.  The intention was certainly to
unbind the device from any low level drivers rather than disable use of
a console driver entirely.  DRM/GPU folks, is that possibly an
interface we could implement?  Thanks,

Alex

Re: [PATCH 2/2] vfio/pci: Remove console drivers

[Alex Williamson]

On Sat, 3 Dec 2022 17:12:38 -0700
"mb@lab.how" <mb@lab.how> wrote:

> Hi,
> 
> I hope it is ok to reply to this old thread.

It is, but the only relic of the thread is the subject.  For reference,
the latest version of this posted is here:

https://lore.kernel.org/all/20220622140134.12763-4-tzimmermann@suse.de/

Which is committed as:

d17378062079 ("vfio/pci: Remove console drivers")

> Unfortunately, I found a
> problem only now after upgrading to 6.0.
> 
> My setup has multiple GPUs (2), and I depend on EFIFB to have a working console.
> pre-patch behavior, when I bind the vfio-pci to my secondary GPU both
> the passthrough and the EFIFB keep working fine.
> post-patch behavior, when I bind the vfio-pci to the secondary GPU,
> the EFIFB disappears from the system, binding the console to the
> "dummy console".
> Whenever you try to access the terminal, you have the screen stuck in
> whatever was the last buffer content, which gives the impression of
> "freezing," but I can still type.
> Everything else works, including the passthrough.

This sounds like the call to aperture_remove_conflicting_pci_devices()
is removing the conflicting driver itself rather than removing the
device from the driver.  Is it not possible to unbind the GPU from
efifb before binding the GPU to vfio-pci to effectively nullify the
added call?
 
> I can only think about a few options:
> 
> - Is there a way to have EFIFB show up again? After all it looks like
> the kernel has just abandoned it, but the buffer is still there. I
> can't find a single message about the secondary card and EFIFB in
> dmesg, but there's a message for the primary card and EFIFB.
> - Can we have a boolean controlling the behavior of vfio-pci
> altogether or at least controlling the behavior of vfio-pci for that
> specific ID? I know there's already some option for vfio-pci and VGA
> cards, would it be appropriate to attach this behavior to that option?

I suppose we could have an opt-out module option on vfio-pci to skip
the above call, but clearly it would be better if things worked by
default.  We cannot make full use of GPUs with vfio-pci if they're
still in use by host console drivers.  The intention was certainly to
unbind the device from any low level drivers rather than disable use of
a console driver entirely.  DRM/GPU folks, is that possibly an
interface we could implement?  Thanks,

Alex

Re: [PATCH 2/2] vfio/pci: Remove console drivers

[Thomas Zimmermann]

[-- Attachment #1.1: Type: text/plain, Size: 3118 bytes --]

Hi

Am 05.12.22 um 01:51 schrieb Alex Williamson:
> On Sat, 3 Dec 2022 17:12:38 -0700
> "mb@lab.how" <mb@lab.how> wrote:
> 
>> Hi,
>>
>> I hope it is ok to reply to this old thread.
> 
> It is, but the only relic of the thread is the subject.  For reference,
> the latest version of this posted is here:
> 
> https://lore.kernel.org/all/20220622140134.12763-4-tzimmermann@suse.de/
> 
> Which is committed as:
> 
> d17378062079 ("vfio/pci: Remove console drivers")
> 
>> Unfortunately, I found a
>> problem only now after upgrading to 6.0.
>>
>> My setup has multiple GPUs (2), and I depend on EFIFB to have a working console.

Which GPUs do you have?

>> pre-patch behavior, when I bind the vfio-pci to my secondary GPU both
>> the passthrough and the EFIFB keep working fine.
>> post-patch behavior, when I bind the vfio-pci to the secondary GPU,
>> the EFIFB disappears from the system, binding the console to the
>> "dummy console".

The efifb would likely use the first GPU. And vfio-pci should only 
remove the generic driver from the second device. Are you sure that 
you're not somehow using the first GPU with vfio-pci.

>> Whenever you try to access the terminal, you have the screen stuck in
>> whatever was the last buffer content, which gives the impression of
>> "freezing," but I can still type.
>> Everything else works, including the passthrough.
> 
> This sounds like the call to aperture_remove_conflicting_pci_devices()
> is removing the conflicting driver itself rather than removing the
> device from the driver.  Is it not possible to unbind the GPU from
> efifb before binding the GPU to vfio-pci to effectively nullify the
> added call?
>   
>> I can only think about a few options:
>>
>> - Is there a way to have EFIFB show up again? After all it looks like
>> the kernel has just abandoned it, but the buffer is still there. I
>> can't find a single message about the secondary card and EFIFB in
>> dmesg, but there's a message for the primary card and EFIFB.
>> - Can we have a boolean controlling the behavior of vfio-pci
>> altogether or at least controlling the behavior of vfio-pci for that
>> specific ID? I know there's already some option for vfio-pci and VGA
>> cards, would it be appropriate to attach this behavior to that option?
> 
> I suppose we could have an opt-out module option on vfio-pci to skip
> the above call, but clearly it would be better if things worked by
> default.  We cannot make full use of GPUs with vfio-pci if they're
> still in use by host console drivers.  The intention was certainly to
> unbind the device from any low level drivers rather than disable use of
> a console driver entirely.  DRM/GPU folks, is that possibly an
> interface we could implement?  Thanks,

When vfio-pci gives the GPU device to the guest, which driver driver is 
bound to it?

Best regards
Thomas

> 
> Alex
> 

-- 
Thomas Zimmermann
Graphics Driver Developer
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5, 90409 Nürnberg, Germany
(HRB 36809, AG Nürnberg)
Geschäftsführer: Ivo Totev

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]

Re: [PATCH 2/2] vfio/pci: Remove console drivers

[mb]

[-- Attachment #1: Type: text/plain, Size: 4375 bytes --]

I have a rtx 3070 and a 3090, I am absolutely sure I am binding vfio-pci to
the 3090 and not the 3070.

I have bound the driver in two different ways, first by passing the IDs to
the module and alternatively by manipulating the system interface and use
the override (this is what I originally had to do when I used two 1080s, so
I know it works).

While the 3090 doesn't show a console, there's a remnant from the refund
(and grub previously) there.

The assessment Alex made previously, where
aperture_remove_conflicting_pci_devices() is removing the driver (EFIFB)
instead of the device seems correct, but it could also can be a quirky of
how EFIFB is implemented. I recall reading a long time ago that EFIFB is a
special device and once it detects changes it would simply give up. There
was also no way to attach a device to it again as it depends on being
preloaded outside the kernel; once something takes over the buffer
reinitializing is "impossible". I never went deeper to try and understand
it.


On Mon, Dec 5, 2022, 2:00 AM Thomas Zimmermann <tzimmermann@suse.de> wrote:

> Hi
>
> Am 05.12.22 um 01:51 schrieb Alex Williamson:
> > On Sat, 3 Dec 2022 17:12:38 -0700
> > "mb@lab.how" <mb@lab.how> wrote:
> >
> >> Hi,
> >>
> >> I hope it is ok to reply to this old thread.
> >
> > It is, but the only relic of the thread is the subject.  For reference,
> > the latest version of this posted is here:
> >
> > https://lore.kernel.org/all/20220622140134.12763-4-tzimmermann@suse.de/
> >
> > Which is committed as:
> >
> > d17378062079 ("vfio/pci: Remove console drivers")
> >
> >> Unfortunately, I found a
> >> problem only now after upgrading to 6.0.
> >>
> >> My setup has multiple GPUs (2), and I depend on EFIFB to have a working
> console.
>
> Which GPUs do you have?
>
> >> pre-patch behavior, when I bind the vfio-pci to my secondary GPU both
> >> the passthrough and the EFIFB keep working fine.
> >> post-patch behavior, when I bind the vfio-pci to the secondary GPU,
> >> the EFIFB disappears from the system, binding the console to the
> >> "dummy console".
>
> The efifb would likely use the first GPU. And vfio-pci should only
> remove the generic driver from the second device. Are you sure that
> you're not somehow using the first GPU with vfio-pci.
>
> >> Whenever you try to access the terminal, you have the screen stuck in
> >> whatever was the last buffer content, which gives the impression of
> >> "freezing," but I can still type.
> >> Everything else works, including the passthrough.
> >
> > This sounds like the call to aperture_remove_conflicting_pci_devices()
> > is removing the conflicting driver itself rather than removing the
> > device from the driver.  Is it not possible to unbind the GPU from
> > efifb before binding the GPU to vfio-pci to effectively nullify the
> > added call?
> >
> >> I can only think about a few options:
> >>
> >> - Is there a way to have EFIFB show up again? After all it looks like
> >> the kernel has just abandoned it, but the buffer is still there. I
> >> can't find a single message about the secondary card and EFIFB in
> >> dmesg, but there's a message for the primary card and EFIFB.
> >> - Can we have a boolean controlling the behavior of vfio-pci
> >> altogether or at least controlling the behavior of vfio-pci for that
> >> specific ID? I know there's already some option for vfio-pci and VGA
> >> cards, would it be appropriate to attach this behavior to that option?
> >
> > I suppose we could have an opt-out module option on vfio-pci to skip
> > the above call, but clearly it would be better if things worked by
> > default.  We cannot make full use of GPUs with vfio-pci if they're
> > still in use by host console drivers.  The intention was certainly to
> > unbind the device from any low level drivers rather than disable use of
> > a console driver entirely.  DRM/GPU folks, is that possibly an
> > interface we could implement?  Thanks,
>
> When vfio-pci gives the GPU device to the guest, which driver driver is
> bound to it?
>
> Best regards
> Thomas
>
> >
> > Alex
> >
>
> --
> Thomas Zimmermann
> Graphics Driver Developer
> SUSE Software Solutions Germany GmbH
> Maxfeldstr. 5, 90409 Nürnberg, Germany
> (HRB 36809, AG Nürnberg)
> Geschäftsführer: Ivo Totev
>
>

[-- Attachment #2: Type: text/html, Size: 5426 bytes --]

Re: [PATCH 2/2] vfio/pci: Remove console drivers

[Thomas Zimmermann]

[-- Attachment #1.1: Type: text/plain, Size: 5583 bytes --]

Hi

Am 05.12.22 um 10:32 schrieb mb@lab.how:
> I have a rtx 3070 and a 3090, I am absolutely sure I am binding vfio-pci 
> to the 3090 and not the 3070.
> 
> I have bound the driver in two different ways, first by passing the IDs 
> to the module and alternatively by manipulating the system interface and 
> use the override (this is what I originally had to do when I used two 
> 1080s, so I know it works).
> 
> While the 3090 doesn't show a console, there's a remnant from the refund 
> (and grub previously) there.
> 
> The assessment Alex made previously, where 
> aperture_remove_conflicting_pci_devices() is removing the driver (EFIFB) 
> instead of the device seems correct, but it could also can be a quirky 
> of how EFIFB is implemented. I recall reading a long time ago that EFIFB 
> is a special device and once it detects changes it would simply give up. 
> There was also no way to attach a device to it again as it depends on 
> being preloaded outside the kernel; once something takes over the buffer 
> reinitializing is "impossible". I never went deeper to try and 
> understand it.

We recently reworked fbdev's interaction with the aperture helpers. [1] 
All devices should now be removed iff the driver has been bound to it 
(which should be the case here) The patches went into an v6.1-rc.

Could you try the most recent v6.1-rc and report if this fixes the problem?

Best regards
Thomas

[1] https://patchwork.freedesktop.org/series/106040/

> 
> 
> On Mon, Dec 5, 2022, 2:00 AM Thomas Zimmermann <tzimmermann@suse.de 
> <mailto:tzimmermann@suse.de>> wrote:
> 
>     Hi
> 
>     Am 05.12.22 um 01:51 schrieb Alex Williamson:
>      > On Sat, 3 Dec 2022 17:12:38 -0700
>      > "mb@lab.how" <mb@lab.how> wrote:
>      >
>      >> Hi,
>      >>
>      >> I hope it is ok to reply to this old thread.
>      >
>      > It is, but the only relic of the thread is the subject.  For
>     reference,
>      > the latest version of this posted is here:
>      >
>      >
>     https://lore.kernel.org/all/20220622140134.12763-4-tzimmermann@suse.de/ <https://lore.kernel.org/all/20220622140134.12763-4-tzimmermann@suse.de/>
>      >
>      > Which is committed as:
>      >
>      > d17378062079 ("vfio/pci: Remove console drivers")
>      >
>      >> Unfortunately, I found a
>      >> problem only now after upgrading to 6.0.
>      >>
>      >> My setup has multiple GPUs (2), and I depend on EFIFB to have a
>     working console.
> 
>     Which GPUs do you have?
> 
>      >> pre-patch behavior, when I bind the vfio-pci to my secondary GPU
>     both
>      >> the passthrough and the EFIFB keep working fine.
>      >> post-patch behavior, when I bind the vfio-pci to the secondary GPU,
>      >> the EFIFB disappears from the system, binding the console to the
>      >> "dummy console".
> 
>     The efifb would likely use the first GPU. And vfio-pci should only
>     remove the generic driver from the second device. Are you sure that
>     you're not somehow using the first GPU with vfio-pci.
> 
>      >> Whenever you try to access the terminal, you have the screen
>     stuck in
>      >> whatever was the last buffer content, which gives the impression of
>      >> "freezing," but I can still type.
>      >> Everything else works, including the passthrough.
>      >
>      > This sounds like the call to
>     aperture_remove_conflicting_pci_devices()
>      > is removing the conflicting driver itself rather than removing the
>      > device from the driver.  Is it not possible to unbind the GPU from
>      > efifb before binding the GPU to vfio-pci to effectively nullify the
>      > added call?
>      >
>      >> I can only think about a few options:
>      >>
>      >> - Is there a way to have EFIFB show up again? After all it looks
>     like
>      >> the kernel has just abandoned it, but the buffer is still there. I
>      >> can't find a single message about the secondary card and EFIFB in
>      >> dmesg, but there's a message for the primary card and EFIFB.
>      >> - Can we have a boolean controlling the behavior of vfio-pci
>      >> altogether or at least controlling the behavior of vfio-pci for that
>      >> specific ID? I know there's already some option for vfio-pci and VGA
>      >> cards, would it be appropriate to attach this behavior to that
>     option?
>      >
>      > I suppose we could have an opt-out module option on vfio-pci to skip
>      > the above call, but clearly it would be better if things worked by
>      > default.  We cannot make full use of GPUs with vfio-pci if they're
>      > still in use by host console drivers.  The intention was certainly to
>      > unbind the device from any low level drivers rather than disable
>     use of
>      > a console driver entirely.  DRM/GPU folks, is that possibly an
>      > interface we could implement?  Thanks,
> 
>     When vfio-pci gives the GPU device to the guest, which driver driver is
>     bound to it?
> 
>     Best regards
>     Thomas
> 
>      >
>      > Alex
>      >
> 
>     -- 
>     Thomas Zimmermann
>     Graphics Driver Developer
>     SUSE Software Solutions Germany GmbH
>     Maxfeldstr. 5, 90409 Nürnberg, Germany
>     (HRB 36809, AG Nürnberg)
>     Geschäftsführer: Ivo Totev
> 

-- 
Thomas Zimmermann
Graphics Driver Developer
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5, 90409 Nürnberg, Germany
(HRB 36809, AG Nürnberg)
Geschäftsführer: Ivo Totev

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]

Re: [PATCH 2/2] vfio/pci: Remove console drivers

[mb]

Hi Thomas,

On Mon, Dec 5, 2022 at 3:11 AM Thomas Zimmermann <tzimmermann@suse.de> wrote:
>
> Hi
>
> Am 05.12.22 um 10:32 schrieb mb@lab.how:
> > I have a rtx 3070 and a 3090, I am absolutely sure I am binding vfio-pci
> > to the 3090 and not the 3070.
> >
> > I have bound the driver in two different ways, first by passing the IDs
> > to the module and alternatively by manipulating the system interface and
> > use the override (this is what I originally had to do when I used two
> > 1080s, so I know it works).
> >
> > While the 3090 doesn't show a console, there's a remnant from the refund
> > (and grub previously) there.
> >
> > The assessment Alex made previously, where
> > aperture_remove_conflicting_pci_devices() is removing the driver (EFIFB)
> > instead of the device seems correct, but it could also can be a quirky
> > of how EFIFB is implemented. I recall reading a long time ago that EFIFB
> > is a special device and once it detects changes it would simply give up.
> > There was also no way to attach a device to it again as it depends on
> > being preloaded outside the kernel; once something takes over the buffer
> > reinitializing is "impossible". I never went deeper to try and
> > understand it.
>
> We recently reworked fbdev's interaction with the aperture helpers. [1]
> All devices should now be removed iff the driver has been bound to it
> (which should be the case here) The patches went into an v6.1-rc.
>
> Could you try the most recent v6.1-rc and report if this fixes the problem?

I just tried the latest one, v6.1-rc8, and I can see all the commits
for the series you mentioned there.

The same freeze behavior happens when I load vfio-pci:

[ 6.525463] VFIO - User Level meta-driver version: 0.3
[ 6.528231] Console: switching to colour dummy device 320x90

--
Carlos

>
> Best regards
> Thomas
>
> [1] https://patchwork.freedesktop.org/series/106040/
>
> >
> >
> > On Mon, Dec 5, 2022, 2:00 AM Thomas Zimmermann <tzimmermann@suse.de
> > <mailto:tzimmermann@suse.de>> wrote:
> >
> >     Hi
> >
> >     Am 05.12.22 um 01:51 schrieb Alex Williamson:
> >      > On Sat, 3 Dec 2022 17:12:38 -0700
> >      > "mb@lab.how" <mb@lab.how> wrote:
> >      >
> >      >> Hi,
> >      >>
> >      >> I hope it is ok to reply to this old thread.
> >      >
> >      > It is, but the only relic of the thread is the subject.  For
> >     reference,
> >      > the latest version of this posted is here:
> >      >
> >      >
> >     https://lore.kernel.org/all/20220622140134.12763-4-tzimmermann@suse.de/ <https://lore.kernel.org/all/20220622140134.12763-4-tzimmermann@suse.de/>
> >      >
> >      > Which is committed as:
> >      >
> >      > d17378062079 ("vfio/pci: Remove console drivers")
> >      >
> >      >> Unfortunately, I found a
> >      >> problem only now after upgrading to 6.0.
> >      >>
> >      >> My setup has multiple GPUs (2), and I depend on EFIFB to have a
> >     working console.
> >
> >     Which GPUs do you have?
> >
> >      >> pre-patch behavior, when I bind the vfio-pci to my secondary GPU
> >     both
> >      >> the passthrough and the EFIFB keep working fine.
> >      >> post-patch behavior, when I bind the vfio-pci to the secondary GPU,
> >      >> the EFIFB disappears from the system, binding the console to the
> >      >> "dummy console".
> >
> >     The efifb would likely use the first GPU. And vfio-pci should only
> >     remove the generic driver from the second device. Are you sure that
> >     you're not somehow using the first GPU with vfio-pci.
> >
> >      >> Whenever you try to access the terminal, you have the screen
> >     stuck in
> >      >> whatever was the last buffer content, which gives the impression of
> >      >> "freezing," but I can still type.
> >      >> Everything else works, including the passthrough.
> >      >
> >      > This sounds like the call to
> >     aperture_remove_conflicting_pci_devices()
> >      > is removing the conflicting driver itself rather than removing the
> >      > device from the driver.  Is it not possible to unbind the GPU from
> >      > efifb before binding the GPU to vfio-pci to effectively nullify the
> >      > added call?
> >      >
> >      >> I can only think about a few options:
> >      >>
> >      >> - Is there a way to have EFIFB show up again? After all it looks
> >     like
> >      >> the kernel has just abandoned it, but the buffer is still there. I
> >      >> can't find a single message about the secondary card and EFIFB in
> >      >> dmesg, but there's a message for the primary card and EFIFB.
> >      >> - Can we have a boolean controlling the behavior of vfio-pci
> >      >> altogether or at least controlling the behavior of vfio-pci for that
> >      >> specific ID? I know there's already some option for vfio-pci and VGA
> >      >> cards, would it be appropriate to attach this behavior to that
> >     option?
> >      >
> >      > I suppose we could have an opt-out module option on vfio-pci to skip
> >      > the above call, but clearly it would be better if things worked by
> >      > default.  We cannot make full use of GPUs with vfio-pci if they're
> >      > still in use by host console drivers.  The intention was certainly to
> >      > unbind the device from any low level drivers rather than disable
> >     use of
> >      > a console driver entirely.  DRM/GPU folks, is that possibly an
> >      > interface we could implement?  Thanks,
> >
> >     When vfio-pci gives the GPU device to the guest, which driver driver is
> >     bound to it?
> >
> >     Best regards
> >     Thomas
> >
> >      >
> >      > Alex
> >      >
> >
> >     --
> >     Thomas Zimmermann
> >     Graphics Driver Developer
> >     SUSE Software Solutions Germany GmbH
> >     Maxfeldstr. 5, 90409 Nürnberg, Germany
> >     (HRB 36809, AG Nürnberg)
> >     Geschäftsführer: Ivo Totev
> >
>
> --
> Thomas Zimmermann
> Graphics Driver Developer
> SUSE Software Solutions Germany GmbH
> Maxfeldstr. 5, 90409 Nürnberg, Germany
> (HRB 36809, AG Nürnberg)
> Geschäftsführer: Ivo Totev

Re: [PATCH 2/2] vfio/pci: Remove console drivers

[Shawn Michaels]

Hi,

I just upgraded my system (after 7 months) and I also lost my framebuffer on boot.
At first, I thought that my computer was freezing on startup, but it turns out
that it is running fine (I can SSH to it and even startx remotely), only the
framebuffer stops working very early on boot:

	:: running early hook [udev]
	Starting systemd-udevd version 252.4-2-arch
	[3.400568] VFIO - User Level meta-driver version: 0.3
	*LOSING FRAMEBUFFER HERE*

Running startx from an SSH session starts my X server and display works again.

I have two identical GPU cards (nVidia GTX 1070). I have been using the method
from [1] for years in order to prevent the nvidia driver from grabbing my guest
GPU. As mb already pointed out, vfio_pci now kills the framebuffer of the host
GPU even though it is assigned to the guest GPU. I could isolate it to the
following line from [1]:

	echo "vfio-pci" > "$GPU/driver_override"

I also double checked and this is correctly written to the guest GPU, and not
to the host GPU. My kernel version is:

	Linux cc 6.1.1-arch1-1 #1 SMP PREEMPT_DYNAMIC Wed, 21 Dec 2022 22:27:55 +0000 x86_64 GNU/Linux

You can find people having the same issue in threads from [2] and [3]. There are
also some reports on the VFIO discord server.

This is a problem for people using an encrypted filesystem (password needs to be
typed blindly) or in case boot fails for some reason (and you cannot see console
logs). In my case, I switched from manually starting X with startx to using a
graphical login manager. Blacklisting the vfio_pci module by passing the following
kernel parameter brings the boot console back:

	module_blacklist=vfio_pci

Happy new year everyone

Shawn

[1] https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF#Passthrough_all_GPUs_but_the_boot_GPU
[2] https://bbs.archlinux.org/viewtopic.php?pid=2063423
[3] https://forum.level1techs.com/t/linux-kernel-6-seems-to-be-incompatible-with-the-vfio-pci-module-needed-for-pci-passthrough/190039/11

[PATCH 0/2] Improve vfio-pci primary GPU assignment behavior

[Alex Williamson]

Users attempting to enable vfio PCI device assignment with a GPU will
often block the default PCI driver from the device to avoid conflicts
with the device initialization or release path.  This means that
vfio-pci is sometimes the first PCI driver to bind to the device.  In 
the case of assigning the primary graphics device, low-level console
drivers may still generate resource conflicts.  Users often employ
kernel command line arguments to disable conflicting drivers or
perform unbinding in userspace to avoid this, but the actual solution
is often distribution/kernel config specific based on the included
drivers.

We can instead allow vfio-pci to copy the behavior of
drm_aperture_remove_conflicting_pci_framebuffers() in order to remove
these low-level drivers with conflicting resources.  vfio-pci is not
however a DRM driver, nor does vfio-pci depend on DRM config options,
thus we split out and export the necessary DRM apterture support and
mirror the framebuffer and VGA support.

I'd be happy to pull this series in through the vfio branch if
approved by the DRM maintainers.  Thanks,

Alex

---

Alex Williamson (2):
      drm/aperture: Split conflicting platform driver removal
      vfio/pci: Remove console drivers


 drivers/gpu/drm/drm_aperture.c   | 33 +++++++++++++++++++++++---------
 drivers/vfio/pci/vfio_pci_core.c | 17 ++++++++++++++++
 include/drm/drm_aperture.h       |  2 ++
 3 files changed, 43 insertions(+), 9 deletions(-)

[PATCH 2/2] vfio/pci: Remove console drivers

[Alex Williamson]

Console drivers can create conflicts with PCI resources resulting in
userspace getting mmap failures to memory BARs.  This is especially evident
when trying to re-use the system primary console for userspace drivers.
Attempt to remove all nature of conflicting drivers as part of our VGA
initialization.

Reported-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Suggested-by: Gerd Hoffmann <kraxel@redhat.com>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
---
 drivers/vfio/pci/vfio_pci_core.c |   17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index a0d69ddaf90d..e0cbcbc2aee1 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -13,6 +13,7 @@
 #include <linux/device.h>
 #include <linux/eventfd.h>
 #include <linux/file.h>
+#include <linux/fb.h>
 #include <linux/interrupt.h>
 #include <linux/iommu.h>
 #include <linux/module.h>
@@ -29,6 +30,8 @@
 
 #include <linux/vfio_pci_core.h>
 
+#include <drm/drm_aperture.h>
+
 #define DRIVER_AUTHOR   "Alex Williamson <alex.williamson@redhat.com>"
 #define DRIVER_DESC "core driver for VFIO based PCI devices"
 
@@ -1793,6 +1796,20 @@ static int vfio_pci_vga_init(struct vfio_pci_core_device *vdev)
 	if (!vfio_pci_is_vga(pdev))
 		return 0;
 
+#if IS_REACHABLE(CONFIG_DRM)
+	drm_aperture_detach_platform_drivers(pdev);
+#endif
+
+#if IS_REACHABLE(CONFIG_FB)
+	ret = remove_conflicting_pci_framebuffers(pdev, vdev->vdev.ops->name);
+	if (ret)
+		return ret;
+#endif
+
+	ret = vga_remove_vgacon(pdev);
+	if (ret)
+		return ret;
+
 	ret = vga_client_register(pdev, vfio_pci_set_decode);
 	if (ret)
 		return ret;

[PATCH 2/2] vfio/pci: Remove console drivers

[Alex Williamson]

Console drivers can create conflicts with PCI resources resulting in
userspace getting mmap failures to memory BARs.  This is especially evident
when trying to re-use the system primary console for userspace drivers.
Attempt to remove all nature of conflicting drivers as part of our VGA
initialization.

Reported-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Suggested-by: Gerd Hoffmann <kraxel@redhat.com>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
---
 drivers/vfio/pci/vfio_pci_core.c |   17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index a0d69ddaf90d..e0cbcbc2aee1 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -13,6 +13,7 @@
 #include <linux/device.h>
 #include <linux/eventfd.h>
 #include <linux/file.h>
+#include <linux/fb.h>
 #include <linux/interrupt.h>
 #include <linux/iommu.h>
 #include <linux/module.h>
@@ -29,6 +30,8 @@
 
 #include <linux/vfio_pci_core.h>
 
+#include <drm/drm_aperture.h>
+
 #define DRIVER_AUTHOR   "Alex Williamson <alex.williamson@redhat.com>"
 #define DRIVER_DESC "core driver for VFIO based PCI devices"
 
@@ -1793,6 +1796,20 @@ static int vfio_pci_vga_init(struct vfio_pci_core_device *vdev)
 	if (!vfio_pci_is_vga(pdev))
 		return 0;
 
+#if IS_REACHABLE(CONFIG_DRM)
+	drm_aperture_detach_platform_drivers(pdev);
+#endif
+
+#if IS_REACHABLE(CONFIG_FB)
+	ret = remove_conflicting_pci_framebuffers(pdev, vdev->vdev.ops->name);
+	if (ret)
+		return ret;
+#endif
+
+	ret = vga_remove_vgacon(pdev);
+	if (ret)
+		return ret;
+
 	ret = vga_client_register(pdev, vfio_pci_set_decode);
 	if (ret)
 		return ret;

Re: [PATCH 2/2] vfio/pci: Remove console drivers

[Thomas Zimmermann]

[-- Attachment #1.1: Type: text/plain, Size: 3433 bytes --]

Hi Alex

Am 06.06.22 um 19:53 schrieb Alex Williamson:
> Console drivers can create conflicts with PCI resources resulting in
> userspace getting mmap failures to memory BARs.  This is especially evident
> when trying to re-use the system primary console for userspace drivers.
> Attempt to remove all nature of conflicting drivers as part of our VGA
> initialization.

First a dumb question about your use case.  You want to assign a PCI 
graphics card to a virtual machine and need to remove the generic driver 
from the framebuffer?

> 
> Reported-by: Laszlo Ersek <lersek@redhat.com>
> Tested-by: Laszlo Ersek <lersek@redhat.com>
> Suggested-by: Gerd Hoffmann <kraxel@redhat.com>
> Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
> ---
>   drivers/vfio/pci/vfio_pci_core.c |   17 +++++++++++++++++
>   1 file changed, 17 insertions(+)
> 
> diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
> index a0d69ddaf90d..e0cbcbc2aee1 100644
> --- a/drivers/vfio/pci/vfio_pci_core.c
> +++ b/drivers/vfio/pci/vfio_pci_core.c
> @@ -13,6 +13,7 @@
>   #include <linux/device.h>
>   #include <linux/eventfd.h>
>   #include <linux/file.h>
> +#include <linux/fb.h>
>   #include <linux/interrupt.h>
>   #include <linux/iommu.h>
>   #include <linux/module.h>
> @@ -29,6 +30,8 @@
>   
>   #include <linux/vfio_pci_core.h>
>   
> +#include <drm/drm_aperture.h>
> +
>   #define DRIVER_AUTHOR   "Alex Williamson <alex.williamson@redhat.com>"
>   #define DRIVER_DESC "core driver for VFIO based PCI devices"
>   
> @@ -1793,6 +1796,20 @@ static int vfio_pci_vga_init(struct vfio_pci_core_device *vdev)
>   	if (!vfio_pci_is_vga(pdev))
>   		return 0;
>   
> +#if IS_REACHABLE(CONFIG_DRM)
> +	drm_aperture_detach_platform_drivers(pdev);
> +#endif
> +
> +#if IS_REACHABLE(CONFIG_FB)
> +	ret = remove_conflicting_pci_framebuffers(pdev, vdev->vdev.ops->name);
> +	if (ret)
> +		return ret;
> +#endif
> +
> +	ret = vga_remove_vgacon(pdev);
> +	if (ret)
> +		return ret;
> +

You shouldn't have to copy any of the implementation of the aperture 
helpers.

If you call drm_aperture_remove_conflicting_pci_framebuffers() it should 
work correctly. The only reason why it requires a DRM driver structure 
as second argument is for the driver's name. [1] And that name is only 
used for printing an info message. [2]

The plan forward would be to drop patch 1 entirely.

For patch 2, the most trivial workaround is to instanciate struct 
drm_driver here and set the name field to 'vdev->vdev.ops->name'. In the 
longer term, the aperture helpers will be moved out of DRM and into a 
more prominent location. That workaround will be cleaned up then.

Alternatively, drm_aperture_remove_conflicting_pci_framebuffers() could 
be changed to accept the name string as second argument, but that's 
quite a bit of churn within the DRM code.

Best regards
Thomas

[1] 
https://elixir.bootlin.com/linux/v5.18.2/source/drivers/gpu/drm/drm_aperture.c#L347
[2] 
https://elixir.bootlin.com/linux/v5.18.2/source/drivers/video/fbdev/core/fbmem.c#L1570


>   	ret = vga_client_register(pdev, vfio_pci_set_decode);
>   	if (ret)
>   		return ret;
> 
> 

-- 
Thomas Zimmermann
Graphics Driver Developer
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5, 90409 Nürnberg, Germany
(HRB 36809, AG Nürnberg)
Geschäftsführer: Ivo Totev

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]

Re: [PATCH 2/2] vfio/pci: Remove console drivers

[Thomas Zimmermann]

[-- Attachment #1.1: Type: text/plain, Size: 3433 bytes --]

Hi Alex

Am 06.06.22 um 19:53 schrieb Alex Williamson:
> Console drivers can create conflicts with PCI resources resulting in
> userspace getting mmap failures to memory BARs.  This is especially evident
> when trying to re-use the system primary console for userspace drivers.
> Attempt to remove all nature of conflicting drivers as part of our VGA
> initialization.

First a dumb question about your use case.  You want to assign a PCI 
graphics card to a virtual machine and need to remove the generic driver 
from the framebuffer?

> 
> Reported-by: Laszlo Ersek <lersek@redhat.com>
> Tested-by: Laszlo Ersek <lersek@redhat.com>
> Suggested-by: Gerd Hoffmann <kraxel@redhat.com>
> Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
> ---
>   drivers/vfio/pci/vfio_pci_core.c |   17 +++++++++++++++++
>   1 file changed, 17 insertions(+)
> 
> diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
> index a0d69ddaf90d..e0cbcbc2aee1 100644
> --- a/drivers/vfio/pci/vfio_pci_core.c
> +++ b/drivers/vfio/pci/vfio_pci_core.c
> @@ -13,6 +13,7 @@
>   #include <linux/device.h>
>   #include <linux/eventfd.h>
>   #include <linux/file.h>
> +#include <linux/fb.h>
>   #include <linux/interrupt.h>
>   #include <linux/iommu.h>
>   #include <linux/module.h>
> @@ -29,6 +30,8 @@
>   
>   #include <linux/vfio_pci_core.h>
>   
> +#include <drm/drm_aperture.h>
> +
>   #define DRIVER_AUTHOR   "Alex Williamson <alex.williamson@redhat.com>"
>   #define DRIVER_DESC "core driver for VFIO based PCI devices"
>   
> @@ -1793,6 +1796,20 @@ static int vfio_pci_vga_init(struct vfio_pci_core_device *vdev)
>   	if (!vfio_pci_is_vga(pdev))
>   		return 0;
>   
> +#if IS_REACHABLE(CONFIG_DRM)
> +	drm_aperture_detach_platform_drivers(pdev);
> +#endif
> +
> +#if IS_REACHABLE(CONFIG_FB)
> +	ret = remove_conflicting_pci_framebuffers(pdev, vdev->vdev.ops->name);
> +	if (ret)
> +		return ret;
> +#endif
> +
> +	ret = vga_remove_vgacon(pdev);
> +	if (ret)
> +		return ret;
> +

You shouldn't have to copy any of the implementation of the aperture 
helpers.

If you call drm_aperture_remove_conflicting_pci_framebuffers() it should 
work correctly. The only reason why it requires a DRM driver structure 
as second argument is for the driver's name. [1] And that name is only 
used for printing an info message. [2]

The plan forward would be to drop patch 1 entirely.

For patch 2, the most trivial workaround is to instanciate struct 
drm_driver here and set the name field to 'vdev->vdev.ops->name'. In the 
longer term, the aperture helpers will be moved out of DRM and into a 
more prominent location. That workaround will be cleaned up then.

Alternatively, drm_aperture_remove_conflicting_pci_framebuffers() could 
be changed to accept the name string as second argument, but that's 
quite a bit of churn within the DRM code.

Best regards
Thomas

[1] 
https://elixir.bootlin.com/linux/v5.18.2/source/drivers/gpu/drm/drm_aperture.c#L347
[2] 
https://elixir.bootlin.com/linux/v5.18.2/source/drivers/video/fbdev/core/fbmem.c#L1570


>   	ret = vga_client_register(pdev, vfio_pci_set_decode);
>   	if (ret)
>   		return ret;
> 
> 

-- 
Thomas Zimmermann
Graphics Driver Developer
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5, 90409 Nürnberg, Germany
(HRB 36809, AG Nürnberg)
Geschäftsführer: Ivo Totev

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]

Re: [PATCH 2/2] vfio/pci: Remove console drivers

[Alex Williamson]

Hi Thomas,

On Wed, 8 Jun 2022 13:11:21 +0200
Thomas Zimmermann <tzimmermann@suse.de> wrote:

> Hi Alex
> 
> Am 06.06.22 um 19:53 schrieb Alex Williamson:
> > Console drivers can create conflicts with PCI resources resulting in
> > userspace getting mmap failures to memory BARs.  This is especially evident
> > when trying to re-use the system primary console for userspace drivers.
> > Attempt to remove all nature of conflicting drivers as part of our VGA
> > initialization.  
> 
> First a dumb question about your use case.  You want to assign a PCI 
> graphics card to a virtual machine and need to remove the generic driver 
> from the framebuffer?

Exactly.
 
> > Reported-by: Laszlo Ersek <lersek@redhat.com>
> > Tested-by: Laszlo Ersek <lersek@redhat.com>
> > Suggested-by: Gerd Hoffmann <kraxel@redhat.com>
> > Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
> > ---
> >   drivers/vfio/pci/vfio_pci_core.c |   17 +++++++++++++++++
> >   1 file changed, 17 insertions(+)
> > 
> > diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
> > index a0d69ddaf90d..e0cbcbc2aee1 100644
> > --- a/drivers/vfio/pci/vfio_pci_core.c
> > +++ b/drivers/vfio/pci/vfio_pci_core.c
> > @@ -13,6 +13,7 @@
> >   #include <linux/device.h>
> >   #include <linux/eventfd.h>
> >   #include <linux/file.h>
> > +#include <linux/fb.h>
> >   #include <linux/interrupt.h>
> >   #include <linux/iommu.h>
> >   #include <linux/module.h>
> > @@ -29,6 +30,8 @@
> >   
> >   #include <linux/vfio_pci_core.h>
> >   
> > +#include <drm/drm_aperture.h>
> > +
> >   #define DRIVER_AUTHOR   "Alex Williamson <alex.williamson@redhat.com>"
> >   #define DRIVER_DESC "core driver for VFIO based PCI devices"
> >   
> > @@ -1793,6 +1796,20 @@ static int vfio_pci_vga_init(struct vfio_pci_core_device *vdev)
> >   	if (!vfio_pci_is_vga(pdev))
> >   		return 0;
> >   
> > +#if IS_REACHABLE(CONFIG_DRM)
> > +	drm_aperture_detach_platform_drivers(pdev);
> > +#endif
> > +
> > +#if IS_REACHABLE(CONFIG_FB)
> > +	ret = remove_conflicting_pci_framebuffers(pdev, vdev->vdev.ops->name);
> > +	if (ret)
> > +		return ret;
> > +#endif
> > +
> > +	ret = vga_remove_vgacon(pdev);
> > +	if (ret)
> > +		return ret;
> > +  
> 
> You shouldn't have to copy any of the implementation of the aperture 
> helpers.
> 
> If you call drm_aperture_remove_conflicting_pci_framebuffers() it should 
> work correctly. The only reason why it requires a DRM driver structure 
> as second argument is for the driver's name. [1] And that name is only 
> used for printing an info message. [2]

vfio-pci is not dependent on CONFIG_DRM, therefore we need to open code
this regardless.  The only difference if we were to use the existing
function would be something like:

#if IS_REACHABLE(CONFIG_DRM)
	ret = drm_aperture_remove_conflicting_pci_framebuffers(pdev, &dummy_drm_driver);
	if (ret)
		return ret;
#else
#if IS_REACHABLE(CONFIG_FB)
	ret = remove_conflicting_pci_framebuffers(pdev, vdev->vdev.ops->name);
	if (ret)
		return ret;
#endif
	ret = vga_remove_vgacon(pdev);
	if (ret)
		return ret;
#endif

It's also bad practice to create a dummy DRM driver struct with some
assumption which fields are used.  If the usage is later expanded, we'd
only discover it by users getting segfaults.  If DRM wanted to make
such an API guarantee, then we shouldn't have commit 97c9bfe3f660
("drm/aperture: Pass DRM driver structure instead of driver name").

> The plan forward would be to drop patch 1 entirely.
> 
> For patch 2, the most trivial workaround is to instanciate struct 
> drm_driver here and set the name field to 'vdev->vdev.ops->name'. In the 
> longer term, the aperture helpers will be moved out of DRM and into a 
> more prominent location. That workaround will be cleaned up then.

Trivial in execution, but as above, this is poor practice and should be
avoided.

> Alternatively, drm_aperture_remove_conflicting_pci_framebuffers() could 
> be changed to accept the name string as second argument, but that's 
> quite a bit of churn within the DRM code.

The series as presented was exactly meant to provide the most correct
solution with the least churn to the DRM code.  The above referenced
commit precludes us from calling the existing DRM function directly
without resorting to poor practices of assuming the usage of the DRM
driver struct.  Using the existing DRM function also does not prevent
us from open coding the remainder of the function to avoid a CONFIG_DRM
dependency.  Thanks,

Alex

Re: [PATCH 2/2] vfio/pci: Remove console drivers

[Alex Williamson]

Hi Thomas,

On Wed, 8 Jun 2022 13:11:21 +0200
Thomas Zimmermann <tzimmermann@suse.de> wrote:

> Hi Alex
> 
> Am 06.06.22 um 19:53 schrieb Alex Williamson:
> > Console drivers can create conflicts with PCI resources resulting in
> > userspace getting mmap failures to memory BARs.  This is especially evident
> > when trying to re-use the system primary console for userspace drivers.
> > Attempt to remove all nature of conflicting drivers as part of our VGA
> > initialization.  
> 
> First a dumb question about your use case.  You want to assign a PCI 
> graphics card to a virtual machine and need to remove the generic driver 
> from the framebuffer?

Exactly.
 
> > Reported-by: Laszlo Ersek <lersek@redhat.com>
> > Tested-by: Laszlo Ersek <lersek@redhat.com>
> > Suggested-by: Gerd Hoffmann <kraxel@redhat.com>
> > Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
> > ---
> >   drivers/vfio/pci/vfio_pci_core.c |   17 +++++++++++++++++
> >   1 file changed, 17 insertions(+)
> > 
> > diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
> > index a0d69ddaf90d..e0cbcbc2aee1 100644
> > --- a/drivers/vfio/pci/vfio_pci_core.c
> > +++ b/drivers/vfio/pci/vfio_pci_core.c
> > @@ -13,6 +13,7 @@
> >   #include <linux/device.h>
> >   #include <linux/eventfd.h>
> >   #include <linux/file.h>
> > +#include <linux/fb.h>
> >   #include <linux/interrupt.h>
> >   #include <linux/iommu.h>
> >   #include <linux/module.h>
> > @@ -29,6 +30,8 @@
> >   
> >   #include <linux/vfio_pci_core.h>
> >   
> > +#include <drm/drm_aperture.h>
> > +
> >   #define DRIVER_AUTHOR   "Alex Williamson <alex.williamson@redhat.com>"
> >   #define DRIVER_DESC "core driver for VFIO based PCI devices"
> >   
> > @@ -1793,6 +1796,20 @@ static int vfio_pci_vga_init(struct vfio_pci_core_device *vdev)
> >   	if (!vfio_pci_is_vga(pdev))
> >   		return 0;
> >   
> > +#if IS_REACHABLE(CONFIG_DRM)
> > +	drm_aperture_detach_platform_drivers(pdev);
> > +#endif
> > +
> > +#if IS_REACHABLE(CONFIG_FB)
> > +	ret = remove_conflicting_pci_framebuffers(pdev, vdev->vdev.ops->name);
> > +	if (ret)
> > +		return ret;
> > +#endif
> > +
> > +	ret = vga_remove_vgacon(pdev);
> > +	if (ret)
> > +		return ret;
> > +  
> 
> You shouldn't have to copy any of the implementation of the aperture 
> helpers.
> 
> If you call drm_aperture_remove_conflicting_pci_framebuffers() it should 
> work correctly. The only reason why it requires a DRM driver structure 
> as second argument is for the driver's name. [1] And that name is only 
> used for printing an info message. [2]

vfio-pci is not dependent on CONFIG_DRM, therefore we need to open code
this regardless.  The only difference if we were to use the existing
function would be something like:

#if IS_REACHABLE(CONFIG_DRM)
	ret = drm_aperture_remove_conflicting_pci_framebuffers(pdev, &dummy_drm_driver);
	if (ret)
		return ret;
#else
#if IS_REACHABLE(CONFIG_FB)
	ret = remove_conflicting_pci_framebuffers(pdev, vdev->vdev.ops->name);
	if (ret)
		return ret;
#endif
	ret = vga_remove_vgacon(pdev);
	if (ret)
		return ret;
#endif

It's also bad practice to create a dummy DRM driver struct with some
assumption which fields are used.  If the usage is later expanded, we'd
only discover it by users getting segfaults.  If DRM wanted to make
such an API guarantee, then we shouldn't have commit 97c9bfe3f660
("drm/aperture: Pass DRM driver structure instead of driver name").

> The plan forward would be to drop patch 1 entirely.
> 
> For patch 2, the most trivial workaround is to instanciate struct 
> drm_driver here and set the name field to 'vdev->vdev.ops->name'. In the 
> longer term, the aperture helpers will be moved out of DRM and into a 
> more prominent location. That workaround will be cleaned up then.

Trivial in execution, but as above, this is poor practice and should be
avoided.

> Alternatively, drm_aperture_remove_conflicting_pci_framebuffers() could 
> be changed to accept the name string as second argument, but that's 
> quite a bit of churn within the DRM code.

The series as presented was exactly meant to provide the most correct
solution with the least churn to the DRM code.  The above referenced
commit precludes us from calling the existing DRM function directly
without resorting to poor practices of assuming the usage of the DRM
driver struct.  Using the existing DRM function also does not prevent
us from open coding the remainder of the function to avoid a CONFIG_DRM
dependency.  Thanks,

Alex

Re: [PATCH 2/2] vfio/pci: Remove console drivers

[Thomas Zimmermann]

[-- Attachment #1.1.1: Type: text/plain, Size: 3607 bytes --]

Hi

Am 08.06.22 um 16:04 schrieb Alex Williamson:
>>
>> You shouldn't have to copy any of the implementation of the aperture
>> helpers.
>>
>> If you call drm_aperture_remove_conflicting_pci_framebuffers() it should
>> work correctly. The only reason why it requires a DRM driver structure
>> as second argument is for the driver's name. [1] And that name is only
>> used for printing an info message. [2]
> 
> vfio-pci is not dependent on CONFIG_DRM, therefore we need to open code
> this regardless.  The only difference if we were to use the existing
> function would be something like:
> 
> #if IS_REACHABLE(CONFIG_DRM)
> 	ret = drm_aperture_remove_conflicting_pci_framebuffers(pdev, &dummy_drm_driver);
> 	if (ret)
> 		return ret;
> #else
> #if IS_REACHABLE(CONFIG_FB)
> 	ret = remove_conflicting_pci_framebuffers(pdev, vdev->vdev.ops->name);
> 	if (ret)
> 		return ret;
> #endif
> 	ret = vga_remove_vgacon(pdev);
> 	if (ret)
> 		return ret;
> #endif
> 
> It's also bad practice to create a dummy DRM driver struct with some
> assumption which fields are used.  If the usage is later expanded, we'd
> only discover it by users getting segfaults.  If DRM wanted to make
> such an API guarantee, then we shouldn't have commit 97c9bfe3f660
> ("drm/aperture: Pass DRM driver structure instead of driver name").

What you're open coding within vfio is legacy code and we want to remove 
it as much as possible from the aperture helpers.

Tying the helpers to DRM was in mistake in retrospective. We wanted 
something tailored to the needs of DRM. Now that we've seen quite a few 
corner cases in the interaction among graphics subsystems, we need 
something else. The order of creating devices and loading driver modules 
is crucial to making the hand-over between drivers work reliably. So 
far, this luckily has only been a problem in theory, but not practice.

> 
>> The plan forward would be to drop patch 1 entirely.
>>
>> For patch 2, the most trivial workaround is to instanciate struct
>> drm_driver here and set the name field to 'vdev->vdev.ops->name'. In the
>> longer term, the aperture helpers will be moved out of DRM and into a
>> more prominent location. That workaround will be cleaned up then.
> 
> Trivial in execution, but as above, this is poor practice and should be
> avoided.
> 
>> Alternatively, drm_aperture_remove_conflicting_pci_framebuffers() could
>> be changed to accept the name string as second argument, but that's
>> quite a bit of churn within the DRM code.
> 
> The series as presented was exactly meant to provide the most correct
> solution with the least churn to the DRM code.  The above referenced
> commit precludes us from calling the existing DRM function directly
> without resorting to poor practices of assuming the usage of the DRM
> driver struct.  Using the existing DRM function also does not prevent
> us from open coding the remainder of the function to avoid a CONFIG_DRM
> dependency.  Thanks,

Please have a look at the attached patch. It moves the aperture helpers 
to a location common to the various possible users (DRM, fbdev, vfio). 
The DRM interfaces remain untouched for now.  The patch should provide 
what you need in vfio and also serve our future use cases for graphics 
drivers. If possible, please create your patch on top of it.

Best regards
Thomas

> 
> Alex
> 

-- 
Thomas Zimmermann
Graphics Driver Developer
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5, 90409 Nürnberg, Germany
(HRB 36809, AG Nürnberg)
Geschäftsführer: Ivo Totev

[-- Attachment #1.1.2: 0001-drm-Implement-DRM-aperture-helpers-under-video.patch --]
[-- Type: text/x-patch, Size: 24431 bytes --]

From ae8d8ebcf3884529ca1a9b659603a59271a8a553 Mon Sep 17 00:00:00 2001
From: Thomas Zimmermann <tzimmermann@suse.de>
Date: Wed, 8 Jun 2022 20:52:50 +0200
Subject: [PATCH] drm: Implement DRM aperture helpers under video/

Implement DRM's aperture helpers under video/ for sharing with other
sub-systems. Remove DRM-isms from the interface. The helpers track
the ownership of framebuffer apertures and provide hand-over from
firmware, such as EFI and VESA, to native graphics drivers.

Other subsystems, such as fbdev and vfio, also have to maintain ownership
of framebuffer apertures. Moving DRM's aperture helpers to a more public
location allows all subsystems to interact with each other and share a
common implementation.

The aperture helpers are selected by the various firmware drivers within
DRM and fbdev, and the VGA text-console driver.

The original DRM interface is kept in place for use by DRM drivers.

Signed-off-by: Thomas Zimmermann <tzimmermann@suse.de>
---
 Documentation/driver-api/aperture.rst |  13 +
 Documentation/driver-api/index.rst    |   1 +
 drivers/gpu/drm/drm_aperture.c        | 174 +------------
 drivers/gpu/drm/tiny/Kconfig          |   1 +
 drivers/video/Kconfig                 |   6 +
 drivers/video/Makefile                |   2 +
 drivers/video/aperture.c              | 340 ++++++++++++++++++++++++++
 drivers/video/console/Kconfig         |   1 +
 drivers/video/fbdev/Kconfig           |   7 +-
 include/linux/aperture.h              |  56 +++++
 10 files changed, 435 insertions(+), 166 deletions(-)
 create mode 100644 Documentation/driver-api/aperture.rst
 create mode 100644 drivers/video/aperture.c
 create mode 100644 include/linux/aperture.h

diff --git a/Documentation/driver-api/aperture.rst b/Documentation/driver-api/aperture.rst
new file mode 100644
index 000000000000..d173f4e7a7d9
--- /dev/null
+++ b/Documentation/driver-api/aperture.rst
@@ -0,0 +1,13 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+Managing Ownership of the Framebuffer Aperture
+==============================================
+
+.. kernel-doc:: drivers/video/aperture.c
+   :doc: overview
+
+.. kernel-doc:: include/linux/aperture.h
+   :internal:
+
+.. kernel-doc:: drivers/video/aperture.c
+   :export:
diff --git a/Documentation/driver-api/index.rst b/Documentation/driver-api/index.rst
index a7b0223e2886..d6d6e77d8afc 100644
--- a/Documentation/driver-api/index.rst
+++ b/Documentation/driver-api/index.rst
@@ -27,6 +27,7 @@ available subsections can be seen below.
    component
    message-based
    infiniband
+   aperture
    frame-buffer
    regulator
    reset
diff --git a/drivers/gpu/drm/drm_aperture.c b/drivers/gpu/drm/drm_aperture.c
index 74bd4a76b253..388a205bd023 100644
--- a/drivers/gpu/drm/drm_aperture.c
+++ b/drivers/gpu/drm/drm_aperture.c
@@ -1,14 +1,7 @@
 // SPDX-License-Identifier: MIT
 
-#include <linux/device.h>
-#include <linux/fb.h>
-#include <linux/list.h>
-#include <linux/mutex.h>
-#include <linux/pci.h>
-#include <linux/platform_device.h> /* for firmware helpers */
-#include <linux/slab.h>
-#include <linux/types.h>
-#include <linux/vgaarb.h>
+#include <linux/aperture.h>
+#include <linux/platform_device.h>
 
 #include <drm/drm_aperture.h>
 #include <drm/drm_drv.h>
@@ -126,92 +119,6 @@
  * afterwards.
  */
 
-struct drm_aperture {
-	struct drm_device *dev;
-	resource_size_t base;
-	resource_size_t size;
-	struct list_head lh;
-	void (*detach)(struct drm_device *dev);
-};
-
-static LIST_HEAD(drm_apertures);
-static DEFINE_MUTEX(drm_apertures_lock);
-
-static bool overlap(resource_size_t base1, resource_size_t end1,
-		    resource_size_t base2, resource_size_t end2)
-{
-	return (base1 < end2) && (end1 > base2);
-}
-
-static void devm_aperture_acquire_release(void *data)
-{
-	struct drm_aperture *ap = data;
-	bool detached = !ap->dev;
-
-	if (detached)
-		return;
-
-	mutex_lock(&drm_apertures_lock);
-	list_del(&ap->lh);
-	mutex_unlock(&drm_apertures_lock);
-}
-
-static int devm_aperture_acquire(struct drm_device *dev,
-				 resource_size_t base, resource_size_t size,
-				 void (*detach)(struct drm_device *))
-{
-	size_t end = base + size;
-	struct list_head *pos;
-	struct drm_aperture *ap;
-
-	mutex_lock(&drm_apertures_lock);
-
-	list_for_each(pos, &drm_apertures) {
-		ap = container_of(pos, struct drm_aperture, lh);
-		if (overlap(base, end, ap->base, ap->base + ap->size)) {
-			mutex_unlock(&drm_apertures_lock);
-			return -EBUSY;
-		}
-	}
-
-	ap = devm_kzalloc(dev->dev, sizeof(*ap), GFP_KERNEL);
-	if (!ap) {
-		mutex_unlock(&drm_apertures_lock);
-		return -ENOMEM;
-	}
-
-	ap->dev = dev;
-	ap->base = base;
-	ap->size = size;
-	ap->detach = detach;
-	INIT_LIST_HEAD(&ap->lh);
-
-	list_add(&ap->lh, &drm_apertures);
-
-	mutex_unlock(&drm_apertures_lock);
-
-	return devm_add_action_or_reset(dev->dev, devm_aperture_acquire_release, ap);
-}
-
-static void drm_aperture_detach_firmware(struct drm_device *dev)
-{
-	struct platform_device *pdev = to_platform_device(dev->dev);
-
-	/*
-	 * Remove the device from the device hierarchy. This is the right thing
-	 * to do for firmware-based DRM drivers, such as EFI, VESA or VGA. After
-	 * the new driver takes over the hardware, the firmware device's state
-	 * will be lost.
-	 *
-	 * For non-platform devices, a new callback would be required.
-	 *
-	 * If the aperture helpers ever need to handle native drivers, this call
-	 * would only have to unplug the DRM device, so that the hardware device
-	 * stays around after detachment.
-	 */
-	platform_device_unregister(pdev);
-}
-
 /**
  * devm_aperture_acquire_from_firmware - Acquires ownership of a firmware framebuffer
  *                                       on behalf of a DRM driver.
@@ -239,39 +146,16 @@ static void drm_aperture_detach_firmware(struct drm_device *dev)
 int devm_aperture_acquire_from_firmware(struct drm_device *dev, resource_size_t base,
 					resource_size_t size)
 {
+	struct platform_device *pdev;
+
 	if (drm_WARN_ON(dev, !dev_is_platform(dev->dev)))
 		return -EINVAL;
 
-	return devm_aperture_acquire(dev, base, size, drm_aperture_detach_firmware);
-}
-EXPORT_SYMBOL(devm_aperture_acquire_from_firmware);
-
-static void drm_aperture_detach_drivers(resource_size_t base, resource_size_t size)
-{
-	resource_size_t end = base + size;
-	struct list_head *pos, *n;
-
-	mutex_lock(&drm_apertures_lock);
-
-	list_for_each_safe(pos, n, &drm_apertures) {
-		struct drm_aperture *ap =
-			container_of(pos, struct drm_aperture, lh);
-		struct drm_device *dev = ap->dev;
-
-		if (WARN_ON_ONCE(!dev))
-			continue;
-
-		if (!overlap(base, end, ap->base, ap->base + ap->size))
-			continue;
+	pdev = to_platform_device(dev->dev);
 
-		ap->dev = NULL; /* detach from device */
-		list_del(&ap->lh);
-
-		ap->detach(dev);
-	}
-
-	mutex_unlock(&drm_apertures_lock);
+	return devm_acquire_aperture_of_platform_device(pdev, base, size);
 }
+EXPORT_SYMBOL(devm_aperture_acquire_from_firmware);
 
 /**
  * drm_aperture_remove_conflicting_framebuffers - remove existing framebuffers in the given range
@@ -289,27 +173,7 @@ static void drm_aperture_detach_drivers(resource_size_t base, resource_size_t si
 int drm_aperture_remove_conflicting_framebuffers(resource_size_t base, resource_size_t size,
 						 bool primary, const struct drm_driver *req_driver)
 {
-#if IS_REACHABLE(CONFIG_FB)
-	struct apertures_struct *a;
-	int ret;
-
-	a = alloc_apertures(1);
-	if (!a)
-		return -ENOMEM;
-
-	a->ranges[0].base = base;
-	a->ranges[0].size = size;
-
-	ret = remove_conflicting_framebuffers(a, req_driver->name, primary);
-	kfree(a);
-
-	if (ret)
-		return ret;
-#endif
-
-	drm_aperture_detach_drivers(base, size);
-
-	return 0;
+	return remove_conflicting_devices(base, size, primary, req_driver->name);
 }
 EXPORT_SYMBOL(drm_aperture_remove_conflicting_framebuffers);
 
@@ -328,26 +192,6 @@ EXPORT_SYMBOL(drm_aperture_remove_conflicting_framebuffers);
 int drm_aperture_remove_conflicting_pci_framebuffers(struct pci_dev *pdev,
 						     const struct drm_driver *req_driver)
 {
-	resource_size_t base, size;
-	int bar, ret = 0;
-
-	for (bar = 0; bar < PCI_STD_NUM_BARS; ++bar) {
-		if (!(pci_resource_flags(pdev, bar) & IORESOURCE_MEM))
-			continue;
-		base = pci_resource_start(pdev, bar);
-		size = pci_resource_len(pdev, bar);
-		drm_aperture_detach_drivers(base, size);
-	}
-
-	/*
-	 * WARNING: Apparently we must kick fbdev drivers before vgacon,
-	 * otherwise the vga fbdev driver falls over.
-	 */
-#if IS_REACHABLE(CONFIG_FB)
-	ret = remove_conflicting_pci_framebuffers(pdev, req_driver->name);
-#endif
-	if (ret == 0)
-		ret = vga_remove_vgacon(pdev);
-	return ret;
+	return remove_conflicting_pci_devices(pdev, req_driver->name);
 }
 EXPORT_SYMBOL(drm_aperture_remove_conflicting_pci_framebuffers);
diff --git a/drivers/gpu/drm/tiny/Kconfig b/drivers/gpu/drm/tiny/Kconfig
index 627d637a1e7e..027cd87c3d0d 100644
--- a/drivers/gpu/drm/tiny/Kconfig
+++ b/drivers/gpu/drm/tiny/Kconfig
@@ -69,6 +69,7 @@ config DRM_PANEL_MIPI_DBI
 config DRM_SIMPLEDRM
 	tristate "Simple framebuffer driver"
 	depends on DRM && MMU
+	select APERTURE_HELPERS
 	select DRM_GEM_SHMEM_HELPER
 	select DRM_KMS_HELPER
 	help
diff --git a/drivers/video/Kconfig b/drivers/video/Kconfig
index 427a993c7f57..c69b45f8c427 100644
--- a/drivers/video/Kconfig
+++ b/drivers/video/Kconfig
@@ -5,6 +5,12 @@
 
 menu "Graphics support"
 
+config APERTURE_HELPERS
+	bool
+	help
+	  Support tracking and hand-over of aperture ownership. Required
+	  for firmware graphics drivers.
+
 if HAS_IOMEM
 
 config HAVE_FB_ATMEL
diff --git a/drivers/video/Makefile b/drivers/video/Makefile
index df7650adede9..5bb6b452cc83 100644
--- a/drivers/video/Makefile
+++ b/drivers/video/Makefile
@@ -1,4 +1,6 @@
 # SPDX-License-Identifier: GPL-2.0
+
+obj-$(CONFIG_APERTURE_HELPERS)    += aperture.o
 obj-$(CONFIG_VGASTATE)            += vgastate.o
 obj-$(CONFIG_HDMI)                += hdmi.o
 
diff --git a/drivers/video/aperture.c b/drivers/video/aperture.c
new file mode 100644
index 000000000000..f0b877e9c256
--- /dev/null
+++ b/drivers/video/aperture.c
@@ -0,0 +1,340 @@
+// SPDX-License-Identifier: MIT
+
+#include <linux/aperture.h>
+#include <linux/device.h>
+#include <linux/fb.h> /* for old fbdev helpers */
+#include <linux/list.h>
+#include <linux/mutex.h>
+#include <linux/pci.h>
+#include <linux/platform_device.h>
+#include <linux/slab.h>
+#include <linux/types.h>
+#include <linux/vgaarb.h>
+
+/**
+ * DOC: overview
+ *
+ * A graphics device might be supported by different drivers, but only one
+ * driver can be active at any given time. Many systems load a generic
+ * graphics drivers, such as EFI-GOP or VESA, early during the boot process.
+ * During later boot stages, they replace the generic driver with a dedicated,
+ * hardware-specific driver. To take over the device the dedicated driver
+ * first has to remove the generic driver. Aperture functions manage
+ * ownership of framebuffer memory and hand-over between drivers.
+ *
+ * Graphics drivers should call remove_conflicting_devices()
+ * at the top of their probe function. The function removes any generic
+ * driver that is currently associated with the given framebuffer memory.
+ * If the framebuffer is located at PCI BAR 0, the rsp code looks as in the
+ * example given below. The cod assumes a DRM driver.
+ *
+ * .. code-block:: c
+ *
+ *	static const struct drm_driver example_driver = {
+ *		.name = "exampledrm",
+ *		...
+ *	};
+ *
+ *	static int remove_conflicting_framebuffers(struct pci_dev *pdev)
+ *	{
+ *		bool primary = false;
+ *		resource_size_t base, size;
+ *		int ret;
+ *
+ *		base = pci_resource_start(pdev, 0);
+ *		size = pci_resource_len(pdev, 0);
+ *	#ifdef CONFIG_X86
+ *		primary = pdev->resource[PCI_ROM_RESOURCE].flags & IORESOURCE_ROM_SHADOW;
+ *	#endif
+ *
+ *		return remove_conflicting_devices(base, size, primary, &example_driver->name);
+ *	}
+ *
+ *	static int probe(struct pci_dev *pdev)
+ *	{
+ *		int ret;
+ *
+ *		// Remove any generic drivers...
+ *		ret = remove_conflicting_framebuffers(pdev);
+ *		if (ret)
+ *			return ret;
+ *
+ *		// ... and initialize the hardware.
+ *		...
+ *
+ *		drm_dev_register();
+ *
+ *		return 0;
+ *	}
+ *
+ * PCI device drivers can also call remove_conflicting_pci_devices() and let the
+ * function detect the apertures automatically. Device drivers without knowledge of
+ * the framebuffer's location shall call remove_all_conflicting_devices(),
+ * which removes all known devices.
+ *
+ * Drivers that are susceptible to being removed by other drivers, such as
+ * generic EFI or VESA drivers, have to register themselves as owners of their
+ * framebuffer apertures. Ownership of the framebuffer memory is achieved
+ * by calling devm_acquire_aperture_of_platform_device(). On success, the driver
+ * is the owner of the framebuffer range. The function fails if the
+ * framebuffer is already owned by another driver. See below for an example.
+ *
+ * .. code-block:: c
+ *
+ *	static int acquire_framebuffers(struct drm_device *dev, struct platform_device *pdev)
+ *	{
+ *		resource_size_t base, size;
+ *
+ *		mem = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ *		if (!mem)
+ *			return -EINVAL;
+ *		base = mem->start;
+ *		size = resource_size(mem);
+ *
+ *		return devm_acquire_aperture_of_platform_device(pdev, base, size);
+ *	}
+ *
+ *	static int probe(struct platform_device *pdev)
+ *	{
+ *		struct drm_device *dev;
+ *		int ret;
+ *
+ *		// ... Initialize the device...
+ *		dev = devm_drm_dev_alloc();
+ *		...
+ *
+ *		// ... and acquire ownership of the framebuffer.
+ *		ret = acquire_framebuffers(dev, pdev);
+ *		if (ret)
+ *			return ret;
+ *
+ *		drm_dev_register(dev, 0);
+ *
+ *		return 0;
+ *	}
+ *
+ * The generic driver is now subject to forced removal by other drivers. This
+ * only works for platform drivers that support hot unplugging.
+ * When a driver calls remove_conflicting_devices() et al
+ * for the registered framebuffer range, the aperture helpers call
+ * platform_device_unregister() and the generic driver unloads itself. It
+ * may not access the device's registers, framebuffer memory, ROM, etc
+ * afterwards.
+ */
+
+struct dev_aperture {
+	struct device *dev;
+	resource_size_t base;
+	resource_size_t size;
+	struct list_head lh;
+	void (*detach)(struct device *dev);
+};
+
+static LIST_HEAD(apertures);
+static DEFINE_MUTEX(apertures_lock);
+
+static bool overlap(resource_size_t base1, resource_size_t end1,
+		    resource_size_t base2, resource_size_t end2)
+{
+	return (base1 < end2) && (end1 > base2);
+}
+
+static void devm_aperture_acquire_release(void *data)
+{
+	struct dev_aperture *ap = data;
+	bool detached = !ap->dev;
+
+	if (detached)
+		return;
+
+	mutex_lock(&apertures_lock);
+	list_del(&ap->lh);
+	mutex_unlock(&apertures_lock);
+}
+
+static int devm_aperture_acquire(struct device *dev,
+				 resource_size_t base, resource_size_t size,
+				 void (*detach)(struct device *))
+{
+	size_t end = base + size;
+	struct list_head *pos;
+	struct dev_aperture *ap;
+
+	mutex_lock(&apertures_lock);
+
+	list_for_each(pos, &apertures) {
+		ap = container_of(pos, struct dev_aperture, lh);
+		if (overlap(base, end, ap->base, ap->base + ap->size)) {
+			mutex_unlock(&apertures_lock);
+			return -EBUSY;
+		}
+	}
+
+	ap = devm_kzalloc(dev, sizeof(*ap), GFP_KERNEL);
+	if (!ap) {
+		mutex_unlock(&apertures_lock);
+		return -ENOMEM;
+	}
+
+	ap->dev = dev;
+	ap->base = base;
+	ap->size = size;
+	ap->detach = detach;
+	INIT_LIST_HEAD(&ap->lh);
+
+	list_add(&ap->lh, &apertures);
+
+	mutex_unlock(&apertures_lock);
+
+	return devm_add_action_or_reset(dev, devm_aperture_acquire_release, ap);
+}
+
+static void detach_platform_device(struct device *dev)
+{
+	struct platform_device *pdev = to_platform_device(dev);
+
+	/*
+	 * Remove the device from the device hierarchy. This is the right thing
+	 * to do for firmware-based DRM drivers, such as EFI, VESA or VGA. After
+	 * the new driver takes over the hardware, the firmware device's state
+	 * will be lost.
+	 *
+	 * For non-platform devices, a new callback would be required.
+	 *
+	 * If the aperture helpers ever need to handle native drivers, this call
+	 * would only have to unplug the DRM device, so that the hardware device
+	 * stays around after detachment.
+	 */
+	platform_device_unregister(pdev);
+}
+
+/**
+ * devm_acquire_aperture_of_platform_device - Acquires ownership of an aperture
+ *                                            on behalf of a platform device.
+ * @pdev:	the platform device to own the aperture
+ * @base:	the aperture's byte offset in physical memory
+ * @size:	the aperture size in bytes
+ *
+ * Installs the given device as the new owner of the aperture. The function
+ * expects the aperture to be provided by a platform device. If another
+ * driver takes over ownership of the aperture, aperture helpers will then
+ * unregister the platform device automatically. All acquired apertures are
+ * released automatically when the underlying device goes away.
+ *
+ * The function fails if the aperture, or parts of it, is currently
+ * owned by another device. To evict current owners, callers should use
+ * remove_conflicting_devices() et al. before calling this function.
+ *
+ * Returns:
+ * 0 on success, or a negative errno value otherwise.
+ */
+int devm_acquire_aperture_of_platform_device(struct platform_device *pdev,
+					     resource_size_t base,
+					     resource_size_t size)
+{
+	return devm_aperture_acquire(&pdev->dev, base, size, detach_platform_device);
+}
+EXPORT_SYMBOL(devm_acquire_aperture_of_platform_device);
+
+static void detach_devices(resource_size_t base, resource_size_t size)
+{
+	resource_size_t end = base + size;
+	struct list_head *pos, *n;
+
+	mutex_lock(&apertures_lock);
+
+	list_for_each_safe(pos, n, &apertures) {
+		struct dev_aperture *ap = container_of(pos, struct dev_aperture, lh);
+		struct device *dev = ap->dev;
+
+		if (WARN_ON_ONCE(!dev))
+			continue;
+
+		if (!overlap(base, end, ap->base, ap->base + ap->size))
+			continue;
+
+		ap->dev = NULL; /* detach from device */
+		list_del(&ap->lh);
+
+		ap->detach(dev);
+	}
+
+	mutex_unlock(&apertures_lock);
+}
+
+/**
+ * remove_conflicting_devices - remove devices in the given range
+ * @base: the aperture's base address in physical memory
+ * @size: aperture size in bytes
+ * @primary: also kick vga16fb if present; only relevant for VGA devices
+ * @name: a descriptive name of the requesting driver
+ *
+ * This function removes devices that own apertures within @base and @size.
+ *
+ * Returns:
+ * 0 on success, or a negative errno code otherwise
+ */
+int remove_conflicting_devices(resource_size_t base, resource_size_t size, bool primary,
+			       const char *name)
+{
+#if IS_REACHABLE(CONFIG_FB)
+	struct apertures_struct *a;
+	int ret;
+
+	a = alloc_apertures(1);
+	if (!a)
+		return -ENOMEM;
+
+	a->ranges[0].base = base;
+	a->ranges[0].size = size;
+
+	ret = remove_conflicting_framebuffers(a, name, primary);
+	kfree(a);
+
+	if (ret)
+		return ret;
+#endif
+
+	detach_devices(base, size);
+
+	return 0;
+}
+EXPORT_SYMBOL(remove_conflicting_devices);
+
+/**
+ * remove_conflicting_pci_devices - remove existing framebuffers for PCI devices
+ * @pdev: PCI device
+ * @name: a descriptive name of the requesting driver
+ *
+ * This function removes devices that own apertures within any of @pdev's
+ * memory bars. The function assumes that PCI device with shadowed ROM
+ * drives a primary display and therefore kicks out vga16fb as well.
+ *
+ * Returns:
+ * 0 on success, or a negative errno code otherwise
+ */
+int remove_conflicting_pci_devices(struct pci_dev *pdev, const char *name)
+{
+	resource_size_t base, size;
+	int bar, ret = 0;
+
+	for (bar = 0; bar < PCI_STD_NUM_BARS; ++bar) {
+		if (!(pci_resource_flags(pdev, bar) & IORESOURCE_MEM))
+			continue;
+		base = pci_resource_start(pdev, bar);
+		size = pci_resource_len(pdev, bar);
+		detach_devices(base, size);
+	}
+
+	/*
+	 * WARNING: Apparently we must kick fbdev drivers before vgacon,
+	 * otherwise the vga fbdev driver falls over.
+	 */
+#if IS_REACHABLE(CONFIG_FB)
+	ret = remove_conflicting_pci_framebuffers(pdev, name);
+#endif
+	if (ret == 0)
+		ret = vga_remove_vgacon(pdev);
+	return ret;
+}
+EXPORT_SYMBOL(remove_conflicting_pci_devices);
diff --git a/drivers/video/console/Kconfig b/drivers/video/console/Kconfig
index 40c50fa2dd70..7f3c44e1538b 100644
--- a/drivers/video/console/Kconfig
+++ b/drivers/video/console/Kconfig
@@ -10,6 +10,7 @@ config VGA_CONSOLE
 	depends on !4xx && !PPC_8xx && !SPARC && !M68K && !PARISC &&  !SUPERH && \
 		(!ARM || ARCH_FOOTBRIDGE || ARCH_INTEGRATOR || ARCH_NETWINDER) && \
 		!ARM64 && !ARC && !MICROBLAZE && !OPENRISC && !S390 && !UML
+	select APERTURE_HELPERS if (DRM || FB || VFIO_PCI)
 	default y
 	help
 	  Saying Y here will allow you to use Linux in text mode through a
diff --git a/drivers/video/fbdev/Kconfig b/drivers/video/fbdev/Kconfig
index bd849013f16f..7b398e7e3cc8 100644
--- a/drivers/video/fbdev/Kconfig
+++ b/drivers/video/fbdev/Kconfig
@@ -455,6 +455,7 @@ config FB_ATARI
 config FB_OF
 	bool "Open Firmware frame buffer device support"
 	depends on (FB = y) && PPC && (!PPC_PSERIES || PCI)
+	select APERTURE_HELPERS
 	select FB_CFB_FILLRECT
 	select FB_CFB_COPYAREA
 	select FB_CFB_IMAGEBLIT
@@ -527,6 +528,7 @@ config FB_IMSTT
 config FB_VGA16
 	tristate "VGA 16-color graphics support"
 	depends on FB && (X86 || PPC)
+	select APERTURE_HELPERS
 	select FB_CFB_FILLRECT
 	select FB_CFB_COPYAREA
 	select FB_CFB_IMAGEBLIT
@@ -551,7 +553,7 @@ config FB_STI
 	  BIOS routines contained in a ROM chip in HP PA-RISC based machines.
 	  Enabling this option will implement the linux framebuffer device
 	  using calls to the STI BIOS routines for initialisation.
-	
+
 	  If you enable this option, you will get a planar framebuffer device
 	  /dev/fb which will work on the most common HP graphic cards of the
 	  NGLE family, including the artist chips (in the 7xx and Bxxx series),
@@ -617,6 +619,7 @@ config FB_UVESA
 config FB_VESA
 	bool "VESA VGA graphics support"
 	depends on (FB = y) && (X86 || COMPILE_TEST)
+	select APERTURE_HELPERS
 	select FB_CFB_FILLRECT
 	select FB_CFB_COPYAREA
 	select FB_CFB_IMAGEBLIT
@@ -630,6 +633,7 @@ config FB_VESA
 config FB_EFI
 	bool "EFI-based Framebuffer Support"
 	depends on (FB = y) && !IA64 && EFI
+	select APERTURE_HELPERS
 	select DRM_PANEL_ORIENTATION_QUIRKS
 	select FB_CFB_FILLRECT
 	select FB_CFB_COPYAREA
@@ -2190,6 +2194,7 @@ config FB_SIMPLE
 	tristate "Simple framebuffer support"
 	depends on FB
 	depends on !DRM_SIMPLEDRM
+	select APERTURE_HELPERS
 	select FB_CFB_FILLRECT
 	select FB_CFB_COPYAREA
 	select FB_CFB_IMAGEBLIT
diff --git a/include/linux/aperture.h b/include/linux/aperture.h
new file mode 100644
index 000000000000..0a206cdce606
--- /dev/null
+++ b/include/linux/aperture.h
@@ -0,0 +1,56 @@
+/* SPDX-License-Identifier: MIT */
+
+#ifndef _LINUX_APERTURE_H_
+#define _LINUX_APERTURE_H_
+
+#include <linux/types.h>
+
+struct pci_dev;
+struct platform_device;
+
+#if defined(CONFIG_APERTURE_HELPERS)
+int devm_acquire_aperture_of_platform_device(struct platform_device *pdev,
+					     resource_size_t base,
+					     resource_size_t size);
+
+int remove_conflicting_devices(resource_size_t base, resource_size_t size,
+			       bool primary, const char *name);
+
+int remove_conflicting_pci_devices(struct pci_dev *pdev, const char *name);
+#else
+static inline int devm_acquire_aperture_of_platform_device(struct platform_device *pdev,
+							   resource_size_t base,
+							   resource_size_t size)
+{
+	return 0;
+}
+
+static inline int remove_conflicting_devices(resource_size_t base, resource_size_t size,
+					     bool primary, const char *name)
+{
+	return 0;
+}
+
+static inline int remove_conflicting_pci_devices(struct pci_dev *pdev, const char *name)
+{
+	return 0;
+}
+#endif
+
+/**
+ * remove_all_conflicting_devices - remove all existing framebuffers
+ * @primary: also kick vga16fb if present; only relevant for VGA devices
+ * @name: a descriptive name of the requesting driver
+ *
+ * This function removes all graphics device drivers. Use this function on systems
+ * that can have their framebuffer located anywhere in memory.
+ *
+ * Returns:
+ * 0 on success, or a negative errno code otherwise
+ */
+static inline int remove_all_conflicting_devices(bool primary, const char *name)
+{
+	return remove_conflicting_devices(0, (resource_size_t)-1, primary, name);
+}
+
+#endif
-- 
2.36.1


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]

Re: [PATCH 2/2] vfio/pci: Remove console drivers

[Thomas Zimmermann]

[-- Attachment #1.1.1: Type: text/plain, Size: 3607 bytes --]

Hi

Am 08.06.22 um 16:04 schrieb Alex Williamson:
>>
>> You shouldn't have to copy any of the implementation of the aperture
>> helpers.
>>
>> If you call drm_aperture_remove_conflicting_pci_framebuffers() it should
>> work correctly. The only reason why it requires a DRM driver structure
>> as second argument is for the driver's name. [1] And that name is only
>> used for printing an info message. [2]
> 
> vfio-pci is not dependent on CONFIG_DRM, therefore we need to open code
> this regardless.  The only difference if we were to use the existing
> function would be something like:
> 
> #if IS_REACHABLE(CONFIG_DRM)
> 	ret = drm_aperture_remove_conflicting_pci_framebuffers(pdev, &dummy_drm_driver);
> 	if (ret)
> 		return ret;
> #else
> #if IS_REACHABLE(CONFIG_FB)
> 	ret = remove_conflicting_pci_framebuffers(pdev, vdev->vdev.ops->name);
> 	if (ret)
> 		return ret;
> #endif
> 	ret = vga_remove_vgacon(pdev);
> 	if (ret)
> 		return ret;
> #endif
> 
> It's also bad practice to create a dummy DRM driver struct with some
> assumption which fields are used.  If the usage is later expanded, we'd
> only discover it by users getting segfaults.  If DRM wanted to make
> such an API guarantee, then we shouldn't have commit 97c9bfe3f660
> ("drm/aperture: Pass DRM driver structure instead of driver name").

What you're open coding within vfio is legacy code and we want to remove 
it as much as possible from the aperture helpers.

Tying the helpers to DRM was in mistake in retrospective. We wanted 
something tailored to the needs of DRM. Now that we've seen quite a few 
corner cases in the interaction among graphics subsystems, we need 
something else. The order of creating devices and loading driver modules 
is crucial to making the hand-over between drivers work reliably. So 
far, this luckily has only been a problem in theory, but not practice.

> 
>> The plan forward would be to drop patch 1 entirely.
>>
>> For patch 2, the most trivial workaround is to instanciate struct
>> drm_driver here and set the name field to 'vdev->vdev.ops->name'. In the
>> longer term, the aperture helpers will be moved out of DRM and into a
>> more prominent location. That workaround will be cleaned up then.
> 
> Trivial in execution, but as above, this is poor practice and should be
> avoided.
> 
>> Alternatively, drm_aperture_remove_conflicting_pci_framebuffers() could
>> be changed to accept the name string as second argument, but that's
>> quite a bit of churn within the DRM code.
> 
> The series as presented was exactly meant to provide the most correct
> solution with the least churn to the DRM code.  The above referenced
> commit precludes us from calling the existing DRM function directly
> without resorting to poor practices of assuming the usage of the DRM
> driver struct.  Using the existing DRM function also does not prevent
> us from open coding the remainder of the function to avoid a CONFIG_DRM
> dependency.  Thanks,

Please have a look at the attached patch. It moves the aperture helpers 
to a location common to the various possible users (DRM, fbdev, vfio). 
The DRM interfaces remain untouched for now.  The patch should provide 
what you need in vfio and also serve our future use cases for graphics 
drivers. If possible, please create your patch on top of it.

Best regards
Thomas

> 
> Alex
> 

-- 
Thomas Zimmermann
Graphics Driver Developer
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5, 90409 Nürnberg, Germany
(HRB 36809, AG Nürnberg)
Geschäftsführer: Ivo Totev

[-- Attachment #1.1.2: 0001-drm-Implement-DRM-aperture-helpers-under-video.patch --]
[-- Type: text/x-patch, Size: 24431 bytes --]

From ae8d8ebcf3884529ca1a9b659603a59271a8a553 Mon Sep 17 00:00:00 2001
From: Thomas Zimmermann <tzimmermann@suse.de>
Date: Wed, 8 Jun 2022 20:52:50 +0200
Subject: [PATCH] drm: Implement DRM aperture helpers under video/

Implement DRM's aperture helpers under video/ for sharing with other
sub-systems. Remove DRM-isms from the interface. The helpers track
the ownership of framebuffer apertures and provide hand-over from
firmware, such as EFI and VESA, to native graphics drivers.

Other subsystems, such as fbdev and vfio, also have to maintain ownership
of framebuffer apertures. Moving DRM's aperture helpers to a more public
location allows all subsystems to interact with each other and share a
common implementation.

The aperture helpers are selected by the various firmware drivers within
DRM and fbdev, and the VGA text-console driver.

The original DRM interface is kept in place for use by DRM drivers.

Signed-off-by: Thomas Zimmermann <tzimmermann@suse.de>
---
 Documentation/driver-api/aperture.rst |  13 +
 Documentation/driver-api/index.rst    |   1 +
 drivers/gpu/drm/drm_aperture.c        | 174 +------------
 drivers/gpu/drm/tiny/Kconfig          |   1 +
 drivers/video/Kconfig                 |   6 +
 drivers/video/Makefile                |   2 +
 drivers/video/aperture.c              | 340 ++++++++++++++++++++++++++
 drivers/video/console/Kconfig         |   1 +
 drivers/video/fbdev/Kconfig           |   7 +-
 include/linux/aperture.h              |  56 +++++
 10 files changed, 435 insertions(+), 166 deletions(-)
 create mode 100644 Documentation/driver-api/aperture.rst
 create mode 100644 drivers/video/aperture.c
 create mode 100644 include/linux/aperture.h

diff --git a/Documentation/driver-api/aperture.rst b/Documentation/driver-api/aperture.rst
new file mode 100644
index 000000000000..d173f4e7a7d9
--- /dev/null
+++ b/Documentation/driver-api/aperture.rst
@@ -0,0 +1,13 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+Managing Ownership of the Framebuffer Aperture
+==============================================
+
+.. kernel-doc:: drivers/video/aperture.c
+   :doc: overview
+
+.. kernel-doc:: include/linux/aperture.h
+   :internal:
+
+.. kernel-doc:: drivers/video/aperture.c
+   :export:
diff --git a/Documentation/driver-api/index.rst b/Documentation/driver-api/index.rst
index a7b0223e2886..d6d6e77d8afc 100644
--- a/Documentation/driver-api/index.rst
+++ b/Documentation/driver-api/index.rst
@@ -27,6 +27,7 @@ available subsections can be seen below.
    component
    message-based
    infiniband
+   aperture
    frame-buffer
    regulator
    reset
diff --git a/drivers/gpu/drm/drm_aperture.c b/drivers/gpu/drm/drm_aperture.c
index 74bd4a76b253..388a205bd023 100644
--- a/drivers/gpu/drm/drm_aperture.c
+++ b/drivers/gpu/drm/drm_aperture.c
@@ -1,14 +1,7 @@
 // SPDX-License-Identifier: MIT
 
-#include <linux/device.h>
-#include <linux/fb.h>
-#include <linux/list.h>
-#include <linux/mutex.h>
-#include <linux/pci.h>
-#include <linux/platform_device.h> /* for firmware helpers */
-#include <linux/slab.h>
-#include <linux/types.h>
-#include <linux/vgaarb.h>
+#include <linux/aperture.h>
+#include <linux/platform_device.h>
 
 #include <drm/drm_aperture.h>
 #include <drm/drm_drv.h>
@@ -126,92 +119,6 @@
  * afterwards.
  */
 
-struct drm_aperture {
-	struct drm_device *dev;
-	resource_size_t base;
-	resource_size_t size;
-	struct list_head lh;
-	void (*detach)(struct drm_device *dev);
-};
-
-static LIST_HEAD(drm_apertures);
-static DEFINE_MUTEX(drm_apertures_lock);
-
-static bool overlap(resource_size_t base1, resource_size_t end1,
-		    resource_size_t base2, resource_size_t end2)
-{
-	return (base1 < end2) && (end1 > base2);
-}
-
-static void devm_aperture_acquire_release(void *data)
-{
-	struct drm_aperture *ap = data;
-	bool detached = !ap->dev;
-
-	if (detached)
-		return;
-
-	mutex_lock(&drm_apertures_lock);
-	list_del(&ap->lh);
-	mutex_unlock(&drm_apertures_lock);
-}
-
-static int devm_aperture_acquire(struct drm_device *dev,
-				 resource_size_t base, resource_size_t size,
-				 void (*detach)(struct drm_device *))
-{
-	size_t end = base + size;
-	struct list_head *pos;
-	struct drm_aperture *ap;
-
-	mutex_lock(&drm_apertures_lock);
-
-	list_for_each(pos, &drm_apertures) {
-		ap = container_of(pos, struct drm_aperture, lh);
-		if (overlap(base, end, ap->base, ap->base + ap->size)) {
-			mutex_unlock(&drm_apertures_lock);
-			return -EBUSY;
-		}
-	}
-
-	ap = devm_kzalloc(dev->dev, sizeof(*ap), GFP_KERNEL);
-	if (!ap) {
-		mutex_unlock(&drm_apertures_lock);
-		return -ENOMEM;
-	}
-
-	ap->dev = dev;
-	ap->base = base;
-	ap->size = size;
-	ap->detach = detach;
-	INIT_LIST_HEAD(&ap->lh);
-
-	list_add(&ap->lh, &drm_apertures);
-
-	mutex_unlock(&drm_apertures_lock);
-
-	return devm_add_action_or_reset(dev->dev, devm_aperture_acquire_release, ap);
-}
-
-static void drm_aperture_detach_firmware(struct drm_device *dev)
-{
-	struct platform_device *pdev = to_platform_device(dev->dev);
-
-	/*
-	 * Remove the device from the device hierarchy. This is the right thing
-	 * to do for firmware-based DRM drivers, such as EFI, VESA or VGA. After
-	 * the new driver takes over the hardware, the firmware device's state
-	 * will be lost.
-	 *
-	 * For non-platform devices, a new callback would be required.
-	 *
-	 * If the aperture helpers ever need to handle native drivers, this call
-	 * would only have to unplug the DRM device, so that the hardware device
-	 * stays around after detachment.
-	 */
-	platform_device_unregister(pdev);
-}
-
 /**
  * devm_aperture_acquire_from_firmware - Acquires ownership of a firmware framebuffer
  *                                       on behalf of a DRM driver.
@@ -239,39 +146,16 @@ static void drm_aperture_detach_firmware(struct drm_device *dev)
 int devm_aperture_acquire_from_firmware(struct drm_device *dev, resource_size_t base,
 					resource_size_t size)
 {
+	struct platform_device *pdev;
+
 	if (drm_WARN_ON(dev, !dev_is_platform(dev->dev)))
 		return -EINVAL;
 
-	return devm_aperture_acquire(dev, base, size, drm_aperture_detach_firmware);
-}
-EXPORT_SYMBOL(devm_aperture_acquire_from_firmware);
-
-static void drm_aperture_detach_drivers(resource_size_t base, resource_size_t size)
-{
-	resource_size_t end = base + size;
-	struct list_head *pos, *n;
-
-	mutex_lock(&drm_apertures_lock);
-
-	list_for_each_safe(pos, n, &drm_apertures) {
-		struct drm_aperture *ap =
-			container_of(pos, struct drm_aperture, lh);
-		struct drm_device *dev = ap->dev;
-
-		if (WARN_ON_ONCE(!dev))
-			continue;
-
-		if (!overlap(base, end, ap->base, ap->base + ap->size))
-			continue;
+	pdev = to_platform_device(dev->dev);
 
-		ap->dev = NULL; /* detach from device */
-		list_del(&ap->lh);
-
-		ap->detach(dev);
-	}
-
-	mutex_unlock(&drm_apertures_lock);
+	return devm_acquire_aperture_of_platform_device(pdev, base, size);
 }
+EXPORT_SYMBOL(devm_aperture_acquire_from_firmware);
 
 /**
  * drm_aperture_remove_conflicting_framebuffers - remove existing framebuffers in the given range
@@ -289,27 +173,7 @@ static void drm_aperture_detach_drivers(resource_size_t base, resource_size_t si
 int drm_aperture_remove_conflicting_framebuffers(resource_size_t base, resource_size_t size,
 						 bool primary, const struct drm_driver *req_driver)
 {
-#if IS_REACHABLE(CONFIG_FB)
-	struct apertures_struct *a;
-	int ret;
-
-	a = alloc_apertures(1);
-	if (!a)
-		return -ENOMEM;
-
-	a->ranges[0].base = base;
-	a->ranges[0].size = size;
-
-	ret = remove_conflicting_framebuffers(a, req_driver->name, primary);
-	kfree(a);
-
-	if (ret)
-		return ret;
-#endif
-
-	drm_aperture_detach_drivers(base, size);
-
-	return 0;
+	return remove_conflicting_devices(base, size, primary, req_driver->name);
 }
 EXPORT_SYMBOL(drm_aperture_remove_conflicting_framebuffers);
 
@@ -328,26 +192,6 @@ EXPORT_SYMBOL(drm_aperture_remove_conflicting_framebuffers);
 int drm_aperture_remove_conflicting_pci_framebuffers(struct pci_dev *pdev,
 						     const struct drm_driver *req_driver)
 {
-	resource_size_t base, size;
-	int bar, ret = 0;
-
-	for (bar = 0; bar < PCI_STD_NUM_BARS; ++bar) {
-		if (!(pci_resource_flags(pdev, bar) & IORESOURCE_MEM))
-			continue;
-		base = pci_resource_start(pdev, bar);
-		size = pci_resource_len(pdev, bar);
-		drm_aperture_detach_drivers(base, size);
-	}
-
-	/*
-	 * WARNING: Apparently we must kick fbdev drivers before vgacon,
-	 * otherwise the vga fbdev driver falls over.
-	 */
-#if IS_REACHABLE(CONFIG_FB)
-	ret = remove_conflicting_pci_framebuffers(pdev, req_driver->name);
-#endif
-	if (ret == 0)
-		ret = vga_remove_vgacon(pdev);
-	return ret;
+	return remove_conflicting_pci_devices(pdev, req_driver->name);
 }
 EXPORT_SYMBOL(drm_aperture_remove_conflicting_pci_framebuffers);
diff --git a/drivers/gpu/drm/tiny/Kconfig b/drivers/gpu/drm/tiny/Kconfig
index 627d637a1e7e..027cd87c3d0d 100644
--- a/drivers/gpu/drm/tiny/Kconfig
+++ b/drivers/gpu/drm/tiny/Kconfig
@@ -69,6 +69,7 @@ config DRM_PANEL_MIPI_DBI
 config DRM_SIMPLEDRM
 	tristate "Simple framebuffer driver"
 	depends on DRM && MMU
+	select APERTURE_HELPERS
 	select DRM_GEM_SHMEM_HELPER
 	select DRM_KMS_HELPER
 	help
diff --git a/drivers/video/Kconfig b/drivers/video/Kconfig
index 427a993c7f57..c69b45f8c427 100644
--- a/drivers/video/Kconfig
+++ b/drivers/video/Kconfig
@@ -5,6 +5,12 @@
 
 menu "Graphics support"
 
+config APERTURE_HELPERS
+	bool
+	help
+	  Support tracking and hand-over of aperture ownership. Required
+	  for firmware graphics drivers.
+
 if HAS_IOMEM
 
 config HAVE_FB_ATMEL
diff --git a/drivers/video/Makefile b/drivers/video/Makefile
index df7650adede9..5bb6b452cc83 100644
--- a/drivers/video/Makefile
+++ b/drivers/video/Makefile
@@ -1,4 +1,6 @@
 # SPDX-License-Identifier: GPL-2.0
+
+obj-$(CONFIG_APERTURE_HELPERS)    += aperture.o
 obj-$(CONFIG_VGASTATE)            += vgastate.o
 obj-$(CONFIG_HDMI)                += hdmi.o
 
diff --git a/drivers/video/aperture.c b/drivers/video/aperture.c
new file mode 100644
index 000000000000..f0b877e9c256
--- /dev/null
+++ b/drivers/video/aperture.c
@@ -0,0 +1,340 @@
+// SPDX-License-Identifier: MIT
+
+#include <linux/aperture.h>
+#include <linux/device.h>
+#include <linux/fb.h> /* for old fbdev helpers */
+#include <linux/list.h>
+#include <linux/mutex.h>
+#include <linux/pci.h>
+#include <linux/platform_device.h>
+#include <linux/slab.h>
+#include <linux/types.h>
+#include <linux/vgaarb.h>
+
+/**
+ * DOC: overview
+ *
+ * A graphics device might be supported by different drivers, but only one
+ * driver can be active at any given time. Many systems load a generic
+ * graphics drivers, such as EFI-GOP or VESA, early during the boot process.
+ * During later boot stages, they replace the generic driver with a dedicated,
+ * hardware-specific driver. To take over the device the dedicated driver
+ * first has to remove the generic driver. Aperture functions manage
+ * ownership of framebuffer memory and hand-over between drivers.
+ *
+ * Graphics drivers should call remove_conflicting_devices()
+ * at the top of their probe function. The function removes any generic
+ * driver that is currently associated with the given framebuffer memory.
+ * If the framebuffer is located at PCI BAR 0, the rsp code looks as in the
+ * example given below. The cod assumes a DRM driver.
+ *
+ * .. code-block:: c
+ *
+ *	static const struct drm_driver example_driver = {
+ *		.name = "exampledrm",
+ *		...
+ *	};
+ *
+ *	static int remove_conflicting_framebuffers(struct pci_dev *pdev)
+ *	{
+ *		bool primary = false;
+ *		resource_size_t base, size;
+ *		int ret;
+ *
+ *		base = pci_resource_start(pdev, 0);
+ *		size = pci_resource_len(pdev, 0);
+ *	#ifdef CONFIG_X86
+ *		primary = pdev->resource[PCI_ROM_RESOURCE].flags & IORESOURCE_ROM_SHADOW;
+ *	#endif
+ *
+ *		return remove_conflicting_devices(base, size, primary, &example_driver->name);
+ *	}
+ *
+ *	static int probe(struct pci_dev *pdev)
+ *	{
+ *		int ret;
+ *
+ *		// Remove any generic drivers...
+ *		ret = remove_conflicting_framebuffers(pdev);
+ *		if (ret)
+ *			return ret;
+ *
+ *		// ... and initialize the hardware.
+ *		...
+ *
+ *		drm_dev_register();
+ *
+ *		return 0;
+ *	}
+ *
+ * PCI device drivers can also call remove_conflicting_pci_devices() and let the
+ * function detect the apertures automatically. Device drivers without knowledge of
+ * the framebuffer's location shall call remove_all_conflicting_devices(),
+ * which removes all known devices.
+ *
+ * Drivers that are susceptible to being removed by other drivers, such as
+ * generic EFI or VESA drivers, have to register themselves as owners of their
+ * framebuffer apertures. Ownership of the framebuffer memory is achieved
+ * by calling devm_acquire_aperture_of_platform_device(). On success, the driver
+ * is the owner of the framebuffer range. The function fails if the
+ * framebuffer is already owned by another driver. See below for an example.
+ *
+ * .. code-block:: c
+ *
+ *	static int acquire_framebuffers(struct drm_device *dev, struct platform_device *pdev)
+ *	{
+ *		resource_size_t base, size;
+ *
+ *		mem = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ *		if (!mem)
+ *			return -EINVAL;
+ *		base = mem->start;
+ *		size = resource_size(mem);
+ *
+ *		return devm_acquire_aperture_of_platform_device(pdev, base, size);
+ *	}
+ *
+ *	static int probe(struct platform_device *pdev)
+ *	{
+ *		struct drm_device *dev;
+ *		int ret;
+ *
+ *		// ... Initialize the device...
+ *		dev = devm_drm_dev_alloc();
+ *		...
+ *
+ *		// ... and acquire ownership of the framebuffer.
+ *		ret = acquire_framebuffers(dev, pdev);
+ *		if (ret)
+ *			return ret;
+ *
+ *		drm_dev_register(dev, 0);
+ *
+ *		return 0;
+ *	}
+ *
+ * The generic driver is now subject to forced removal by other drivers. This
+ * only works for platform drivers that support hot unplugging.
+ * When a driver calls remove_conflicting_devices() et al
+ * for the registered framebuffer range, the aperture helpers call
+ * platform_device_unregister() and the generic driver unloads itself. It
+ * may not access the device's registers, framebuffer memory, ROM, etc
+ * afterwards.
+ */
+
+struct dev_aperture {
+	struct device *dev;
+	resource_size_t base;
+	resource_size_t size;
+	struct list_head lh;
+	void (*detach)(struct device *dev);
+};
+
+static LIST_HEAD(apertures);
+static DEFINE_MUTEX(apertures_lock);
+
+static bool overlap(resource_size_t base1, resource_size_t end1,
+		    resource_size_t base2, resource_size_t end2)
+{
+	return (base1 < end2) && (end1 > base2);
+}
+
+static void devm_aperture_acquire_release(void *data)
+{
+	struct dev_aperture *ap = data;
+	bool detached = !ap->dev;
+
+	if (detached)
+		return;
+
+	mutex_lock(&apertures_lock);
+	list_del(&ap->lh);
+	mutex_unlock(&apertures_lock);
+}
+
+static int devm_aperture_acquire(struct device *dev,
+				 resource_size_t base, resource_size_t size,
+				 void (*detach)(struct device *))
+{
+	size_t end = base + size;
+	struct list_head *pos;
+	struct dev_aperture *ap;
+
+	mutex_lock(&apertures_lock);
+
+	list_for_each(pos, &apertures) {
+		ap = container_of(pos, struct dev_aperture, lh);
+		if (overlap(base, end, ap->base, ap->base + ap->size)) {
+			mutex_unlock(&apertures_lock);
+			return -EBUSY;
+		}
+	}
+
+	ap = devm_kzalloc(dev, sizeof(*ap), GFP_KERNEL);
+	if (!ap) {
+		mutex_unlock(&apertures_lock);
+		return -ENOMEM;
+	}
+
+	ap->dev = dev;
+	ap->base = base;
+	ap->size = size;
+	ap->detach = detach;
+	INIT_LIST_HEAD(&ap->lh);
+
+	list_add(&ap->lh, &apertures);
+
+	mutex_unlock(&apertures_lock);
+
+	return devm_add_action_or_reset(dev, devm_aperture_acquire_release, ap);
+}
+
+static void detach_platform_device(struct device *dev)
+{
+	struct platform_device *pdev = to_platform_device(dev);
+
+	/*
+	 * Remove the device from the device hierarchy. This is the right thing
+	 * to do for firmware-based DRM drivers, such as EFI, VESA or VGA. After
+	 * the new driver takes over the hardware, the firmware device's state
+	 * will be lost.
+	 *
+	 * For non-platform devices, a new callback would be required.
+	 *
+	 * If the aperture helpers ever need to handle native drivers, this call
+	 * would only have to unplug the DRM device, so that the hardware device
+	 * stays around after detachment.
+	 */
+	platform_device_unregister(pdev);
+}
+
+/**
+ * devm_acquire_aperture_of_platform_device - Acquires ownership of an aperture
+ *                                            on behalf of a platform device.
+ * @pdev:	the platform device to own the aperture
+ * @base:	the aperture's byte offset in physical memory
+ * @size:	the aperture size in bytes
+ *
+ * Installs the given device as the new owner of the aperture. The function
+ * expects the aperture to be provided by a platform device. If another
+ * driver takes over ownership of the aperture, aperture helpers will then
+ * unregister the platform device automatically. All acquired apertures are
+ * released automatically when the underlying device goes away.
+ *
+ * The function fails if the aperture, or parts of it, is currently
+ * owned by another device. To evict current owners, callers should use
+ * remove_conflicting_devices() et al. before calling this function.
+ *
+ * Returns:
+ * 0 on success, or a negative errno value otherwise.
+ */
+int devm_acquire_aperture_of_platform_device(struct platform_device *pdev,
+					     resource_size_t base,
+					     resource_size_t size)
+{
+	return devm_aperture_acquire(&pdev->dev, base, size, detach_platform_device);
+}
+EXPORT_SYMBOL(devm_acquire_aperture_of_platform_device);
+
+static void detach_devices(resource_size_t base, resource_size_t size)
+{
+	resource_size_t end = base + size;
+	struct list_head *pos, *n;
+
+	mutex_lock(&apertures_lock);
+
+	list_for_each_safe(pos, n, &apertures) {
+		struct dev_aperture *ap = container_of(pos, struct dev_aperture, lh);
+		struct device *dev = ap->dev;
+
+		if (WARN_ON_ONCE(!dev))
+			continue;
+
+		if (!overlap(base, end, ap->base, ap->base + ap->size))
+			continue;
+
+		ap->dev = NULL; /* detach from device */
+		list_del(&ap->lh);
+
+		ap->detach(dev);
+	}
+
+	mutex_unlock(&apertures_lock);
+}
+
+/**
+ * remove_conflicting_devices - remove devices in the given range
+ * @base: the aperture's base address in physical memory
+ * @size: aperture size in bytes
+ * @primary: also kick vga16fb if present; only relevant for VGA devices
+ * @name: a descriptive name of the requesting driver
+ *
+ * This function removes devices that own apertures within @base and @size.
+ *
+ * Returns:
+ * 0 on success, or a negative errno code otherwise
+ */
+int remove_conflicting_devices(resource_size_t base, resource_size_t size, bool primary,
+			       const char *name)
+{
+#if IS_REACHABLE(CONFIG_FB)
+	struct apertures_struct *a;
+	int ret;
+
+	a = alloc_apertures(1);
+	if (!a)
+		return -ENOMEM;
+
+	a->ranges[0].base = base;
+	a->ranges[0].size = size;
+
+	ret = remove_conflicting_framebuffers(a, name, primary);
+	kfree(a);
+
+	if (ret)
+		return ret;
+#endif
+
+	detach_devices(base, size);
+
+	return 0;
+}
+EXPORT_SYMBOL(remove_conflicting_devices);
+
+/**
+ * remove_conflicting_pci_devices - remove existing framebuffers for PCI devices
+ * @pdev: PCI device
+ * @name: a descriptive name of the requesting driver
+ *
+ * This function removes devices that own apertures within any of @pdev's
+ * memory bars. The function assumes that PCI device with shadowed ROM
+ * drives a primary display and therefore kicks out vga16fb as well.
+ *
+ * Returns:
+ * 0 on success, or a negative errno code otherwise
+ */
+int remove_conflicting_pci_devices(struct pci_dev *pdev, const char *name)
+{
+	resource_size_t base, size;
+	int bar, ret = 0;
+
+	for (bar = 0; bar < PCI_STD_NUM_BARS; ++bar) {
+		if (!(pci_resource_flags(pdev, bar) & IORESOURCE_MEM))
+			continue;
+		base = pci_resource_start(pdev, bar);
+		size = pci_resource_len(pdev, bar);
+		detach_devices(base, size);
+	}
+
+	/*
+	 * WARNING: Apparently we must kick fbdev drivers before vgacon,
+	 * otherwise the vga fbdev driver falls over.
+	 */
+#if IS_REACHABLE(CONFIG_FB)
+	ret = remove_conflicting_pci_framebuffers(pdev, name);
+#endif
+	if (ret == 0)
+		ret = vga_remove_vgacon(pdev);
+	return ret;
+}
+EXPORT_SYMBOL(remove_conflicting_pci_devices);
diff --git a/drivers/video/console/Kconfig b/drivers/video/console/Kconfig
index 40c50fa2dd70..7f3c44e1538b 100644
--- a/drivers/video/console/Kconfig
+++ b/drivers/video/console/Kconfig
@@ -10,6 +10,7 @@ config VGA_CONSOLE
 	depends on !4xx && !PPC_8xx && !SPARC && !M68K && !PARISC &&  !SUPERH && \
 		(!ARM || ARCH_FOOTBRIDGE || ARCH_INTEGRATOR || ARCH_NETWINDER) && \
 		!ARM64 && !ARC && !MICROBLAZE && !OPENRISC && !S390 && !UML
+	select APERTURE_HELPERS if (DRM || FB || VFIO_PCI)
 	default y
 	help
 	  Saying Y here will allow you to use Linux in text mode through a
diff --git a/drivers/video/fbdev/Kconfig b/drivers/video/fbdev/Kconfig
index bd849013f16f..7b398e7e3cc8 100644
--- a/drivers/video/fbdev/Kconfig
+++ b/drivers/video/fbdev/Kconfig
@@ -455,6 +455,7 @@ config FB_ATARI
 config FB_OF
 	bool "Open Firmware frame buffer device support"
 	depends on (FB = y) && PPC && (!PPC_PSERIES || PCI)
+	select APERTURE_HELPERS
 	select FB_CFB_FILLRECT
 	select FB_CFB_COPYAREA
 	select FB_CFB_IMAGEBLIT
@@ -527,6 +528,7 @@ config FB_IMSTT
 config FB_VGA16
 	tristate "VGA 16-color graphics support"
 	depends on FB && (X86 || PPC)
+	select APERTURE_HELPERS
 	select FB_CFB_FILLRECT
 	select FB_CFB_COPYAREA
 	select FB_CFB_IMAGEBLIT
@@ -551,7 +553,7 @@ config FB_STI
 	  BIOS routines contained in a ROM chip in HP PA-RISC based machines.
 	  Enabling this option will implement the linux framebuffer device
 	  using calls to the STI BIOS routines for initialisation.
-	
+
 	  If you enable this option, you will get a planar framebuffer device
 	  /dev/fb which will work on the most common HP graphic cards of the
 	  NGLE family, including the artist chips (in the 7xx and Bxxx series),
@@ -617,6 +619,7 @@ config FB_UVESA
 config FB_VESA
 	bool "VESA VGA graphics support"
 	depends on (FB = y) && (X86 || COMPILE_TEST)
+	select APERTURE_HELPERS
 	select FB_CFB_FILLRECT
 	select FB_CFB_COPYAREA
 	select FB_CFB_IMAGEBLIT
@@ -630,6 +633,7 @@ config FB_VESA
 config FB_EFI
 	bool "EFI-based Framebuffer Support"
 	depends on (FB = y) && !IA64 && EFI
+	select APERTURE_HELPERS
 	select DRM_PANEL_ORIENTATION_QUIRKS
 	select FB_CFB_FILLRECT
 	select FB_CFB_COPYAREA
@@ -2190,6 +2194,7 @@ config FB_SIMPLE
 	tristate "Simple framebuffer support"
 	depends on FB
 	depends on !DRM_SIMPLEDRM
+	select APERTURE_HELPERS
 	select FB_CFB_FILLRECT
 	select FB_CFB_COPYAREA
 	select FB_CFB_IMAGEBLIT
diff --git a/include/linux/aperture.h b/include/linux/aperture.h
new file mode 100644
index 000000000000..0a206cdce606
--- /dev/null
+++ b/include/linux/aperture.h
@@ -0,0 +1,56 @@
+/* SPDX-License-Identifier: MIT */
+
+#ifndef _LINUX_APERTURE_H_
+#define _LINUX_APERTURE_H_
+
+#include <linux/types.h>
+
+struct pci_dev;
+struct platform_device;
+
+#if defined(CONFIG_APERTURE_HELPERS)
+int devm_acquire_aperture_of_platform_device(struct platform_device *pdev,
+					     resource_size_t base,
+					     resource_size_t size);
+
+int remove_conflicting_devices(resource_size_t base, resource_size_t size,
+			       bool primary, const char *name);
+
+int remove_conflicting_pci_devices(struct pci_dev *pdev, const char *name);
+#else
+static inline int devm_acquire_aperture_of_platform_device(struct platform_device *pdev,
+							   resource_size_t base,
+							   resource_size_t size)
+{
+	return 0;
+}
+
+static inline int remove_conflicting_devices(resource_size_t base, resource_size_t size,
+					     bool primary, const char *name)
+{
+	return 0;
+}
+
+static inline int remove_conflicting_pci_devices(struct pci_dev *pdev, const char *name)
+{
+	return 0;
+}
+#endif
+
+/**
+ * remove_all_conflicting_devices - remove all existing framebuffers
+ * @primary: also kick vga16fb if present; only relevant for VGA devices
+ * @name: a descriptive name of the requesting driver
+ *
+ * This function removes all graphics device drivers. Use this function on systems
+ * that can have their framebuffer located anywhere in memory.
+ *
+ * Returns:
+ * 0 on success, or a negative errno code otherwise
+ */
+static inline int remove_all_conflicting_devices(bool primary, const char *name)
+{
+	return remove_conflicting_devices(0, (resource_size_t)-1, primary, name);
+}
+
+#endif
-- 
2.36.1


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]

Re: [PATCH 2/2] vfio/pci: Remove console drivers

[Alex Williamson]

On Thu, 9 Jun 2022 11:13:22 +0200
Thomas Zimmermann <tzimmermann@suse.de> wrote:
> 
> Please have a look at the attached patch. It moves the aperture helpers 
> to a location common to the various possible users (DRM, fbdev, vfio). 
> The DRM interfaces remain untouched for now.  The patch should provide 
> what you need in vfio and also serve our future use cases for graphics 
> drivers. If possible, please create your patch on top of it.

Looks good to me, this of course makes the vfio change quite trivial.
One change I'd request:

diff --git a/drivers/video/console/Kconfig b/drivers/video/console/Kconfig
index 40c50fa2dd70..7f3c44e1538b 100644
--- a/drivers/video/console/Kconfig
+++ b/drivers/video/console/Kconfig
@@ -10,6 +10,7 @@ config VGA_CONSOLE
 	depends on !4xx && !PPC_8xx && !SPARC && !M68K && !PARISC &&  !SUPERH && \
 		(!ARM || ARCH_FOOTBRIDGE || ARCH_INTEGRATOR || ARCH_NETWINDER) && \
 		!ARM64 && !ARC && !MICROBLAZE && !OPENRISC && !S390 && !UML
+	select APERTURE_HELPERS if (DRM || FB || VFIO_PCI)
 	default y
 	help
 	  Saying Y here will allow you to use Linux in text mode through a

This should be VFIO_PCI_CORE.  Thanks,

Alex

Re: [PATCH 2/2] vfio/pci: Remove console drivers

[Alex Williamson]

On Thu, 9 Jun 2022 11:13:22 +0200
Thomas Zimmermann <tzimmermann@suse.de> wrote:
> 
> Please have a look at the attached patch. It moves the aperture helpers 
> to a location common to the various possible users (DRM, fbdev, vfio). 
> The DRM interfaces remain untouched for now.  The patch should provide 
> what you need in vfio and also serve our future use cases for graphics 
> drivers. If possible, please create your patch on top of it.

Looks good to me, this of course makes the vfio change quite trivial.
One change I'd request:

diff --git a/drivers/video/console/Kconfig b/drivers/video/console/Kconfig
index 40c50fa2dd70..7f3c44e1538b 100644
--- a/drivers/video/console/Kconfig
+++ b/drivers/video/console/Kconfig
@@ -10,6 +10,7 @@ config VGA_CONSOLE
 	depends on !4xx && !PPC_8xx && !SPARC && !M68K && !PARISC &&  !SUPERH && \
 		(!ARM || ARCH_FOOTBRIDGE || ARCH_INTEGRATOR || ARCH_NETWINDER) && \
 		!ARM64 && !ARC && !MICROBLAZE && !OPENRISC && !S390 && !UML
+	select APERTURE_HELPERS if (DRM || FB || VFIO_PCI)
 	default y
 	help
 	  Saying Y here will allow you to use Linux in text mode through a

This should be VFIO_PCI_CORE.  Thanks,

Alex

Re: [PATCH 2/2] vfio/pci: Remove console drivers

[Alex Williamson]

On Thu, 9 Jun 2022 15:41:02 -0600
Alex Williamson <alex.williamson@redhat.com> wrote:

> On Thu, 9 Jun 2022 11:13:22 +0200
> Thomas Zimmermann <tzimmermann@suse.de> wrote:
> > 
> > Please have a look at the attached patch. It moves the aperture helpers 
> > to a location common to the various possible users (DRM, fbdev, vfio). 
> > The DRM interfaces remain untouched for now.  The patch should provide 
> > what you need in vfio and also serve our future use cases for graphics 
> > drivers. If possible, please create your patch on top of it.  
> 
> Looks good to me, this of course makes the vfio change quite trivial.
> One change I'd request:
> 
> diff --git a/drivers/video/console/Kconfig b/drivers/video/console/Kconfig
> index 40c50fa2dd70..7f3c44e1538b 100644
> --- a/drivers/video/console/Kconfig
> +++ b/drivers/video/console/Kconfig
> @@ -10,6 +10,7 @@ config VGA_CONSOLE
>  	depends on !4xx && !PPC_8xx && !SPARC && !M68K && !PARISC &&  !SUPERH && \
>  		(!ARM || ARCH_FOOTBRIDGE || ARCH_INTEGRATOR || ARCH_NETWINDER) && \
>  		!ARM64 && !ARC && !MICROBLAZE && !OPENRISC && !S390 && !UML
> +	select APERTURE_HELPERS if (DRM || FB || VFIO_PCI)
>  	default y
>  	help
>  	  Saying Y here will allow you to use Linux in text mode through a
> 
> This should be VFIO_PCI_CORE.  Thanks,

Also, whatever tree this lands in, I'd appreciate a topic branch being
made available so I can more easily get the vfio change in on the same
release.  Thanks,

Alex

Re: [PATCH 2/2] vfio/pci: Remove console drivers

[Alex Williamson]

On Thu, 9 Jun 2022 15:41:02 -0600
Alex Williamson <alex.williamson@redhat.com> wrote:

> On Thu, 9 Jun 2022 11:13:22 +0200
> Thomas Zimmermann <tzimmermann@suse.de> wrote:
> > 
> > Please have a look at the attached patch. It moves the aperture helpers 
> > to a location common to the various possible users (DRM, fbdev, vfio). 
> > The DRM interfaces remain untouched for now.  The patch should provide 
> > what you need in vfio and also serve our future use cases for graphics 
> > drivers. If possible, please create your patch on top of it.  
> 
> Looks good to me, this of course makes the vfio change quite trivial.
> One change I'd request:
> 
> diff --git a/drivers/video/console/Kconfig b/drivers/video/console/Kconfig
> index 40c50fa2dd70..7f3c44e1538b 100644
> --- a/drivers/video/console/Kconfig
> +++ b/drivers/video/console/Kconfig
> @@ -10,6 +10,7 @@ config VGA_CONSOLE
>  	depends on !4xx && !PPC_8xx && !SPARC && !M68K && !PARISC &&  !SUPERH && \
>  		(!ARM || ARCH_FOOTBRIDGE || ARCH_INTEGRATOR || ARCH_NETWINDER) && \
>  		!ARM64 && !ARC && !MICROBLAZE && !OPENRISC && !S390 && !UML
> +	select APERTURE_HELPERS if (DRM || FB || VFIO_PCI)
>  	default y
>  	help
>  	  Saying Y here will allow you to use Linux in text mode through a
> 
> This should be VFIO_PCI_CORE.  Thanks,

Also, whatever tree this lands in, I'd appreciate a topic branch being
made available so I can more easily get the vfio change in on the same
release.  Thanks,

Alex

Re: [PATCH 2/2] vfio/pci: Remove console drivers

[Thomas Zimmermann]

[-- Attachment #1.1.1: Type: text/plain, Size: 2087 bytes --]

Hi

Am 09.06.22 um 23:44 schrieb Alex Williamson:
> On Thu, 9 Jun 2022 15:41:02 -0600
> Alex Williamson <alex.williamson@redhat.com> wrote:
> 
>> On Thu, 9 Jun 2022 11:13:22 +0200
>> Thomas Zimmermann <tzimmermann@suse.de> wrote:
>>>
>>> Please have a look at the attached patch. It moves the aperture helpers
>>> to a location common to the various possible users (DRM, fbdev, vfio).
>>> The DRM interfaces remain untouched for now.  The patch should provide
>>> what you need in vfio and also serve our future use cases for graphics
>>> drivers. If possible, please create your patch on top of it.
>>
>> Looks good to me, this of course makes the vfio change quite trivial.
>> One change I'd request:
>>
>> diff --git a/drivers/video/console/Kconfig b/drivers/video/console/Kconfig
>> index 40c50fa2dd70..7f3c44e1538b 100644
>> --- a/drivers/video/console/Kconfig
>> +++ b/drivers/video/console/Kconfig
>> @@ -10,6 +10,7 @@ config VGA_CONSOLE
>>   	depends on !4xx && !PPC_8xx && !SPARC && !M68K && !PARISC &&  !SUPERH && \
>>   		(!ARM || ARCH_FOOTBRIDGE || ARCH_INTEGRATOR || ARCH_NETWINDER) && \
>>   		!ARM64 && !ARC && !MICROBLAZE && !OPENRISC && !S390 && !UML
>> +	select APERTURE_HELPERS if (DRM || FB || VFIO_PCI)
>>   	default y
>>   	help
>>   	  Saying Y here will allow you to use Linux in text mode through a
>>
>> This should be VFIO_PCI_CORE.  Thanks,

I attached an updated patch to this email.

> 
> Also, whatever tree this lands in, I'd appreciate a topic branch being
> made available so I can more easily get the vfio change in on the same
> release.  Thanks,

You can add my patch to your series and merge it through vfio. You'd 
only have to cc dri-devel for the patch's review. I guess it's more 
important for vfio than DRM. We have no hurry on the DRM side, but v5.20 
would be nice.

Best regards
Thomas

> 
> Alex
> 

-- 
Thomas Zimmermann
Graphics Driver Developer
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5, 90409 Nürnberg, Germany
(HRB 36809, AG Nürnberg)
Geschäftsführer: Ivo Totev

[-- Attachment #1.1.2: 0001-drm-Implement-DRM-aperture-helpers-under-video.patch --]
[-- Type: text/x-patch, Size: 24436 bytes --]

From 5e0293b7102c0772568ca490ef53b7e8775ed761 Mon Sep 17 00:00:00 2001
From: Thomas Zimmermann <tzimmermann@suse.de>
Date: Wed, 8 Jun 2022 20:52:50 +0200
Subject: [PATCH] drm: Implement DRM aperture helpers under video/

Implement DRM's aperture helpers under video/ for sharing with other
sub-systems. Remove DRM-isms from the interface. The helpers track
the ownership of framebuffer apertures and provide hand-over from
firmware, such as EFI and VESA, to native graphics drivers.

Other subsystems, such as fbdev and vfio, also have to maintain ownership
of framebuffer apertures. Moving DRM's aperture helpers to a more public
location allows all subsystems to interact with each other and share a
common implementation.

The aperture helpers are selected by the various firmware drivers within
DRM and fbdev, and the VGA text-console driver.

The original DRM interface is kept in place for use by DRM drivers.

Signed-off-by: Thomas Zimmermann <tzimmermann@suse.de>
---
 Documentation/driver-api/aperture.rst |  13 +
 Documentation/driver-api/index.rst    |   1 +
 drivers/gpu/drm/drm_aperture.c        | 174 +------------
 drivers/gpu/drm/tiny/Kconfig          |   1 +
 drivers/video/Kconfig                 |   6 +
 drivers/video/Makefile                |   2 +
 drivers/video/aperture.c              | 340 ++++++++++++++++++++++++++
 drivers/video/console/Kconfig         |   1 +
 drivers/video/fbdev/Kconfig           |   7 +-
 include/linux/aperture.h              |  56 +++++
 10 files changed, 435 insertions(+), 166 deletions(-)
 create mode 100644 Documentation/driver-api/aperture.rst
 create mode 100644 drivers/video/aperture.c
 create mode 100644 include/linux/aperture.h

diff --git a/Documentation/driver-api/aperture.rst b/Documentation/driver-api/aperture.rst
new file mode 100644
index 000000000000..d173f4e7a7d9
--- /dev/null
+++ b/Documentation/driver-api/aperture.rst
@@ -0,0 +1,13 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+Managing Ownership of the Framebuffer Aperture
+==============================================
+
+.. kernel-doc:: drivers/video/aperture.c
+   :doc: overview
+
+.. kernel-doc:: include/linux/aperture.h
+   :internal:
+
+.. kernel-doc:: drivers/video/aperture.c
+   :export:
diff --git a/Documentation/driver-api/index.rst b/Documentation/driver-api/index.rst
index a7b0223e2886..d6d6e77d8afc 100644
--- a/Documentation/driver-api/index.rst
+++ b/Documentation/driver-api/index.rst
@@ -27,6 +27,7 @@ available subsections can be seen below.
    component
    message-based
    infiniband
+   aperture
    frame-buffer
    regulator
    reset
diff --git a/drivers/gpu/drm/drm_aperture.c b/drivers/gpu/drm/drm_aperture.c
index 74bd4a76b253..388a205bd023 100644
--- a/drivers/gpu/drm/drm_aperture.c
+++ b/drivers/gpu/drm/drm_aperture.c
@@ -1,14 +1,7 @@
 // SPDX-License-Identifier: MIT
 
-#include <linux/device.h>
-#include <linux/fb.h>
-#include <linux/list.h>
-#include <linux/mutex.h>
-#include <linux/pci.h>
-#include <linux/platform_device.h> /* for firmware helpers */
-#include <linux/slab.h>
-#include <linux/types.h>
-#include <linux/vgaarb.h>
+#include <linux/aperture.h>
+#include <linux/platform_device.h>
 
 #include <drm/drm_aperture.h>
 #include <drm/drm_drv.h>
@@ -126,92 +119,6 @@
  * afterwards.
  */
 
-struct drm_aperture {
-	struct drm_device *dev;
-	resource_size_t base;
-	resource_size_t size;
-	struct list_head lh;
-	void (*detach)(struct drm_device *dev);
-};
-
-static LIST_HEAD(drm_apertures);
-static DEFINE_MUTEX(drm_apertures_lock);
-
-static bool overlap(resource_size_t base1, resource_size_t end1,
-		    resource_size_t base2, resource_size_t end2)
-{
-	return (base1 < end2) && (end1 > base2);
-}
-
-static void devm_aperture_acquire_release(void *data)
-{
-	struct drm_aperture *ap = data;
-	bool detached = !ap->dev;
-
-	if (detached)
-		return;
-
-	mutex_lock(&drm_apertures_lock);
-	list_del(&ap->lh);
-	mutex_unlock(&drm_apertures_lock);
-}
-
-static int devm_aperture_acquire(struct drm_device *dev,
-				 resource_size_t base, resource_size_t size,
-				 void (*detach)(struct drm_device *))
-{
-	size_t end = base + size;
-	struct list_head *pos;
-	struct drm_aperture *ap;
-
-	mutex_lock(&drm_apertures_lock);
-
-	list_for_each(pos, &drm_apertures) {
-		ap = container_of(pos, struct drm_aperture, lh);
-		if (overlap(base, end, ap->base, ap->base + ap->size)) {
-			mutex_unlock(&drm_apertures_lock);
-			return -EBUSY;
-		}
-	}
-
-	ap = devm_kzalloc(dev->dev, sizeof(*ap), GFP_KERNEL);
-	if (!ap) {
-		mutex_unlock(&drm_apertures_lock);
-		return -ENOMEM;
-	}
-
-	ap->dev = dev;
-	ap->base = base;
-	ap->size = size;
-	ap->detach = detach;
-	INIT_LIST_HEAD(&ap->lh);
-
-	list_add(&ap->lh, &drm_apertures);
-
-	mutex_unlock(&drm_apertures_lock);
-
-	return devm_add_action_or_reset(dev->dev, devm_aperture_acquire_release, ap);
-}
-
-static void drm_aperture_detach_firmware(struct drm_device *dev)
-{
-	struct platform_device *pdev = to_platform_device(dev->dev);
-
-	/*
-	 * Remove the device from the device hierarchy. This is the right thing
-	 * to do for firmware-based DRM drivers, such as EFI, VESA or VGA. After
-	 * the new driver takes over the hardware, the firmware device's state
-	 * will be lost.
-	 *
-	 * For non-platform devices, a new callback would be required.
-	 *
-	 * If the aperture helpers ever need to handle native drivers, this call
-	 * would only have to unplug the DRM device, so that the hardware device
-	 * stays around after detachment.
-	 */
-	platform_device_unregister(pdev);
-}
-
 /**
  * devm_aperture_acquire_from_firmware - Acquires ownership of a firmware framebuffer
  *                                       on behalf of a DRM driver.
@@ -239,39 +146,16 @@ static void drm_aperture_detach_firmware(struct drm_device *dev)
 int devm_aperture_acquire_from_firmware(struct drm_device *dev, resource_size_t base,
 					resource_size_t size)
 {
+	struct platform_device *pdev;
+
 	if (drm_WARN_ON(dev, !dev_is_platform(dev->dev)))
 		return -EINVAL;
 
-	return devm_aperture_acquire(dev, base, size, drm_aperture_detach_firmware);
-}
-EXPORT_SYMBOL(devm_aperture_acquire_from_firmware);
-
-static void drm_aperture_detach_drivers(resource_size_t base, resource_size_t size)
-{
-	resource_size_t end = base + size;
-	struct list_head *pos, *n;
-
-	mutex_lock(&drm_apertures_lock);
-
-	list_for_each_safe(pos, n, &drm_apertures) {
-		struct drm_aperture *ap =
-			container_of(pos, struct drm_aperture, lh);
-		struct drm_device *dev = ap->dev;
-
-		if (WARN_ON_ONCE(!dev))
-			continue;
-
-		if (!overlap(base, end, ap->base, ap->base + ap->size))
-			continue;
+	pdev = to_platform_device(dev->dev);
 
-		ap->dev = NULL; /* detach from device */
-		list_del(&ap->lh);
-
-		ap->detach(dev);
-	}
-
-	mutex_unlock(&drm_apertures_lock);
+	return devm_acquire_aperture_of_platform_device(pdev, base, size);
 }
+EXPORT_SYMBOL(devm_aperture_acquire_from_firmware);
 
 /**
  * drm_aperture_remove_conflicting_framebuffers - remove existing framebuffers in the given range
@@ -289,27 +173,7 @@ static void drm_aperture_detach_drivers(resource_size_t base, resource_size_t si
 int drm_aperture_remove_conflicting_framebuffers(resource_size_t base, resource_size_t size,
 						 bool primary, const struct drm_driver *req_driver)
 {
-#if IS_REACHABLE(CONFIG_FB)
-	struct apertures_struct *a;
-	int ret;
-
-	a = alloc_apertures(1);
-	if (!a)
-		return -ENOMEM;
-
-	a->ranges[0].base = base;
-	a->ranges[0].size = size;
-
-	ret = remove_conflicting_framebuffers(a, req_driver->name, primary);
-	kfree(a);
-
-	if (ret)
-		return ret;
-#endif
-
-	drm_aperture_detach_drivers(base, size);
-
-	return 0;
+	return remove_conflicting_devices(base, size, primary, req_driver->name);
 }
 EXPORT_SYMBOL(drm_aperture_remove_conflicting_framebuffers);
 
@@ -328,26 +192,6 @@ EXPORT_SYMBOL(drm_aperture_remove_conflicting_framebuffers);
 int drm_aperture_remove_conflicting_pci_framebuffers(struct pci_dev *pdev,
 						     const struct drm_driver *req_driver)
 {
-	resource_size_t base, size;
-	int bar, ret = 0;
-
-	for (bar = 0; bar < PCI_STD_NUM_BARS; ++bar) {
-		if (!(pci_resource_flags(pdev, bar) & IORESOURCE_MEM))
-			continue;
-		base = pci_resource_start(pdev, bar);
-		size = pci_resource_len(pdev, bar);
-		drm_aperture_detach_drivers(base, size);
-	}
-
-	/*
-	 * WARNING: Apparently we must kick fbdev drivers before vgacon,
-	 * otherwise the vga fbdev driver falls over.
-	 */
-#if IS_REACHABLE(CONFIG_FB)
-	ret = remove_conflicting_pci_framebuffers(pdev, req_driver->name);
-#endif
-	if (ret == 0)
-		ret = vga_remove_vgacon(pdev);
-	return ret;
+	return remove_conflicting_pci_devices(pdev, req_driver->name);
 }
 EXPORT_SYMBOL(drm_aperture_remove_conflicting_pci_framebuffers);
diff --git a/drivers/gpu/drm/tiny/Kconfig b/drivers/gpu/drm/tiny/Kconfig
index 627d637a1e7e..027cd87c3d0d 100644
--- a/drivers/gpu/drm/tiny/Kconfig
+++ b/drivers/gpu/drm/tiny/Kconfig
@@ -69,6 +69,7 @@ config DRM_PANEL_MIPI_DBI
 config DRM_SIMPLEDRM
 	tristate "Simple framebuffer driver"
 	depends on DRM && MMU
+	select APERTURE_HELPERS
 	select DRM_GEM_SHMEM_HELPER
 	select DRM_KMS_HELPER
 	help
diff --git a/drivers/video/Kconfig b/drivers/video/Kconfig
index 427a993c7f57..c69b45f8c427 100644
--- a/drivers/video/Kconfig
+++ b/drivers/video/Kconfig
@@ -5,6 +5,12 @@
 
 menu "Graphics support"
 
+config APERTURE_HELPERS
+	bool
+	help
+	  Support tracking and hand-over of aperture ownership. Required
+	  for firmware graphics drivers.
+
 if HAS_IOMEM
 
 config HAVE_FB_ATMEL
diff --git a/drivers/video/Makefile b/drivers/video/Makefile
index df7650adede9..5bb6b452cc83 100644
--- a/drivers/video/Makefile
+++ b/drivers/video/Makefile
@@ -1,4 +1,6 @@
 # SPDX-License-Identifier: GPL-2.0
+
+obj-$(CONFIG_APERTURE_HELPERS)    += aperture.o
 obj-$(CONFIG_VGASTATE)            += vgastate.o
 obj-$(CONFIG_HDMI)                += hdmi.o
 
diff --git a/drivers/video/aperture.c b/drivers/video/aperture.c
new file mode 100644
index 000000000000..f0b877e9c256
--- /dev/null
+++ b/drivers/video/aperture.c
@@ -0,0 +1,340 @@
+// SPDX-License-Identifier: MIT
+
+#include <linux/aperture.h>
+#include <linux/device.h>
+#include <linux/fb.h> /* for old fbdev helpers */
+#include <linux/list.h>
+#include <linux/mutex.h>
+#include <linux/pci.h>
+#include <linux/platform_device.h>
+#include <linux/slab.h>
+#include <linux/types.h>
+#include <linux/vgaarb.h>
+
+/**
+ * DOC: overview
+ *
+ * A graphics device might be supported by different drivers, but only one
+ * driver can be active at any given time. Many systems load a generic
+ * graphics drivers, such as EFI-GOP or VESA, early during the boot process.
+ * During later boot stages, they replace the generic driver with a dedicated,
+ * hardware-specific driver. To take over the device the dedicated driver
+ * first has to remove the generic driver. Aperture functions manage
+ * ownership of framebuffer memory and hand-over between drivers.
+ *
+ * Graphics drivers should call remove_conflicting_devices()
+ * at the top of their probe function. The function removes any generic
+ * driver that is currently associated with the given framebuffer memory.
+ * If the framebuffer is located at PCI BAR 0, the rsp code looks as in the
+ * example given below. The cod assumes a DRM driver.
+ *
+ * .. code-block:: c
+ *
+ *	static const struct drm_driver example_driver = {
+ *		.name = "exampledrm",
+ *		...
+ *	};
+ *
+ *	static int remove_conflicting_framebuffers(struct pci_dev *pdev)
+ *	{
+ *		bool primary = false;
+ *		resource_size_t base, size;
+ *		int ret;
+ *
+ *		base = pci_resource_start(pdev, 0);
+ *		size = pci_resource_len(pdev, 0);
+ *	#ifdef CONFIG_X86
+ *		primary = pdev->resource[PCI_ROM_RESOURCE].flags & IORESOURCE_ROM_SHADOW;
+ *	#endif
+ *
+ *		return remove_conflicting_devices(base, size, primary, &example_driver->name);
+ *	}
+ *
+ *	static int probe(struct pci_dev *pdev)
+ *	{
+ *		int ret;
+ *
+ *		// Remove any generic drivers...
+ *		ret = remove_conflicting_framebuffers(pdev);
+ *		if (ret)
+ *			return ret;
+ *
+ *		// ... and initialize the hardware.
+ *		...
+ *
+ *		drm_dev_register();
+ *
+ *		return 0;
+ *	}
+ *
+ * PCI device drivers can also call remove_conflicting_pci_devices() and let the
+ * function detect the apertures automatically. Device drivers without knowledge of
+ * the framebuffer's location shall call remove_all_conflicting_devices(),
+ * which removes all known devices.
+ *
+ * Drivers that are susceptible to being removed by other drivers, such as
+ * generic EFI or VESA drivers, have to register themselves as owners of their
+ * framebuffer apertures. Ownership of the framebuffer memory is achieved
+ * by calling devm_acquire_aperture_of_platform_device(). On success, the driver
+ * is the owner of the framebuffer range. The function fails if the
+ * framebuffer is already owned by another driver. See below for an example.
+ *
+ * .. code-block:: c
+ *
+ *	static int acquire_framebuffers(struct drm_device *dev, struct platform_device *pdev)
+ *	{
+ *		resource_size_t base, size;
+ *
+ *		mem = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ *		if (!mem)
+ *			return -EINVAL;
+ *		base = mem->start;
+ *		size = resource_size(mem);
+ *
+ *		return devm_acquire_aperture_of_platform_device(pdev, base, size);
+ *	}
+ *
+ *	static int probe(struct platform_device *pdev)
+ *	{
+ *		struct drm_device *dev;
+ *		int ret;
+ *
+ *		// ... Initialize the device...
+ *		dev = devm_drm_dev_alloc();
+ *		...
+ *
+ *		// ... and acquire ownership of the framebuffer.
+ *		ret = acquire_framebuffers(dev, pdev);
+ *		if (ret)
+ *			return ret;
+ *
+ *		drm_dev_register(dev, 0);
+ *
+ *		return 0;
+ *	}
+ *
+ * The generic driver is now subject to forced removal by other drivers. This
+ * only works for platform drivers that support hot unplugging.
+ * When a driver calls remove_conflicting_devices() et al
+ * for the registered framebuffer range, the aperture helpers call
+ * platform_device_unregister() and the generic driver unloads itself. It
+ * may not access the device's registers, framebuffer memory, ROM, etc
+ * afterwards.
+ */
+
+struct dev_aperture {
+	struct device *dev;
+	resource_size_t base;
+	resource_size_t size;
+	struct list_head lh;
+	void (*detach)(struct device *dev);
+};
+
+static LIST_HEAD(apertures);
+static DEFINE_MUTEX(apertures_lock);
+
+static bool overlap(resource_size_t base1, resource_size_t end1,
+		    resource_size_t base2, resource_size_t end2)
+{
+	return (base1 < end2) && (end1 > base2);
+}
+
+static void devm_aperture_acquire_release(void *data)
+{
+	struct dev_aperture *ap = data;
+	bool detached = !ap->dev;
+
+	if (detached)
+		return;
+
+	mutex_lock(&apertures_lock);
+	list_del(&ap->lh);
+	mutex_unlock(&apertures_lock);
+}
+
+static int devm_aperture_acquire(struct device *dev,
+				 resource_size_t base, resource_size_t size,
+				 void (*detach)(struct device *))
+{
+	size_t end = base + size;
+	struct list_head *pos;
+	struct dev_aperture *ap;
+
+	mutex_lock(&apertures_lock);
+
+	list_for_each(pos, &apertures) {
+		ap = container_of(pos, struct dev_aperture, lh);
+		if (overlap(base, end, ap->base, ap->base + ap->size)) {
+			mutex_unlock(&apertures_lock);
+			return -EBUSY;
+		}
+	}
+
+	ap = devm_kzalloc(dev, sizeof(*ap), GFP_KERNEL);
+	if (!ap) {
+		mutex_unlock(&apertures_lock);
+		return -ENOMEM;
+	}
+
+	ap->dev = dev;
+	ap->base = base;
+	ap->size = size;
+	ap->detach = detach;
+	INIT_LIST_HEAD(&ap->lh);
+
+	list_add(&ap->lh, &apertures);
+
+	mutex_unlock(&apertures_lock);
+
+	return devm_add_action_or_reset(dev, devm_aperture_acquire_release, ap);
+}
+
+static void detach_platform_device(struct device *dev)
+{
+	struct platform_device *pdev = to_platform_device(dev);
+
+	/*
+	 * Remove the device from the device hierarchy. This is the right thing
+	 * to do for firmware-based DRM drivers, such as EFI, VESA or VGA. After
+	 * the new driver takes over the hardware, the firmware device's state
+	 * will be lost.
+	 *
+	 * For non-platform devices, a new callback would be required.
+	 *
+	 * If the aperture helpers ever need to handle native drivers, this call
+	 * would only have to unplug the DRM device, so that the hardware device
+	 * stays around after detachment.
+	 */
+	platform_device_unregister(pdev);
+}
+
+/**
+ * devm_acquire_aperture_of_platform_device - Acquires ownership of an aperture
+ *                                            on behalf of a platform device.
+ * @pdev:	the platform device to own the aperture
+ * @base:	the aperture's byte offset in physical memory
+ * @size:	the aperture size in bytes
+ *
+ * Installs the given device as the new owner of the aperture. The function
+ * expects the aperture to be provided by a platform device. If another
+ * driver takes over ownership of the aperture, aperture helpers will then
+ * unregister the platform device automatically. All acquired apertures are
+ * released automatically when the underlying device goes away.
+ *
+ * The function fails if the aperture, or parts of it, is currently
+ * owned by another device. To evict current owners, callers should use
+ * remove_conflicting_devices() et al. before calling this function.
+ *
+ * Returns:
+ * 0 on success, or a negative errno value otherwise.
+ */
+int devm_acquire_aperture_of_platform_device(struct platform_device *pdev,
+					     resource_size_t base,
+					     resource_size_t size)
+{
+	return devm_aperture_acquire(&pdev->dev, base, size, detach_platform_device);
+}
+EXPORT_SYMBOL(devm_acquire_aperture_of_platform_device);
+
+static void detach_devices(resource_size_t base, resource_size_t size)
+{
+	resource_size_t end = base + size;
+	struct list_head *pos, *n;
+
+	mutex_lock(&apertures_lock);
+
+	list_for_each_safe(pos, n, &apertures) {
+		struct dev_aperture *ap = container_of(pos, struct dev_aperture, lh);
+		struct device *dev = ap->dev;
+
+		if (WARN_ON_ONCE(!dev))
+			continue;
+
+		if (!overlap(base, end, ap->base, ap->base + ap->size))
+			continue;
+
+		ap->dev = NULL; /* detach from device */
+		list_del(&ap->lh);
+
+		ap->detach(dev);
+	}
+
+	mutex_unlock(&apertures_lock);
+}
+
+/**
+ * remove_conflicting_devices - remove devices in the given range
+ * @base: the aperture's base address in physical memory
+ * @size: aperture size in bytes
+ * @primary: also kick vga16fb if present; only relevant for VGA devices
+ * @name: a descriptive name of the requesting driver
+ *
+ * This function removes devices that own apertures within @base and @size.
+ *
+ * Returns:
+ * 0 on success, or a negative errno code otherwise
+ */
+int remove_conflicting_devices(resource_size_t base, resource_size_t size, bool primary,
+			       const char *name)
+{
+#if IS_REACHABLE(CONFIG_FB)
+	struct apertures_struct *a;
+	int ret;
+
+	a = alloc_apertures(1);
+	if (!a)
+		return -ENOMEM;
+
+	a->ranges[0].base = base;
+	a->ranges[0].size = size;
+
+	ret = remove_conflicting_framebuffers(a, name, primary);
+	kfree(a);
+
+	if (ret)
+		return ret;
+#endif
+
+	detach_devices(base, size);
+
+	return 0;
+}
+EXPORT_SYMBOL(remove_conflicting_devices);
+
+/**
+ * remove_conflicting_pci_devices - remove existing framebuffers for PCI devices
+ * @pdev: PCI device
+ * @name: a descriptive name of the requesting driver
+ *
+ * This function removes devices that own apertures within any of @pdev's
+ * memory bars. The function assumes that PCI device with shadowed ROM
+ * drives a primary display and therefore kicks out vga16fb as well.
+ *
+ * Returns:
+ * 0 on success, or a negative errno code otherwise
+ */
+int remove_conflicting_pci_devices(struct pci_dev *pdev, const char *name)
+{
+	resource_size_t base, size;
+	int bar, ret = 0;
+
+	for (bar = 0; bar < PCI_STD_NUM_BARS; ++bar) {
+		if (!(pci_resource_flags(pdev, bar) & IORESOURCE_MEM))
+			continue;
+		base = pci_resource_start(pdev, bar);
+		size = pci_resource_len(pdev, bar);
+		detach_devices(base, size);
+	}
+
+	/*
+	 * WARNING: Apparently we must kick fbdev drivers before vgacon,
+	 * otherwise the vga fbdev driver falls over.
+	 */
+#if IS_REACHABLE(CONFIG_FB)
+	ret = remove_conflicting_pci_framebuffers(pdev, name);
+#endif
+	if (ret == 0)
+		ret = vga_remove_vgacon(pdev);
+	return ret;
+}
+EXPORT_SYMBOL(remove_conflicting_pci_devices);
diff --git a/drivers/video/console/Kconfig b/drivers/video/console/Kconfig
index 40c50fa2dd70..22cea5082ac4 100644
--- a/drivers/video/console/Kconfig
+++ b/drivers/video/console/Kconfig
@@ -10,6 +10,7 @@ config VGA_CONSOLE
 	depends on !4xx && !PPC_8xx && !SPARC && !M68K && !PARISC &&  !SUPERH && \
 		(!ARM || ARCH_FOOTBRIDGE || ARCH_INTEGRATOR || ARCH_NETWINDER) && \
 		!ARM64 && !ARC && !MICROBLAZE && !OPENRISC && !S390 && !UML
+	select APERTURE_HELPERS if (DRM || FB || VFIO_PCI_CORE)
 	default y
 	help
 	  Saying Y here will allow you to use Linux in text mode through a
diff --git a/drivers/video/fbdev/Kconfig b/drivers/video/fbdev/Kconfig
index bd849013f16f..7b398e7e3cc8 100644
--- a/drivers/video/fbdev/Kconfig
+++ b/drivers/video/fbdev/Kconfig
@@ -455,6 +455,7 @@ config FB_ATARI
 config FB_OF
 	bool "Open Firmware frame buffer device support"
 	depends on (FB = y) && PPC && (!PPC_PSERIES || PCI)
+	select APERTURE_HELPERS
 	select FB_CFB_FILLRECT
 	select FB_CFB_COPYAREA
 	select FB_CFB_IMAGEBLIT
@@ -527,6 +528,7 @@ config FB_IMSTT
 config FB_VGA16
 	tristate "VGA 16-color graphics support"
 	depends on FB && (X86 || PPC)
+	select APERTURE_HELPERS
 	select FB_CFB_FILLRECT
 	select FB_CFB_COPYAREA
 	select FB_CFB_IMAGEBLIT
@@ -551,7 +553,7 @@ config FB_STI
 	  BIOS routines contained in a ROM chip in HP PA-RISC based machines.
 	  Enabling this option will implement the linux framebuffer device
 	  using calls to the STI BIOS routines for initialisation.
-	
+
 	  If you enable this option, you will get a planar framebuffer device
 	  /dev/fb which will work on the most common HP graphic cards of the
 	  NGLE family, including the artist chips (in the 7xx and Bxxx series),
@@ -617,6 +619,7 @@ config FB_UVESA
 config FB_VESA
 	bool "VESA VGA graphics support"
 	depends on (FB = y) && (X86 || COMPILE_TEST)
+	select APERTURE_HELPERS
 	select FB_CFB_FILLRECT
 	select FB_CFB_COPYAREA
 	select FB_CFB_IMAGEBLIT
@@ -630,6 +633,7 @@ config FB_VESA
 config FB_EFI
 	bool "EFI-based Framebuffer Support"
 	depends on (FB = y) && !IA64 && EFI
+	select APERTURE_HELPERS
 	select DRM_PANEL_ORIENTATION_QUIRKS
 	select FB_CFB_FILLRECT
 	select FB_CFB_COPYAREA
@@ -2190,6 +2194,7 @@ config FB_SIMPLE
 	tristate "Simple framebuffer support"
 	depends on FB
 	depends on !DRM_SIMPLEDRM
+	select APERTURE_HELPERS
 	select FB_CFB_FILLRECT
 	select FB_CFB_COPYAREA
 	select FB_CFB_IMAGEBLIT
diff --git a/include/linux/aperture.h b/include/linux/aperture.h
new file mode 100644
index 000000000000..0a206cdce606
--- /dev/null
+++ b/include/linux/aperture.h
@@ -0,0 +1,56 @@
+/* SPDX-License-Identifier: MIT */
+
+#ifndef _LINUX_APERTURE_H_
+#define _LINUX_APERTURE_H_
+
+#include <linux/types.h>
+
+struct pci_dev;
+struct platform_device;
+
+#if defined(CONFIG_APERTURE_HELPERS)
+int devm_acquire_aperture_of_platform_device(struct platform_device *pdev,
+					     resource_size_t base,
+					     resource_size_t size);
+
+int remove_conflicting_devices(resource_size_t base, resource_size_t size,
+			       bool primary, const char *name);
+
+int remove_conflicting_pci_devices(struct pci_dev *pdev, const char *name);
+#else
+static inline int devm_acquire_aperture_of_platform_device(struct platform_device *pdev,
+							   resource_size_t base,
+							   resource_size_t size)
+{
+	return 0;
+}
+
+static inline int remove_conflicting_devices(resource_size_t base, resource_size_t size,
+					     bool primary, const char *name)
+{
+	return 0;
+}
+
+static inline int remove_conflicting_pci_devices(struct pci_dev *pdev, const char *name)
+{
+	return 0;
+}
+#endif
+
+/**
+ * remove_all_conflicting_devices - remove all existing framebuffers
+ * @primary: also kick vga16fb if present; only relevant for VGA devices
+ * @name: a descriptive name of the requesting driver
+ *
+ * This function removes all graphics device drivers. Use this function on systems
+ * that can have their framebuffer located anywhere in memory.
+ *
+ * Returns:
+ * 0 on success, or a negative errno code otherwise
+ */
+static inline int remove_all_conflicting_devices(bool primary, const char *name)
+{
+	return remove_conflicting_devices(0, (resource_size_t)-1, primary, name);
+}
+
+#endif
-- 
2.36.1


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]

Re: [PATCH 2/2] vfio/pci: Remove console drivers

[Alex Williamson]

On Fri, 10 Jun 2022 09:03:15 +0200
Thomas Zimmermann <tzimmermann@suse.de> wrote:

> Hi
> 
> Am 09.06.22 um 23:44 schrieb Alex Williamson:
> > On Thu, 9 Jun 2022 15:41:02 -0600
> > Alex Williamson <alex.williamson@redhat.com> wrote:
> >   
> >> On Thu, 9 Jun 2022 11:13:22 +0200
> >> Thomas Zimmermann <tzimmermann@suse.de> wrote:  
> >>>
> >>> Please have a look at the attached patch. It moves the aperture helpers
> >>> to a location common to the various possible users (DRM, fbdev, vfio).
> >>> The DRM interfaces remain untouched for now.  The patch should provide
> >>> what you need in vfio and also serve our future use cases for graphics
> >>> drivers. If possible, please create your patch on top of it.  
> >>
> >> Looks good to me, this of course makes the vfio change quite trivial.
> >> One change I'd request:
> >>
> >> diff --git a/drivers/video/console/Kconfig b/drivers/video/console/Kconfig
> >> index 40c50fa2dd70..7f3c44e1538b 100644
> >> --- a/drivers/video/console/Kconfig
> >> +++ b/drivers/video/console/Kconfig
> >> @@ -10,6 +10,7 @@ config VGA_CONSOLE
> >>   	depends on !4xx && !PPC_8xx && !SPARC && !M68K && !PARISC &&  !SUPERH && \
> >>   		(!ARM || ARCH_FOOTBRIDGE || ARCH_INTEGRATOR || ARCH_NETWINDER) && \
> >>   		!ARM64 && !ARC && !MICROBLAZE && !OPENRISC && !S390 && !UML
> >> +	select APERTURE_HELPERS if (DRM || FB || VFIO_PCI)
> >>   	default y
> >>   	help
> >>   	  Saying Y here will allow you to use Linux in text mode through a
> >>
> >> This should be VFIO_PCI_CORE.  Thanks,  
> 
> I attached an updated patch to this email.
> 
> > 
> > Also, whatever tree this lands in, I'd appreciate a topic branch being
> > made available so I can more easily get the vfio change in on the same
> > release.  Thanks,  
> 
> You can add my patch to your series and merge it through vfio. You'd 
> only have to cc dri-devel for the patch's review. I guess it's more 
> important for vfio than DRM. We have no hurry on the DRM side, but v5.20 
> would be nice.

Ok, I didn't realize you were offering the patch for me to post and
merge.  I'll do that.  Thanks!

Alex

Re: [PATCH 2/2] vfio/pci: Remove console drivers

[Gerd Hoffmann]

  Hi,

> You shouldn't have to copy any of the implementation of the aperture
> helpers.

That comes from the aperture helpers being part of drm ...

> For patch 2, the most trivial workaround is to instanciate struct drm_driver
> here and set the name field to 'vdev->vdev.ops->name'. In the longer term,
> the aperture helpers will be moved out of DRM and into a more prominent
> location. That workaround will be cleaned up then.

... but if the long-term plan is to clean that up properly anyway I
don't see the point in bike shedding too much on the details of some
temporary solution.

> Alternatively, drm_aperture_remove_conflicting_pci_framebuffers() could be
> changed to accept the name string as second argument, but that's quite a bit
> of churn within the DRM code.

Also pointless churn because you'll have the very same churn again when
moving the aperture helpers out of drm.

take care,
  Gerd

Re: [PATCH 2/2] vfio/pci: Remove console drivers

[Gerd Hoffmann]

  Hi,

> You shouldn't have to copy any of the implementation of the aperture
> helpers.

That comes from the aperture helpers being part of drm ...

> For patch 2, the most trivial workaround is to instanciate struct drm_driver
> here and set the name field to 'vdev->vdev.ops->name'. In the longer term,
> the aperture helpers will be moved out of DRM and into a more prominent
> location. That workaround will be cleaned up then.

... but if the long-term plan is to clean that up properly anyway I
don't see the point in bike shedding too much on the details of some
temporary solution.

> Alternatively, drm_aperture_remove_conflicting_pci_framebuffers() could be
> changed to accept the name string as second argument, but that's quite a bit
> of churn within the DRM code.

Also pointless churn because you'll have the very same churn again when
moving the aperture helpers out of drm.

take care,
  Gerd