From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9E559C4321E for ; Mon, 5 Dec 2022 21:52:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232957AbiLEVwv convert rfc822-to-8bit (ORCPT ); Mon, 5 Dec 2022 16:52:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51548 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234341AbiLEVwR (ORCPT ); Mon, 5 Dec 2022 16:52:17 -0500 Received: from mail-vk1-f181.google.com (mail-vk1-f181.google.com [209.85.221.181]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BA738112F; Mon, 5 Dec 2022 13:51:06 -0800 (PST) Received: by mail-vk1-f181.google.com with SMTP id 6so3892711vkz.0; Mon, 05 Dec 2022 13:51:06 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mFdsLFzswFTMipJStVwFiJsipVOV+7MBmY79S1gML8g=; b=pN3iVsdUe89rB2QlXMZtfkRVEpP/2SDisnAA6dUj78/RsyB2q5TvT2ck1KCZvusOFz fVZ8EU1nSJcwfyV7GfnK+1sB1HYxBg/1BMgA6Kcl8VttbTDksyK8no0OttBbUM4rSJn9 hr5pV5R1JuKnd1pl0w/L1pp8kDDYbnfEG+CCBB8DIp2CLtXboOo5l129rvMLM+pjo70e pwSi1kyXeL7jcASyS+kpjJqdFJZviMxanRiD5pfJ387pYMh2+95aHtTM1rHufr0U4sRQ V4P9MM7meej3b+MyfKiLkFh/7KyJ8KU1XJtiOcWRlKiflwdf59ZVvemGOapS70s+BzDC wi/Q== X-Gm-Message-State: ANoB5pmxVA90FWZy0SYOEADcj+8sKMKQLSKqjs2m947pv9wM4E44ESDb DLJX5+YWCv9Cuxx2+xItXvAnc7NfCfCYWA== X-Google-Smtp-Source: AA0mqf4VsF/MmDcoGXN+DmdBHP+zaV9q/Nr0d2lwAXRkoUQj75D0qLdVJQsskplAGWImSoXFq9FpeQ== X-Received: by 2002:a05:6122:2a8:b0:3b8:b112:5eed with SMTP id 8-20020a05612202a800b003b8b1125eedmr37783652vkq.24.1670277065315; Mon, 05 Dec 2022 13:51:05 -0800 (PST) Received: from mail-vs1-f46.google.com (mail-vs1-f46.google.com. [209.85.217.46]) by smtp.gmail.com with ESMTPSA id j125-20020a1f5583000000b003a9993cdda6sm2367947vkb.54.2022.12.05.13.51.04 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 05 Dec 2022 13:51:04 -0800 (PST) Received: by mail-vs1-f46.google.com with SMTP id 125so12422225vsi.9; Mon, 05 Dec 2022 13:51:04 -0800 (PST) X-Received: by 2002:a05:6102:2221:b0:3b0:4a83:954d with SMTP id d1-20020a056102222100b003b04a83954dmr38222003vsb.62.1670277064155; Mon, 05 Dec 2022 13:51:04 -0800 (PST) MIME-Version: 1.0 References: <20221204175142.658d5c37.alex.williamson@redhat.com> <1e4d62cf-8893-0bff-51f5-5a2e419ed5a0@suse.de> In-Reply-To: From: "mb@lab.how" Date: Mon, 5 Dec 2022 14:50:52 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [PATCH 2/2] vfio/pci: Remove console drivers To: Thomas Zimmermann Cc: kvm@vger.kernel.org, airlied@linux.ie, linux-kernel@vger.kernel.org, dri-devel@lists.freedesktop.org, Alex Williamson , kraxel@redhat.com, lersek@redhat.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8BIT Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Thomas, On Mon, Dec 5, 2022 at 3:11 AM Thomas Zimmermann wrote: > > Hi > > Am 05.12.22 um 10:32 schrieb mb@lab.how: > > I have a rtx 3070 and a 3090, I am absolutely sure I am binding vfio-pci > > to the 3090 and not the 3070. > > > > I have bound the driver in two different ways, first by passing the IDs > > to the module and alternatively by manipulating the system interface and > > use the override (this is what I originally had to do when I used two > > 1080s, so I know it works). > > > > While the 3090 doesn't show a console, there's a remnant from the refund > > (and grub previously) there. > > > > The assessment Alex made previously, where > > aperture_remove_conflicting_pci_devices() is removing the driver (EFIFB) > > instead of the device seems correct, but it could also can be a quirky > > of how EFIFB is implemented. I recall reading a long time ago that EFIFB > > is a special device and once it detects changes it would simply give up. > > There was also no way to attach a device to it again as it depends on > > being preloaded outside the kernel; once something takes over the buffer > > reinitializing is "impossible". I never went deeper to try and > > understand it. > > We recently reworked fbdev's interaction with the aperture helpers. [1] > All devices should now be removed iff the driver has been bound to it > (which should be the case here) The patches went into an v6.1-rc. > > Could you try the most recent v6.1-rc and report if this fixes the problem? I just tried the latest one, v6.1-rc8, and I can see all the commits for the series you mentioned there. The same freeze behavior happens when I load vfio-pci: [ 6.525463] VFIO - User Level meta-driver version: 0.3 [ 6.528231] Console: switching to colour dummy device 320x90 -- Carlos > > Best regards > Thomas > > [1] https://patchwork.freedesktop.org/series/106040/ > > > > > > > On Mon, Dec 5, 2022, 2:00 AM Thomas Zimmermann > > wrote: > > > > Hi > > > > Am 05.12.22 um 01:51 schrieb Alex Williamson: > > > On Sat, 3 Dec 2022 17:12:38 -0700 > > > "mb@lab.how" wrote: > > > > > >> Hi, > > >> > > >> I hope it is ok to reply to this old thread. > > > > > > It is, but the only relic of the thread is the subject. For > > reference, > > > the latest version of this posted is here: > > > > > > > > https://lore.kernel.org/all/20220622140134.12763-4-tzimmermann@suse.de/ > > > > > > Which is committed as: > > > > > > d17378062079 ("vfio/pci: Remove console drivers") > > > > > >> Unfortunately, I found a > > >> problem only now after upgrading to 6.0. > > >> > > >> My setup has multiple GPUs (2), and I depend on EFIFB to have a > > working console. > > > > Which GPUs do you have? > > > > >> pre-patch behavior, when I bind the vfio-pci to my secondary GPU > > both > > >> the passthrough and the EFIFB keep working fine. > > >> post-patch behavior, when I bind the vfio-pci to the secondary GPU, > > >> the EFIFB disappears from the system, binding the console to the > > >> "dummy console". > > > > The efifb would likely use the first GPU. And vfio-pci should only > > remove the generic driver from the second device. Are you sure that > > you're not somehow using the first GPU with vfio-pci. > > > > >> Whenever you try to access the terminal, you have the screen > > stuck in > > >> whatever was the last buffer content, which gives the impression of > > >> "freezing," but I can still type. > > >> Everything else works, including the passthrough. > > > > > > This sounds like the call to > > aperture_remove_conflicting_pci_devices() > > > is removing the conflicting driver itself rather than removing the > > > device from the driver. Is it not possible to unbind the GPU from > > > efifb before binding the GPU to vfio-pci to effectively nullify the > > > added call? > > > > > >> I can only think about a few options: > > >> > > >> - Is there a way to have EFIFB show up again? After all it looks > > like > > >> the kernel has just abandoned it, but the buffer is still there. I > > >> can't find a single message about the secondary card and EFIFB in > > >> dmesg, but there's a message for the primary card and EFIFB. > > >> - Can we have a boolean controlling the behavior of vfio-pci > > >> altogether or at least controlling the behavior of vfio-pci for that > > >> specific ID? I know there's already some option for vfio-pci and VGA > > >> cards, would it be appropriate to attach this behavior to that > > option? > > > > > > I suppose we could have an opt-out module option on vfio-pci to skip > > > the above call, but clearly it would be better if things worked by > > > default. We cannot make full use of GPUs with vfio-pci if they're > > > still in use by host console drivers. The intention was certainly to > > > unbind the device from any low level drivers rather than disable > > use of > > > a console driver entirely. DRM/GPU folks, is that possibly an > > > interface we could implement? Thanks, > > > > When vfio-pci gives the GPU device to the guest, which driver driver is > > bound to it? > > > > Best regards > > Thomas > > > > > > > > Alex > > > > > > > -- > > Thomas Zimmermann > > Graphics Driver Developer > > SUSE Software Solutions Germany GmbH > > Maxfeldstr. 5, 90409 Nürnberg, Germany > > (HRB 36809, AG Nürnberg) > > Geschäftsführer: Ivo Totev > > > > -- > Thomas Zimmermann > Graphics Driver Developer > SUSE Software Solutions Germany GmbH > Maxfeldstr. 5, 90409 Nürnberg, Germany > (HRB 36809, AG Nürnberg) > Geschäftsführer: Ivo Totev