From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from [140.186.70.92] (port=59467 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1Olo1l-0004jA-Pq for qemu-devel@nongnu.org; Wed, 18 Aug 2010 15:10:23 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1Olo1k-0004DV-8D for qemu-devel@nongnu.org; Wed, 18 Aug 2010 15:10:21 -0400 Received: from smtp.mailix.net ([66.11.225.183]:26955) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Olo1k-0004Cs-13 for qemu-devel@nongnu.org; Wed, 18 Aug 2010 15:10:20 -0400 From: "Adnan Khaleel" Message-ID: <20100818191010.87d50a95@shadowfax.no-ip.com> Date: Wed, 18 Aug 2010 14:10:10 -0500 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="-----------bcad05aeb8ef29f443434f2c7728e896" Subject: [Qemu-devel] Template for developing a Qemu device with PCIe and MSI-X Reply-To: adnan@khaleel.us List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: qemu-devel@nongnu.org This is a multi-part message in MIME format. -------------bcad05aeb8ef29f443434f2c7728e896 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hello Qemu developers, I'm interested in developing a device model that plugs into Qemu that is= based on a PCIe interface and uses MSI-X. My goal is to ultimately atta= ch a GPU simulator to this PCIe interface and use the entire platfom (Qe= mu + GPU simulator) for studying cpu, gpu interactions. I'm not terribly familiar with the Qemu device model and I'm looking for= some assistance, perhaps a starting template for pcie and msi-x that wo= uld offer the basic functionality that I could then build upon.=20 I have looked at the various devices that already modelled that are incl= uded with Qemu (v0.12.5 at least) and I've noticed several a few pci dev= ices, eg; ne2k and cirrus-pci etc, however only one device truly seems t= o utilize both the technologies that I'm interested in and that is the v= irtio-pci.c I'm not sure what virtio-pci does so I'm not sure if that is a suitable = starting point for me.=20 Any help, suggestions etc would be extremely helpful and much appreciate= d. Sincerely, AK -------------bcad05aeb8ef29f443434f2c7728e896 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
Hello Qemu developers,

I'm inter= ested in developing a device model that plugs into Qemu that is based on= a PCIe interface and uses MSI-X. My goal is to ultimately attach a GPU = simulator to this PCIe interface and use the entire platfom (Qemu + GPU = simulator) for studying cpu, gpu interactions.

I'm not terribly f= amiliar with the Qemu device model and I'm looking for some assistance, = perhaps a starting template for pcie and msi-x that would offer the basi= c functionality that I could then build upon.

I have looked at t= he various devices that already modelled that are included with Qemu (v0= .12.5 at least) and I've noticed several a few pci devices, eg; ne2k and= cirrus-pci etc, however only one device truly seems to utilize both the= technologies that I'm interested in and that is the virtio-pci.c
I'm not sure what virtio-pci does so I'm not sure if that is a suitable= starting point for me.

Any help, suggestions etc would be extre= mely helpful and much appreciated.

Sincerely,

AK
= -------------bcad05aeb8ef29f443434f2c7728e896-- From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from [140.186.70.92] (port=49883 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OlvVv-0005SR-04 for qemu-devel@nongnu.org; Wed, 18 Aug 2010 23:10:00 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OlvVt-000711-Pr for qemu-devel@nongnu.org; Wed, 18 Aug 2010 23:09:58 -0400 Received: from mail.valinux.co.jp ([210.128.90.3]:45138) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OlvVt-00070g-Ai for qemu-devel@nongnu.org; Wed, 18 Aug 2010 23:09:57 -0400 Date: Thu, 19 Aug 2010 12:19:04 +0900 From: Isaku Yamahata Subject: Re: [Qemu-devel] Template for developing a Qemu device with PCIe and MSI-X Message-ID: <20100819031904.GG11421@valinux.co.jp> References: <20100818191010.87d50a95@shadowfax.no-ip.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20100818191010.87d50a95@shadowfax.no-ip.com> List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Adnan Khaleel Cc: qemu-devel@nongnu.org On Wed, Aug 18, 2010 at 02:10:10PM -0500, Adnan Khaleel wrote: > Hello Qemu developers, > > I'm interested in developing a device model that plugs into Qemu that is based > on a PCIe interface and uses MSI-X. My goal is to ultimately attach a GPU > simulator to this PCIe interface and use the entire platfom (Qemu + GPU > simulator) for studying cpu, gpu interactions. > > I'm not terribly familiar with the Qemu device model and I'm looking for some > assistance, perhaps a starting template for pcie and msi-x that would offer the > basic functionality that I could then build upon. > > I have looked at the various devices that already modelled that are included > with Qemu (v0.12.5 at least) and I've noticed several a few pci devices, eg; > ne2k and cirrus-pci etc, however only one device truly seems to utilize both > the technologies that I'm interested in and that is the virtio-pci.c > > I'm not sure what virtio-pci does so I'm not sure if that is a suitable > starting point for me. > > Any help, suggestions etc would be extremely helpful and much appreciated. Qemu doesn't support pcie at the moment. Only partial patches have been merged, still more patches have to be merged for pcie to fully work. The following repo is available. git clone http://people.valinux.co.jp/~yamahata/qemu/q35/qemu git clone http://people.valinux.co.jp/~yamahata/qemu/q35/seabios git clone http://people.valinux.co.jp/~yamahata/qemu/q35/vgabios Note: patched seabios and vgabios are needed, you have to pass ACPI DSDT for q35. example: qemu-system-x86_64 -M pc_q35 -acpitable load_header,data=roms/seabios/src/q35-acpi-dsdt.aml This repo is for those who want to try/develop pcie support, not for upstream merge. So they include patches unsuitable for upstream. The repo includes pcie port switch emulator which utilize pcie and MSI(not MSI-X). The difference between PCI device and PCIe device is configuration space size. By setting PCIDeviceInfo::is_express = 1, you'll get 4K configuration space. Helper functions for pcie are found in qemu/hw/pcie.c For msi-x, see qemu/hw/msix.c. Thanks, -- yamahata From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from [140.186.70.92] (port=53494 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1Om8Us-0007jr-8t for qemu-devel@nongnu.org; Thu, 19 Aug 2010 13:01:47 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1Om8Uq-0006BF-S2 for qemu-devel@nongnu.org; Thu, 19 Aug 2010 13:01:46 -0400 Received: from smtp.mailix.net ([66.11.225.183]:16183) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Om8Uq-0006B4-Kp for qemu-devel@nongnu.org; Thu, 19 Aug 2010 13:01:44 -0400 From: "Adnan Khaleel" Subject: =?iso-8859-1?Q?Re=3A_=5BQemu-devel=5D_Template_for_developing_a_Qe?= =?iso-8859-1?Q?mu_device_with_PCIe=09and_MSI-X?= In-Reply-To: 20100819031904.GG11421@valinux.co.jp Message-ID: <20100819170140.10dc216a@shadowfax.no-ip.com> Date: Thu, 19 Aug 2010 12:01:40 -0500 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="-----------15dbfed9f90ce565f8a620eddfb9fedd" Reply-To: adnan@khaleel.us List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Isaku Yamahata Cc: qemu-devel@nongnu.org This is a multi-part message in MIME format. -------------15dbfed9f90ce565f8a620eddfb9fedd Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi Isaku, thank you very much for your very detailed response. I have a = few questions, see below. Thanks again, Adnan Qemu doesn't support pcie at the moment. Only partial patches have been merged, still more patches have to be merged for pcie to fully work. The following repo is available. =20 git clone http://people.valinux.co.jp/~yamahata/qemu/q35/qemu git clone http://people.valinux.co.jp/~yamahata/qemu/q35/seabios git clone http://people.valinux.co.jp/~yamahata/qemu/q35/vgabios =20 Note: patched seabios and vgabios are needed, you have to pass ACPI DS= DT for q35. example: qemu-system-x86=5F64 -M pc=5Fq35 -acpitable load=5Fheader,data=3Droms/= seabios/src/q35-acpi-dsdt.aml =20 This repo is for those who want to try/develop pcie support, not for upstream merge. So they include patches unsuitable for upstrea= m.I'm looking at Qemu 0.12.3 and there are 2 files, pci=5Fhost.c and pci= e=5Fhost.c. Can you explain what these do=3F Also, I see virtio=5Fpci.c is the only device that uses msi-x in qemu. C= an you explain what device this is trying to emulate=3F Also, will the support for PCIe be merged with the mail Qemu at some poi= nt=3F =20 The repo includes pcie port switch emulator which utilize pcie and MSI(not MSI-X). I guess I could use this as a template for my qemu device mode correct= =3F The difference between PCI device and PCIe device is configuration space size. By setting PCIDeviceInfo::is=5Fexpress =3D 1, you'll get 4K configurat= ion space. Helper functions for pcie are found in qemu/hw/pcie.c For msi-x, see qemu/hw/msix.c. One last question, does the current implementation allow for 64bit BAR= addresses=3F Thanks, --=20 yamahata =20 -------------15dbfed9f90ce565f8a620eddfb9fedd Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
Hi Isaku, thank you very much for your = very detailed response. I have a few questions, see below.

Thanks= again,

Adnan

Qemu doesn't support pcie at the moment.
Only partial patches have been merged, still more patches have to
be merged for pcie to fully work. The following repo is available.

git clone http://people.valinux.co.jp/~yamahata/qemu/q35/q= emu
git clone http://people.valinux.co.jp/~yamahata/qemu/q3= 5/seabios
git clone http://people.valinux.co.jp/~yamahata/qemu/q3= 5/vgabios

Note: patched seabios and vgabios are needed, you have to pass ACPI DSDT=
for q35.
example:
qemu-system-x86=5F64 -M pc=5Fq35 -acpitable load=5Fheader,data=3Droms/se= abios/src/q35-acpi-dsdt.aml

This repo is for those who want to try/develop pcie support,
not for upstream merge. So they include patches unsuitable for upstream.=
I'm looking at Qemu 0.12.3 and there are 2 files, pci=5Fhos= t.c and pcie=5Fhost.c. Can you explain what these do=3F
Also, I see v= irtio=5Fpci.c is the only device that uses msi-x in qemu. Can you explai= n what device this is trying to emulate=3F
Also, will the support for= PCIe be merged with the mail Qemu at some point=3F

The repo includes pcie port switch emulator which utilize pcie and
MSI(not MSI-X).
I guess I could use this as a template for my qemu device m= ode correct=3F

The difference between PCI device and PCIe device is configuration
space size.
By setting PCIDeviceInfo::is=5Fexpress =3D 1, you'll get 4K configuratio= n
space. Helper functions for pcie are found in qemu/hw/pcie.c
For msi-x, see qemu/hw/msix.c.
One last question, does the current implementation allow fo= r 64bit BAR addresses=3F

Thanks,
--
yamahata
-------------15dbfed9f90ce565f8a620eddfb9fedd-- From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from [140.186.70.92] (port=60343 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1Om9v1-0006Cz-9i for qemu-devel@nongnu.org; Thu, 19 Aug 2010 14:32:52 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1Om9ux-00035E-8O for qemu-devel@nongnu.org; Thu, 19 Aug 2010 14:32:51 -0400 Received: from smtp.mailix.net ([66.11.225.183]:21688) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Om9uw-00034T-Um for qemu-devel@nongnu.org; Thu, 19 Aug 2010 14:32:47 -0400 From: "Adnan Khaleel" Subject: =?iso-8859-1?Q?Re=3A_=5BQemu-devel=5D_Template_for_developing_a_Qe?= =?iso-8859-1?Q?mu_device_with_PCIe=09and_MSI-X?= In-Reply-To: 20100819031904.GG11421@valinux.co.jp Message-ID: <20100819183242.e5078819@shadowfax.no-ip.com> Date: Thu, 19 Aug 2010 13:32:42 -0500 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="-----------7e7565907b5807c8fc6000519141eef2" Reply-To: adnan@khaleel.us List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Isaku Yamahata Cc: qemu-devel@nongnu.org This is a multi-part message in MIME format. -------------7e7565907b5807c8fc6000519141eef2 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Isaku, I'm having some difficulties building the sources, I get the following m= essage *akhaleel@yar95 qemu-q35 $ ./configure --help : bad interpreter: No such file or directory And I get a similar error while compiling seabios as well. What shell are you using or am I missing something=3F I'm compiling from= a typical bash shell and using gcc v4.4.0. In vgabios, there is a requirement for bcc. Is that borland C compiler= =3F Thanks Adnan =5F=5F=5F=5F=5F =20 From: Isaku Yamahata [mailto:yamahata@valinux.co.jp] To: Adnan Khaleel [mailto:adnan@khaleel.us] Cc: qemu-devel@nongnu.org Sent: Wed, 18 Aug 2010 22:19:04 -0500 Subject: Re: [Qemu-devel] Template for developing a Qemu device with PCI= e and MSI-X On Wed, Aug 18, 2010 at 02:10:10PM -0500, Adnan Khaleel wrote: > Hello Qemu developers, > > I'm interested in developing a device model that plugs into Qemu tha= t is based > on a PCIe interface and uses MSI-X. My goal is to ultimately attach = a GPU > simulator to this PCIe interface and use the entire platfom (Qemu + = GPU > simulator) for studying cpu, gpu interactions. >=20 > I'm not terribly familiar with the Qemu device model and I'm looking= for some > assistance, perhaps a starting template for pcie and msi-x that woul= d offer the > basic functionality that I could then build upon. >=20 > I have looked at the various devices that already modelled that are = included > with Qemu (v0.12.5 at least) and I've noticed several a few pci devi= ces, eg; > ne2k and cirrus-pci etc, however only one device truly seems to util= ize both > the technologies that I'm interested in and that is the virtio-pci.c >=20 > I'm not sure what virtio-pci does so I'm not sure if that is a suita= ble > starting point for me. >=20 > Any help, suggestions etc would be extremely helpful and much apprec= iated. =20 Qemu doesn't support pcie at the moment. Only partial patches have been merged, still more patches have to be merged for pcie to fully work. The following repo is available. =20 git clone http://people.valinux.co.jp/~yamahata/qemu/q35/qemu git clone http://people.valinux.co.jp/~yamahata/qemu/q35/seabios git clone http://people.valinux.co.jp/~yamahata/qemu/q35/vgabios =20 Note: patched seabios and vgabios are needed, you have to pass ACPI DS= DT for q35. example: qemu-system-x86=5F64 -M pc=5Fq35 -acpitable load=5Fheader,data=3Droms/= seabios/src/q35-acpi-dsdt.aml =20 This repo is for those who want to try/develop pcie support, not for upstream merge. So they include patches unsuitable for upstrea= m. The repo includes pcie port switch emulator which utilize pcie and MSI(not MSI-X). =20 The difference between PCI device and PCIe device is configuration space size. By setting PCIDeviceInfo::is=5Fexpress =3D 1, you'll get 4K configurat= ion space. Helper functions for pcie are found in qemu/hw/pcie.c For msi-x, see qemu/hw/msix.c. =20 Thanks, --=20 yamahata =20 -------------7e7565907b5807c8fc6000519141eef2 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
Isaku,

I'm having some difficult= ies building the sources, I get the following message

*akhaleel@y= ar95 qemu-q35 $ ./configure --help
: bad interpreter: No such file or= directory

And I get a similar error while compiling seabios as w= ell.

What shell are you using or am I missing something=3F I'm co= mpiling from a typical bash shell and using gcc v4.4.0.

In vgabio= s, there is a requirement for bcc. Is that borland C compiler=3F

= Thanks

Adnan
http://people.valinux.co.jp/~yamahata/qemu/q35/q= emu
git clone http://people.valinux.co.jp/~yamahata/qemu/q3= 5/seabios
git clone http://people.valinux.co.jp/~yamahata/qemu/q3= 5/vgabios

Note: patched seabios and vgabios are needed, you have to pass ACPI DSDT=
for q35.
example:
qemu-system-x86=5F64 -M pc=5Fq35 -acpitable load=5Fheader,data=3Droms/se= abios/src/q35-acpi-dsdt.aml

This repo is for those who want to try/develop pcie support,
not for upstream merge. So they include patches unsuitable for upstream.=
The repo includes pcie port switch emulator which utilize pcie and
MSI(not MSI-X).

The difference between PCI device and PCIe device is configuration
space size.
By setting PCIDeviceInfo::is=5Fexpress =3D 1, you'll get 4K configuratio= n
space. Helper functions for pcie are found in qemu/hw/pcie.c
For msi-x, see qemu/hw/msix.c.

Thanks,
--
yamahata
-------------7e7565907b5807c8fc6000519141eef2-- From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from [140.186.70.92] (port=50664 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OmJqL-0001IA-IT for qemu-devel@nongnu.org; Fri, 20 Aug 2010 01:08:42 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OmJnW-0006zO-5C for qemu-devel@nongnu.org; Fri, 20 Aug 2010 01:05:50 -0400 Received: from mail.valinux.co.jp ([210.128.90.3]:44380) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OmJnV-0006z4-JI for qemu-devel@nongnu.org; Fri, 20 Aug 2010 01:05:46 -0400 Date: Fri, 20 Aug 2010 14:15:05 +0900 From: Isaku Yamahata Subject: Re: [Qemu-devel] Template for developing a Qemu device with PCIe?and MSI-X Message-ID: <20100820051505.GC31720@valinux.co.jp> References: <20100819170140.10dc216a@shadowfax.no-ip.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20100819170140.10dc216a@shadowfax.no-ip.com> List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Adnan Khaleel Cc: qemu-devel@nongnu.org On Thu, Aug 19, 2010 at 12:01:40PM -0500, Adnan Khaleel wrote: > Hi Isaku, thank you very much for your very detailed response. I have a few > questions, see below. > > Thanks again, > > Adnan > > > Qemu doesn't support pcie at the moment. > Only partial patches have been merged, still more patches have to > be merged for pcie to fully work. The following repo is available. > > git clone http://people.valinux.co.jp/~yamahata/qemu/q35/qemu > git clone http://people.valinux.co.jp/~yamahata/qemu/q35/seabios > git clone http://people.valinux.co.jp/~yamahata/qemu/q35/vgabios > > Note: patched seabios and vgabios are needed, you have to pass ACPI DSDT > for q35. > example: > qemu-system-x86_64 -M pc_q35 -acpitable load_header,data=roms/seabios/src/ > q35-acpi-dsdt.aml > > This repo is for those who want to try/develop pcie support, > not for upstream merge. So they include patches unsuitable for upstream. > > I'm looking at Qemu 0.12.3 and there are 2 files, pci_host.c and pcie_host.c. > Can you explain what these do? They are for configuration space. pci_host.c abstracts the emulation of indirect access to configuration space. On PC, ioport 0xcf8 and 0xcfc. pcie_host abtracts the emulation of access to MMCONFIG space. There is no user at the moment, though. > Also, I see virtio_pci.c is the only device that uses msi-x in qemu. Can you > explain what device this is trying to emulate? virtio is a paravirtual IO framework for kvm. There is no corresponding real hardware. > Also, will the support for PCIe be merged with the mail Qemu at some point? I've been trying it. Your help will be appreciated. > The repo includes pcie port switch emulator which utilize pcie and > MSI(not MSI-X). > > I guess I could use this as a template for my qemu device mode correct? Half yes. You need to be aware that port switch is pci-to-pci bridge which is slightly different from normal pci/pcie device. There is no emulator of pcie normal device at the moment. > The difference between PCI device and PCIe device is configuration > space size. > By setting PCIDeviceInfo::is_express = 1, you'll get 4K configuration > space. Helper functions for pcie are found in qemu/hw/pcie.c > For msi-x, see qemu/hw/msix.c. > > One last question, does the current implementation allow for 64bit BAR > addresses? Yes. 64bit BAR emulation was alread merged. seabios is also ready for it. -- yamahata From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from [140.186.70.92] (port=47643 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OmJuE-00037r-4h for qemu-devel@nongnu.org; Fri, 20 Aug 2010 01:12:43 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OmJuD-00083N-3Q for qemu-devel@nongnu.org; Fri, 20 Aug 2010 01:12:42 -0400 Received: from mail.valinux.co.jp ([210.128.90.3]:54449) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OmJuC-000838-KA for qemu-devel@nongnu.org; Fri, 20 Aug 2010 01:12:41 -0400 Date: Fri, 20 Aug 2010 14:22:03 +0900 From: Isaku Yamahata Subject: Re: [Qemu-devel] Template for developing a Qemu device with PCIe?and MSI-X Message-ID: <20100820052203.GD31720@valinux.co.jp> References: <20100819183242.e5078819@shadowfax.no-ip.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-2022-jp Content-Disposition: inline In-Reply-To: <20100819183242.e5078819@shadowfax.no-ip.com> List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Adnan Khaleel Cc: qemu-devel@nongnu.org On Thu, Aug 19, 2010 at 01:32:42PM -0500, Adnan Khaleel wrote: > Isaku, > > I'm having some difficulties building the sources, I get the following message > > *akhaleel@yar95 qemu-q35 $ ./configure --help > : bad interpreter: No such file or directory > > And I get a similar error while compiling seabios as well. > > What shell are you using or am I missing something? I'm compiling from a > typical bash shell and using gcc v4.4.0. I'm not sure. configure script isn't modified. Can you compile normal qemu? The first line of the script is #!/bin/sh. I suppose you have /bin/sh. > In vgabios, there is a requirement for bcc. Is that borland C compiler? No. Most Linux destro has bcc package. Just you need to install it like yum install bcc or something. Thanks, > > Thanks > > Adnan > > ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ > From: Isaku Yamahata [mailto:yamahata@valinux.co.jp] > To: Adnan Khaleel [mailto:adnan@khaleel.us] > Cc: qemu-devel@nongnu.org > Sent: Wed, 18 Aug 2010 22:19:04 -0500 > Subject: Re: [Qemu-devel] Template for developing a Qemu device with PCIe > and MSI-X > > On Wed, Aug 18, 2010 at 02:10:10PM -0500, Adnan Khaleel wrote: > > Hello Qemu developers, > > > > I'm interested in developing a device model that plugs into Qemu that is > based > > on a PCIe interface and uses MSI-X. My goal is to ultimately attach a GPU > > simulator to this PCIe interface and use the entire platfom (Qemu + GPU > > simulator) for studying cpu, gpu interactions. > > > > I'm not terribly familiar with the Qemu device model and I'm looking for > some > > assistance, perhaps a starting template for pcie and msi-x that would > offer the > > basic functionality that I could then build upon. > > > > I have looked at the various devices that already modelled that are > included > > with Qemu (v0.12.5 at least) and I've noticed several a few pci devices, > eg; > > ne2k and cirrus-pci etc, however only one device truly seems to utilize > both > > the technologies that I'm interested in and that is the virtio-pci.c > > > > I'm not sure what virtio-pci does so I'm not sure if that is a suitable > > starting point for me. > > > > Any help, suggestions etc would be extremely helpful and much > appreciated. > > Qemu doesn't support pcie at the moment. > Only partial patches have been merged, still more patches have to > be merged for pcie to fully work. The following repo is available. > > git clone http://people.valinux.co.jp/~yamahata/qemu/q35/qemu > git clone http://people.valinux.co.jp/~yamahata/qemu/q35/seabios > git clone http://people.valinux.co.jp/~yamahata/qemu/q35/vgabios > > Note: patched seabios and vgabios are needed, you have to pass ACPI DSDT > for q35. > example: > qemu-system-x86_64 -M pc_q35 -acpitable load_header,data=roms/seabios/src/ > q35-acpi-dsdt.aml > > This repo is for those who want to try/develop pcie support, > not for upstream merge. So they include patches unsuitable for upstream. > The repo includes pcie port switch emulator which utilize pcie and > MSI(not MSI-X). > > The difference between PCI device and PCIe device is configuration > space size. > By setting PCIDeviceInfo::is_express = 1, you'll get 4K configuration > space. Helper functions for pcie are found in qemu/hw/pcie.c > For msi-x, see qemu/hw/msix.c. > > Thanks, > -- > yamahata > -- yamahata From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from [140.186.70.92] (port=44929 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OmXyA-0003xf-Jl for qemu-devel@nongnu.org; Fri, 20 Aug 2010 16:13:44 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OmXy8-0003Av-Kr for qemu-devel@nongnu.org; Fri, 20 Aug 2010 16:13:42 -0400 Received: from smtp.mailix.net ([66.11.225.183]:46251) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OmXy8-0003Ak-8g for qemu-devel@nongnu.org; Fri, 20 Aug 2010 16:13:40 -0400 From: "Adnan Khaleel" Subject: =?iso-8859-1?Q?Re=3A_=5BQemu-devel=5D_Template_for_developing_a_Qe?= =?iso-8859-1?Q?mu_device_with=09PCIe=3Fand_MSI-X?= In-Reply-To: 20100820052203.GD31720@valinux.co.jp Message-ID: <20100820201336.01013876@shadowfax.no-ip.com> Date: Fri, 20 Aug 2010 15:13:36 -0500 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="-----------56fcb61b88d4201a83393bf9be0de453" Reply-To: adnan@khaleel.us List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Isaku Yamahata Cc: qemu-devel@nongnu.org This is a multi-part message in MIME format. -------------56fcb61b88d4201a83393bf9be0de453 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I'm not sure what the problem was but I had checked out the code on a wi= ndows machine and then copied it over to a linux box. That was causing p= roblems somehow. I've managed to compile qemu and seabios but vgabios is taking forever t= o compile. The problem seems to be bcc. I've tried a simple helloworld p= rogram and even that seems to be taking forever to compile. Not sure if = there is something wrong with my bcc install. AK =5F=5F=5F=5F=5F =20 From: Isaku Yamahata [mailto:yamahata@valinux.co.jp] To: Adnan Khaleel [mailto:adnan@khaleel.us] Cc: qemu-devel@nongnu.org Sent: Fri, 20 Aug 2010 00:22:03 -0500 Subject: Re: [Qemu-devel] Template for developing a Qemu device with PCI= e=3Fand MSI-X On Thu, Aug 19, 2010 at 01:32:42PM -0500, Adnan Khaleel wrote: > Isaku, >=20 > I'm having some difficulties building the sources, I get the followi= ng message >=20 > *akhaleel@yar95 qemu-q35 $ ./configure --help > : bad interpreter: No such file or directory >=20 > And I get a similar error while compiling seabios as well. > > What shell are you using or am I missing something=3F I'm compiling = from a > typical bash shell and using gcc v4.4.0. =20 I'm not sure. configure script isn't modified. Can you compile normal qemu=3F The first line of the script is #!/bin/sh. I suppose you have /bin/sh. =20 =20 > In vgabios, there is a requirement for bcc. Is that borland C compil= er=3F =20 No. Most Linux destro has bcc package. Just you need to install it lik= e yum install bcc or something. =20 Thanks, =20 =20 >=20 > Thanks >=20 > Adnan >=20 > =E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81= =E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81= =E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81= =E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81= =E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81 > From: Isaku Yamahata [mailto:yamahata@valinux.co.jp] > To: Adnan Khaleel [mailto:adnan@khaleel.us] > Cc: qemu-devel@nongnu.org > Sent: Wed, 18 Aug 2010 22:19:04 -0500 > Subject: Re: [Qemu-devel] Template for developing a Qemu device = with PCIe > and MSI-X >=20 > On Wed, Aug 18, 2010 at 02:10:10PM -0500, Adnan Khaleel wrote: > > Hello Qemu developers, > > > > I'm interested in developing a device model that plugs into Qe= mu that is > based > > on a PCIe interface and uses MSI-X. My goal is to ultimately a= ttach a GPU > > simulator to this PCIe interface and use the entire platfom (Q= emu + GPU > > simulator) for studying cpu, gpu interactions. > > > > I'm not terribly familiar with the Qemu device model and I'm l= ooking for > some > > assistance, perhaps a starting template for pcie and msi-x tha= t would > offer the > > basic functionality that I could then build upon. > > > > I have looked at the various devices that already modelled tha= t are > included > > with Qemu (v0.12.5 at least) and I've noticed several a few pc= i devices, > eg; > > ne2k and cirrus-pci etc, however only one device truly seems t= o utilize > both > > the technologies that I'm interested in and that is the virtio= -pci.c > > > > I'm not sure what virtio-pci does so I'm not sure if that is a= suitable > > starting point for me. > > > > Any help, suggestions etc would be extremely helpful and much > appreciated. >=20 > Qemu doesn't support pcie at the moment. > Only partial patches have been merged, still more patches have t= o > be merged for pcie to fully work. The following repo is availabl= e. >=20 > git clone http://people.valinux.co.jp/~yamahata/qemu/q35/qemu > git clone http://people.valinux.co.jp/~yamahata/qemu/q35/seabios > git clone http://people.valinux.co.jp/~yamahata/qemu/q35/vgabios >=20 > Note: patched seabios and vgabios are needed, you have to pass A= CPI DSDT > for q35. > example: > qemu-system-x86=5F64 -M pc=5Fq35 -acpitable load=5Fheader,data= =3Droms/seabios/src/ > q35-acpi-dsdt.aml >=20 > This repo is for those who want to try/develop pcie support, > not for upstream merge. So they include patches unsuitable for u= pstream. > The repo includes pcie port switch emulator which utilize pcie a= nd > MSI(not MSI-X). >=20 > The difference between PCI device and PCIe device is configurati= on > space size. > By setting PCIDeviceInfo::is=5Fexpress =3D 1, you'll get 4K conf= iguration > space. Helper functions for pcie are found in qemu/hw/pcie.c > For msi-x, see qemu/hw/msix.c. >=20 > Thanks, > -- > yamahata >=20 =20 --=20 yamahata =20 -------------56fcb61b88d4201a83393bf9be0de453 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable
I'm not sure what the problem was but I= had checked out the code on a windows machine and then copied it over t= o a linux box. That was causing problems somehow.

I've managed to= compile qemu and seabios but vgabios is taking forever to compile. The = problem seems to be bcc. I've tried a simple helloworld program and even= that seems to be taking forever to compile. Not sure if there is someth= ing wrong with my bcc install.

AK
From: Isaku Yamahata [mailto:yamahata@= valinux.co.jp]
To: Adnan Khaleel [mailto:adnan@khaleel.us]
= Cc: qemu-devel@nongnu.org
Sent: Fri, 20 Aug 2010 00:22:= 03 -0500
Subject: Re: [Qemu-devel] Template for developing a Q= emu device with=09PCIe=3Fand MSI-X

On Thu, Aug 19, 2010 at 01:32:= 42PM -0500, Adnan Khaleel wrote:
> Isaku,
>
> I'm having some difficulties building the sources, I get the follow= ing message
>
> *akhaleel@yar95 qemu-q35 $ ./= configure --help
> : bad interpreter: No such file or directory
>
> And I get a similar error while compiling seabios as well.
>
> What shell are you using or am I missing something=3F I'm compiling= from a
> typical bash shell and using gcc v4.4.0.

I'm not sure. configure script isn't modified.
Can you compile normal qemu=3F
The first line of the script is #!/bin/sh. I suppose you have /bin/sh.

> In vgabios, there is a requirement for bcc. Is that borland C compi= ler=3F

No. Most Linux destro has bcc package. Just you need to install it like<= br> yum install bcc or something.

Thanks,


>
> Thanks
>
> Adnan
>
> =E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81= =E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81= =E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81= =E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81= =E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81
> From: Isaku Yamahata [mailto:yamahata@valinux.co.jp]
> To: Adnan Khaleel [mailto:a= dnan@khaleel.us]
> Cc: qemu-devel@nongnu.= org
> Sent: Wed, 18 Aug 2010 22:19:04 -0500
> Subject: Re: [Qemu-devel] Template for developing a Qemu device= with PCIe
> and MSI-X
>
> On Wed, Aug 18, 2010 at 02:10:10PM -0500, Adnan Khaleel wrote:<= br> > > Hello Qemu developers,
> >
> > I'm interested in developing a device model that plugs int= o Qemu that is
> based
> > on a PCIe interface and uses MSI-X. My goal is to ultimate= ly attach a GPU
> > simulator to this PCIe interface and use the entire platfo= m (Qemu + GPU
> > simulator) for studying cpu, gpu interactions.
> >
> > I'm not terribly familiar with the Qemu device model and I= 'm looking for
> some
> > assistance, perhaps a starting template for pcie and msi-x= that would
> offer the
> > basic functionality that I could then build upon.
> >
> > I have looked at the various devices that already modelled= that are
> included
> > with Qemu (v0.12.5 at least) and I've noticed several a fe= w pci devices,
> eg;
> > ne2k and cirrus-pci etc, however only one device truly see= ms to utilize
> both
> > the technologies that I'm interested in and that is the vi= rtio-pci.c
> >
> > I'm not sure what virtio-pci does so I'm not sure if that = is a suitable
> > starting point for me.
> >
> > Any help, suggestions etc would be extremely helpful and m= uch
> appreciated.
>
> Qemu doesn't support pcie at the moment.
> Only partial patches have been merged, still more patches have = to
> be merged for pcie to fully work. The following repo is availab= le.
>
> git clone http://people.valinux.co.jp/~yamahata/q= emu/q35/qemu
> git clone http://people.valinux.co.jp/~yamahat= a/qemu/q35/seabios
> git clone http://people.valinux.co.jp/~yamahat= a/qemu/q35/vgabios
>
> Note: patched seabios and vgabios are needed, you have to pass = ACPI DSDT
> for q35.
> example:
> qemu-system-x86=5F64 -M pc=5Fq35 -acpitable load=5Fheader,data= =3Droms/seabios/src/
> q35-acpi-dsdt.aml
>
> This repo is for those who want to try/develop pcie support, > not for upstream merge. So they include patches unsuitable for = upstream.
> The repo includes pcie port switch emulator which utilize pcie = and
> MSI(not MSI-X).
>
> The difference between PCI device and PCIe device is configurat= ion
> space size.
> By setting PCIDeviceInfo::is=5Fexpress =3D 1, you'll get 4K con= figuration
> space. Helper functions for pcie are found in qemu/hw/pcie.c > For msi-x, see qemu/hw/msix.c.
>
> Thanks,
> --
> yamahata
>

--
yamahata
-------------56fcb61b88d4201a83393bf9be0de453-- From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from [140.186.70.92] (port=33997 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OmZyY-0000yq-Cf for qemu-devel@nongnu.org; Fri, 20 Aug 2010 18:22:16 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OmZyW-0000zn-2P for qemu-devel@nongnu.org; Fri, 20 Aug 2010 18:22:14 -0400 Received: from smtp.mailix.net ([66.11.225.183]:10317) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OmZyV-0000zg-K9 for qemu-devel@nongnu.org; Fri, 20 Aug 2010 18:22:12 -0400 From: "Adnan Khaleel" Subject: =?iso-8859-1?Q?Re=3A_=5BQemu-devel=5D_Template_for_developing_a_Qe?= =?iso-8859-1?Q?mu_device_with=09PCIe=3Fand_MSI-X?= In-Reply-To: 20100820052203.GD31720@valinux.co.jp Message-ID: <20100820222208.19c41df1@shadowfax.no-ip.com> Date: Fri, 20 Aug 2010 17:22:08 -0500 MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="-----------c9f8c5ecbe18c4949b5f09fb51eece1c" Reply-To: adnan@khaleel.us List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: qemu-devel@nongnu.org Cc: Isaku Yamahata This is a multi-part message in MIME format. -------------c9f8c5ecbe18c4949b5f09fb51eece1c Content-Type: multipart/alternative; boundary="-----------251619693ab541d76724c9cd48bea539" -------------251619693ab541d76724c9cd48bea539 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable After messing around with bcc, as86 and ld86 I finally got the vgabios t= o compile. Everything works as it should I'm guessing. I've attached the output fro= m lspci from the guest. I'll spend sometime looking at the device models= and see how I can implement a model of what I'm interested in. Is the overall architecture of any qemu device still similar to how a ty= pical qemu pci device is or is there anything I should be aware of=3F Thanks again Isaku. AK =5F=5F=5F=5F=5F =20 From: Isaku Yamahata [mailto:yamahata@valinux.co.jp] To: Adnan Khaleel [mailto:adnan@khaleel.us] Cc: qemu-devel@nongnu.org Sent: Fri, 20 Aug 2010 00:22:03 -0500 Subject: Re: [Qemu-devel] Template for developing a Qemu device with PCI= e=3Fand MSI-X On Thu, Aug 19, 2010 at 01:32:42PM -0500, Adnan Khaleel wrote: > Isaku, >=20 > I'm having some difficulties building the sources, I get the followi= ng message >=20 > *akhaleel@yar95 qemu-q35 $ ./configure --help > : bad interpreter: No such file or directory >=20 > And I get a similar error while compiling seabios as well. > > What shell are you using or am I missing something=3F I'm compiling = from a > typical bash shell and using gcc v4.4.0. =20 I'm not sure. configure script isn't modified. Can you compile normal qemu=3F The first line of the script is #!/bin/sh. I suppose you have /bin/sh. =20 =20 > In vgabios, there is a requirement for bcc. Is that borland C compil= er=3F =20 No. Most Linux destro has bcc package. Just you need to install it lik= e yum install bcc or something. =20 Thanks, =20 =20 >=20 > Thanks >=20 > Adnan >=20 > =E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81= =E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81= =E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81= =E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81= =E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81 > From: Isaku Yamahata [mailto:yamahata@valinux.co.jp] > To: Adnan Khaleel [mailto:adnan@khaleel.us] > Cc: qemu-devel@nongnu.org > Sent: Wed, 18 Aug 2010 22:19:04 -0500 > Subject: Re: [Qemu-devel] Template for developing a Qemu device = with PCIe > and MSI-X >=20 > On Wed, Aug 18, 2010 at 02:10:10PM -0500, Adnan Khaleel wrote: > > Hello Qemu developers, > > > > I'm interested in developing a device model that plugs into Qe= mu that is > based > > on a PCIe interface and uses MSI-X. My goal is to ultimately a= ttach a GPU > > simulator to this PCIe interface and use the entire platfom (Q= emu + GPU > > simulator) for studying cpu, gpu interactions. > > > > I'm not terribly familiar with the Qemu device model and I'm l= ooking for > some > > assistance, perhaps a starting template for pcie and msi-x tha= t would > offer the > > basic functionality that I could then build upon. > > > > I have looked at the various devices that already modelled tha= t are > included > > with Qemu (v0.12.5 at least) and I've noticed several a few pc= i devices, > eg; > > ne2k and cirrus-pci etc, however only one device truly seems t= o utilize > both > > the technologies that I'm interested in and that is the virtio= -pci.c > > > > I'm not sure what virtio-pci does so I'm not sure if that is a= suitable > > starting point for me. > > > > Any help, suggestions etc would be extremely helpful and much > appreciated. >=20 > Qemu doesn't support pcie at the moment. > Only partial patches have been merged, still more patches have t= o > be merged for pcie to fully work. The following repo is availabl= e. >=20 > git clone http://people.valinux.co.jp/~yamahata/qemu/q35/qemu > git clone http://people.valinux.co.jp/~yamahata/qemu/q35/seabios > git clone http://people.valinux.co.jp/~yamahata/qemu/q35/vgabios >=20 > Note: patched seabios and vgabios are needed, you have to pass A= CPI DSDT > for q35. > example: > qemu-system-x86=5F64 -M pc=5Fq35 -acpitable load=5Fheader,data= =3Droms/seabios/src/ > q35-acpi-dsdt.aml >=20 > This repo is for those who want to try/develop pcie support, > not for upstream merge. So they include patches unsuitable for u= pstream. > The repo includes pcie port switch emulator which utilize pcie a= nd > MSI(not MSI-X). >=20 > The difference between PCI device and PCIe device is configurati= on > space size. > By setting PCIDeviceInfo::is=5Fexpress =3D 1, you'll get 4K conf= iguration > space. Helper functions for pcie are found in qemu/hw/pcie.c > For msi-x, see qemu/hw/msix.c. >=20 > Thanks, > -- > yamahata >=20 =20 --=20 yamahata =20 -------------251619693ab541d76724c9cd48bea539 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable
After messing around with bcc, as86 and= ld86 I finally got the vgabios to compile.

Everything works as i= t should I'm guessing. I've attached the output from lspci from the gues= t. I'll spend sometime looking at the device models and see how I can im= plement a model of what I'm interested in.

Is the overall archite= cture of any qemu device still similar to how a typical qemu pci device = is or is there anything I should be aware of=3F

Thanks again Isak= u.

AK
<= b>From: Isaku Yamahata [mailto:yamahata@valinux.co.jp]
To:= Adnan Khaleel [mailto:adnan@khaleel.us]
Cc: qemu-devel@nongnu= .org
Sent: Fri, 20 Aug 2010 00:22:03 -0500
Subject: = Re: [Qemu-devel] Template for developing a Qemu device with=09PCIe=3Fand= MSI-X

On Thu, Aug 19, 2010 at 01:32:42PM -0500, Adnan Khaleel wr= ote:
> Isaku,
>
> I'm having some difficulties building the sources, I get the follow= ing message
>
> *akhaleel@yar95 qemu-q35 $ ./= configure --help
> : bad interpreter: No such file or directory
>
> And I get a similar error while compiling seabios as well.
>
> What shell are you using or am I missing something=3F I'm compiling= from a
> typical bash shell and using gcc v4.4.0.

I'm not sure. configure script isn't modified.
Can you compile normal qemu=3F
The first line of the script is #!/bin/sh. I suppose you have /bin/sh.

> In vgabios, there is a requirement for bcc. Is that borland C compi= ler=3F

No. Most Linux destro has bcc package. Just you need to install it like<= br> yum install bcc or something.

Thanks,


>
> Thanks
>
> Adnan
>
> =E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81= =E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81= =E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81= =E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81= =E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81
> From: Isaku Yamahata [mailto:yamahata@valinux.co.jp]
> To: Adnan Khaleel [mailto:a= dnan@khaleel.us]
> Cc: qemu-devel@nongnu.= org
> Sent: Wed, 18 Aug 2010 22:19:04 -0500
> Subject: Re: [Qemu-devel] Template for developing a Qemu device= with PCIe
> and MSI-X
>
> On Wed, Aug 18, 2010 at 02:10:10PM -0500, Adnan Khaleel wrote:<= br> > > Hello Qemu developers,
> >
> > I'm interested in developing a device model that plugs int= o Qemu that is
> based
> > on a PCIe interface and uses MSI-X. My goal is to ultimate= ly attach a GPU
> > simulator to this PCIe interface and use the entire platfo= m (Qemu + GPU
> > simulator) for studying cpu, gpu interactions.
> >
> > I'm not terribly familiar with the Qemu device model and I= 'm looking for
> some
> > assistance, perhaps a starting template for pcie and msi-x= that would
> offer the
> > basic functionality that I could then build upon.
> >
> > I have looked at the various devices that already modelled= that are
> included
> > with Qemu (v0.12.5 at least) and I've noticed several a fe= w pci devices,
> eg;
> > ne2k and cirrus-pci etc, however only one device truly see= ms to utilize
> both
> > the technologies that I'm interested in and that is the vi= rtio-pci.c
> >
> > I'm not sure what virtio-pci does so I'm not sure if that = is a suitable
> > starting point for me.
> >
> > Any help, suggestions etc would be extremely helpful and m= uch
> appreciated.
>
> Qemu doesn't support pcie at the moment.
> Only partial patches have been merged, still more patches have = to
> be merged for pcie to fully work. The following repo is availab= le.
>
> git clone http://people.valinux.co.jp/~yamahata/q= emu/q35/qemu
> git clone http://people.valinux.co.jp/~yamahat= a/qemu/q35/seabios
> git clone http://people.valinux.co.jp/~yamahat= a/qemu/q35/vgabios
>
> Note: patched seabios and vgabios are needed, you have to pass = ACPI DSDT
> for q35.
> example:
> qemu-system-x86=5F64 -M pc=5Fq35 -acpitable load=5Fheader,data= =3Droms/seabios/src/
> q35-acpi-dsdt.aml
>
> This repo is for those who want to try/develop pcie support, > not for upstream merge. So they include patches unsuitable for = upstream.
> The repo includes pcie port switch emulator which utilize pcie = and
> MSI(not MSI-X).
>
> The difference between PCI device and PCIe device is configurat= ion
> space size.
> By setting PCIDeviceInfo::is=5Fexpress =3D 1, you'll get 4K con= figuration
> space. Helper functions for pcie are found in qemu/hw/pcie.c > For msi-x, see qemu/hw/msix.c.
>
> Thanks,
> --
> yamahata
>

--
yamahata
-------------251619693ab541d76724c9cd48bea539-- -------------c9f8c5ecbe18c4949b5f09fb51eece1c Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="pci.txt" 00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM Co= ntroller 00:01.0 VGA compatible controller: Cirrus Logic GD 5446 00:04.0 PCI bridge: Intel Corporation X58 I/O Hub PCI Express Root Port = 0 (rev 02) 00:18.0 PCI bridge: Intel Corporation X58 I/O Hub PCI Express Root Port = 0 (rev 02) 00:18.1 PCI bridge: Intel Corporation X58 I/O Hub PCI Express Root Port = 0 (rev 02) 00:18.2 PCI bridge: Intel Corporation X58 I/O Hub PCI Express Root Port = 0 (rev 02) 00:18.3 PCI bridge: Intel Corporation X58 I/O Hub PCI Express Root Port = 0 (rev 02) 00:18.4 PCI bridge: Intel Corporation X58 I/O Hub PCI Express Root Port = 0 (rev 02) 00:18.5 PCI bridge: Intel Corporation X58 I/O Hub PCI Express Root Port = 0 (rev 02) 00:19.0 PCI bridge: Intel Corporation X58 I/O Hub PCI Express Root Port = 0 (rev 02) 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92) 00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Contr= oller (rev 02) 00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (= rev 02) 36:00.0 PCI bridge: Texas Instruments Device 8232 (rev 02) 36:00.1 PCI bridge: Texas Instruments Device 8232 (rev 02) 36:00.2 PCI bridge: Texas Instruments Device 8232 (rev 02) 36:00.3 PCI bridge: Texas Instruments Device 8232 (rev 02) 36:00.4 PCI bridge: Texas Instruments Device 8232 (rev 02) 36:00.5 PCI bridge: Texas Instruments Device 8232 (rev 02) 36:00.6 PCI bridge: Texas Instruments Device 8232 (rev 02) 36:00.7 PCI bridge: Texas Instruments Device 8232 (rev 02) 37:00.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 37:01.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 37:02.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 37:03.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 37:04.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 37:05.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 37:06.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 37:07.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 37:08.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 37:09.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 37:0a.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 37:0b.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 37:0c.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 37:0d.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 37:0e.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 37:0f.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 48:00.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 48:01.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 48:02.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 48:03.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 48:04.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 48:05.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 48:06.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 48:07.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 48:08.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 48:09.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 48:0a.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 48:0b.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 48:0c.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 48:0d.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 48:0e.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 48:0f.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 59:00.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 59:01.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 59:02.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 59:03.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 59:04.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 59:05.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 59:06.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 59:07.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 59:08.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 59:09.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 59:0a.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 59:0b.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 59:0c.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 59:0d.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 59:0e.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 59:0f.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 6a:00.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 6a:01.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 6a:02.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 6a:03.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 6a:04.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 6a:05.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 6a:06.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 6a:07.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 6a:08.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 6a:09.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 6a:0a.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 6a:0b.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 6a:0c.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 6a:0d.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 6a:0e.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 6a:0f.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 7b:00.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 7b:01.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 7b:02.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 7b:03.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 7b:04.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 7b:05.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 7b:06.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 7b:07.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 7b:08.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 7b:09.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 7b:0a.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 7b:0b.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 7b:0c.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 7b:0d.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 7b:0e.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 7b:0f.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 8c:00.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 8c:01.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 8c:02.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 8c:03.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 8c:04.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 8c:05.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 8c:06.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 8c:07.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 8c:08.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 8c:09.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 8c:0a.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 8c:0b.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 8c:0c.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 8c:0d.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 8c:0e.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 8c:0f.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 9d:00.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 9d:01.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 9d:02.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 9d:03.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 9d:04.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 9d:05.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 9d:06.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 9d:07.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 9d:08.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 9d:09.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 9d:0a.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 9d:0b.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 9d:0c.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 9d:0d.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 9d:0e.0 PCI bridge: Texas Instruments Device 8233 (rev 01) 9d:0f.0 PCI bridge: Texas Instruments Device 8233 (rev 01) ae:00.0 PCI bridge: Texas Instruments Device 8233 (rev 01) ae:01.0 PCI bridge: Texas Instruments Device 8233 (rev 01) ae:02.0 PCI bridge: Texas Instruments Device 8233 (rev 01) ae:03.0 PCI bridge: Texas Instruments Device 8233 (rev 01) ae:04.0 PCI bridge: Texas Instruments Device 8233 (rev 01) ae:05.0 PCI bridge: Texas Instruments Device 8233 (rev 01) ae:06.0 PCI bridge: Texas Instruments Device 8233 (rev 01) ae:07.0 PCI bridge: Texas Instruments Device 8233 (rev 01) ae:08.0 PCI bridge: Texas Instruments Device 8233 (rev 01) ae:09.0 PCI bridge: Texas Instruments Device 8233 (rev 01) ae:0a.0 PCI bridge: Texas Instruments Device 8233 (rev 01) ae:0b.0 PCI bridge: Texas Instruments Device 8233 (rev 01) ae:0c.0 PCI bridge: Texas Instruments Device 8233 (rev 01) ae:0d.0 PCI bridge: Texas Instruments Device 8233 (rev 01) ae:0e.0 PCI bridge: Texas Instruments Device 8233 (rev 01) ae:0f.0 PCI bridge: Texas Instruments Device 8233 (rev 01) c0:00.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet = Controller (rev 03) c0:01.0 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Trito= n II] c0:1c.0 PCI bridge: Digital Equipment Corporation DECchip 21154 (rev 05) c0:1d.0 PCI bridge: Digital Equipment Corporation DECchip 21154 (rev 05) c0:1e.0 PCI bridge: Digital Equipment Corporation DECchip 21154 (rev 05) c0:1f.0 PCI bridge: Digital Equipment Corporation DECchip 21154 (rev 05) -------------c9f8c5ecbe18c4949b5f09fb51eece1c-- From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from [140.186.70.92] (port=59412 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OoOdS-0006Rw-Pv for qemu-devel@nongnu.org; Wed, 25 Aug 2010 18:40:00 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OoOdQ-0002Ev-S0 for qemu-devel@nongnu.org; Wed, 25 Aug 2010 18:39:58 -0400 Received: from smtp.mailix.net ([66.11.225.183]:36598) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OoOdQ-0002En-7C for qemu-devel@nongnu.org; Wed, 25 Aug 2010 18:39:56 -0400 From: "Adnan Khaleel" Subject: =?iso-8859-1?Q?Re=3A_=5BQemu-devel=5D_Template_for_developing_a_Qe?= =?iso-8859-1?Q?mu_device_with_PCIe=09and_MSI-X?= In-Reply-To: 20100819031904.GG11421@valinux.co.jp Message-ID: <20100825223950.7e664cb4@shadowfax.no-ip.com> Date: Wed, 25 Aug 2010 17:39:50 -0500 MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="-----------54b14120bc72690a9f3015af90802d06" Reply-To: adnan@khaleel.us List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Isaku Yamahata Cc: qemu-devel@nongnu.org This is a multi-part message in MIME format. -------------54b14120bc72690a9f3015af90802d06 Content-Type: multipart/alternative; boundary="-----------cbc979a24f4e11807332f62c8d22d8f8" -------------cbc979a24f4e11807332f62c8d22d8f8 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi Isaku, I've made some progress in coding the device template but its no where n= ear complete. I've created some files and am attaching it to this note. Based on what = I could gather from the pcie source files I've made a stab at creating a= simple model. I've also attached a file for a simple pci device that wo= rks under regular Qemu. I would like to duplicate its functionality in y= our pcie environment for starters. Could you please take a look at the files I've created and tell me if I'= ve understood your pcie model correctly. Any help will be truly apprecia= ted. Adnan The five files I've modified from your git repository are as follows hw/pci=5Fids.h // Added vendor id defines hw/pc=5Fq35.c // Device instantiation hw/pcie=5Fmsix=5Ftemplate.h // Device header file hw/pcie=5Fmsix=5Ftemplate.c // Device file Makefile.objs // Added pcie=5Fmsix=5Ftemplate.o to lis= t of objects being built Everything should compile without any warnings or errors. The last file: sc=5Flink=5Fpci.c Is the original PCI device that I'm trying to convert into being PCIe an= d MSI-X and is included merely for reference to help you understand what= I'd like to achieve in your environment. =5F=5F=5F=5F=5F =20 From: Isaku Yamahata [mailto:yamahata@valinux.co.jp] To: Adnan Khaleel [mailto:adnan@khaleel.us] Cc: qemu-devel@nongnu.org Sent: Wed, 18 Aug 2010 22:19:04 -0500 Subject: Re: [Qemu-devel] Template for developing a Qemu device with PCI= e and MSI-X On Wed, Aug 18, 2010 at 02:10:10PM -0500, Adnan Khaleel wrote: > Hello Qemu developers, > > I'm interested in developing a device model that plugs into Qemu tha= t is based > on a PCIe interface and uses MSI-X. My goal is to ultimately attach = a GPU > simulator to this PCIe interface and use the entire platfom (Qemu + = GPU > simulator) for studying cpu, gpu interactions. >=20 > I'm not terribly familiar with the Qemu device model and I'm looking= for some > assistance, perhaps a starting template for pcie and msi-x that woul= d offer the > basic functionality that I could then build upon. >=20 > I have looked at the various devices that already modelled that are = included > with Qemu (v0.12.5 at least) and I've noticed several a few pci devi= ces, eg; > ne2k and cirrus-pci etc, however only one device truly seems to util= ize both > the technologies that I'm interested in and that is the virtio-pci.c >=20 > I'm not sure what virtio-pci does so I'm not sure if that is a suita= ble > starting point for me. >=20 > Any help, suggestions etc would be extremely helpful and much apprec= iated. =20 Qemu doesn't support pcie at the moment. Only partial patches have been merged, still more patches have to be merged for pcie to fully work. The following repo is available. =20 git clone http://people.valinux.co.jp/~yamahata/qemu/q35/qemu git clone http://people.valinux.co.jp/~yamahata/qemu/q35/seabios git clone http://people.valinux.co.jp/~yamahata/qemu/q35/vgabios =20 Note: patched seabios and vgabios are needed, you have to pass ACPI DS= DT for q35. example: qemu-system-x86=5F64 -M pc=5Fq35 -acpitable load=5Fheader,data=3Droms/= seabios/src/q35-acpi-dsdt.aml =20 This repo is for those who want to try/develop pcie support, not for upstream merge. So they include patches unsuitable for upstrea= m. The repo includes pcie port switch emulator which utilize pcie and MSI(not MSI-X). =20 The difference between PCI device and PCIe device is configuration space size. By setting PCIDeviceInfo::is=5Fexpress =3D 1, you'll get 4K configurat= ion space. Helper functions for pcie are found in qemu/hw/pcie.c For msi-x, see qemu/hw/msix.c. =20 Thanks, --=20 yamahata =20 -------------cbc979a24f4e11807332f62c8d22d8f8 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
Hi Isaku,

I've made some progres= s in coding the device template but its no where near complete.

I= 've created some files and am attaching it to this note. Based on what I= could gather from the pcie source files I've made a stab at creating a = simple model. I've also attached a file for a simple pci device that wor= ks under regular Qemu. I would like to duplicate its functionality in yo= ur pcie environment for starters.

Could you please take a look at= the files I've created and tell me if I've understood your pcie model c= orrectly. Any help will be truly appreciated.

Adnan

The fi= ve files I've modified from your git repository are as follows

hw= /pci=5Fids.h          =           // Added vendor i= d defines
hw/pc=5Fq35.c       &nbs= p;            // = Device instantiation
hw/pcie=5Fmsix=5Ftemplate.h  // Device header file
hw/pcie=5Fmsi= x=5Ftemplate.c  // Device file
Makefile.objs   &n= bsp;           &n= bsp;   // Added pcie=5Fmsix=5Ftemplate.o to list of objects be= ing built

Everything should compile without any warnings or error= s.

The last file:
sc=5Flink=5Fpci.c
Is the original PCI dev= ice that I'm trying to convert into being PCIe and MSI-X and is included= merely for reference to help you understand what I'd like to achieve in= your environment.


From: Isaku Yamahata [mailto:yamahata@valinux.co.jp]=
To: Adnan Khaleel [mailto:adnan@khaleel.us]
Cc: qem= u-devel@nongnu.org
Sent: Wed, 18 Aug 2010 22:19:04 -0500
Subject: Re: [Qemu-devel] Template for developing a Qemu device wit= h PCIe=09and MSI-X

On Wed, Aug 18, 2010 at 02:10:10PM -0500, Adna= n Khaleel wrote:
> Hello Qemu developers,
>
> I'm interested in developing a device model that plugs into Qemu th= at is based
> on a PCIe interface and uses MSI-X. My goal is to ultimately attach= a GPU
> simulator to this PCIe interface and use the entire platfom (Qemu += GPU
> simulator) for studying cpu, gpu interactions.
>
> I'm not terribly familiar with the Qemu device model and I'm lookin= g for some
> assistance, perhaps a starting template for pcie and msi-x that wou= ld offer the
> basic functionality that I could then build upon.
>
> I have looked at the various devices that already modelled that are= included
> with Qemu (v0.12.5 at least) and I've noticed several a few pci dev= ices, eg;
> ne2k and cirrus-pci etc, however only one device truly seems to uti= lize both
> the technologies that I'm interested in and that is the virtio-pci.= c
>
> I'm not sure what virtio-pci does so I'm not sure if that is a suit= able
> starting point for me.
>
> Any help, suggestions etc would be extremely helpful and much appre= ciated.

Qemu doesn't support pcie at the moment.
Only partial patches have been merged, still more patches have to
be merged for pcie to fully work. The following repo is available.

git clone http://people.valinux.co.jp/~yamahata/qemu/q35/q= emu
git clone http://people.valinux.co.jp/~yamahata/qemu/q3= 5/seabios
git clone http://people.valinux.co.jp/~yamahata/qemu/q3= 5/vgabios

Note: patched seabios and vgabios are needed, you have to pass ACPI DSDT=
for q35.
example:
qemu-system-x86=5F64 -M pc=5Fq35 -acpitable load=5Fheader,data=3Droms/se= abios/src/q35-acpi-dsdt.aml

This repo is for those who want to try/develop pcie support,
not for upstream merge. So they include patches unsuitable for upstream.=
The repo includes pcie port switch emulator which utilize pcie and
MSI(not MSI-X).

The difference between PCI device and PCIe device is configuration
space size.
By setting PCIDeviceInfo::is=5Fexpress =3D 1, you'll get 4K configuratio= n
space. Helper functions for pcie are found in qemu/hw/pcie.c
For msi-x, see qemu/hw/msix.c.

Thanks,
--
yamahata
-------------cbc979a24f4e11807332f62c8d22d8f8-- -------------54b14120bc72690a9f3015af90802d06 Content-Type: application/zip Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="pcie_msix_template.zip" UEsDBBQAAAAIAFWJGT3T/ahh2Q0AADMoAAAIAAAAcGNfcTM1LmOlWvtz2rgT/71/hSY314GEJEAe TZNrZwiQxncEOCB9TL8Zj7Fl0NWvWnbS3E3/9++uHkY2kKZ3TJtgSbva52dXcg53X5Bd8mf/5paM u2T6yDMakn6YB04WpzCFs904eUzZYpmRmlsn7WbzaB9+HJMrZ54yl5JLGgRO6qnVY5qGjHMWR4Rx sqQpnT+SRepEGfUaxE8pJbFP3KWTLmiDZDFxokeS0JQDQTzPHBaxaEEc4sKuyA8WZ0vgxGM/e3BS Cus94nAeu8wBlsSL3TykUeZkuKXPAspJLVtSsjNVFDt1sY9HnQAZsojgtJ4lDyxbxnlGUsoz0AfZ NGCRG+QeSqKnAxYytQmSC4Nw5Aescw6qoMANEsYe8/E3Ffol+TxgfNkgHkPu8zyDQY6DLo2QCrQ5 jFPCwYbIDJgwUEAovZJRLMONEjRupszFceRhGYdlfZiQys/TCDamgsyLwXxi37+om+EIUvhxEMQP qKMbRx5D1fi58uIM5p15fE+FWtL7UZyhu4Uo6JFk5Wk1xZdOEJA5VeaDzVmE3HBUa5aiGDyDeGBO QJI4FftWNT7Qclz3yXR0NfvQmfSJNSXjyei91ev3yE5nCs87DfLBml2PbmcEVkw6w9knMroineEn 8oc17DVI/+N40p9OyWiC3Kyb8cDqw7A17A5ue9bwHbkE0uFoRgbWjTUDvrOR2FNxs/pT5HfTn3Sv 4bFzaQ2s2acG8rqyZkPkfDWakA4ZdyYzq3s76EzI+HYyHk37IEQPOA+t4dUENurf9IezA9gYxkj/ PTyQ6XVnMNBKdm5BjQkKSrqj8aeJ9e56Rq5Hg14fBi/7IF/nctCXu4F23UHHummQXuem864vqEbA RSiJK6WY5MN1H0dx1w78686s0RD16Y6Gswk8NkDdyayg/mBN+w3SmVhTtMzVZHQjNEXrAtFI8AHS YV8yQsuXHQRL8Pl22i94kl6/MwB24K1h1aHo5MMXh8LR5M+jE0AFlnCakbnDIXYSl3AJSLQMSOuI 9LoBP1tNMbnhY3HnS04+OaGzdDKH/PaovzkZuXcCFuXfID7JX8nbbRzed8hALJMQycnvTuJE5I+D P3SkQspATsA/KTwkReIeuA0CKQ8o6Nyz4FGCA6NeEd1IErB56qSPSCrQUQPdBXmMc+LCLildoQeB 9JeYgQwk2uBYHnk0FSmUQVoW+fRueEsGlHOYe0cjmkLKjQX6iEiRIARoqnEKBAe0RsIrFKXAlKsY +AvwuyAUAJGK3e8BhjD723q3gUY1yPIa2BYUSEmcIF1d4Dx4ESRRdNussFLW02C9jBNQDFyGqj4w CTKAun4eiBBFI2sgwPTQYHBRwDe9p5IVC5MAXEBALyxKjyA7cqikOGrwoww/IGRKqTazMOgmSxdm 9oFnGIM1PQplLuBa/0/gZg5CBh5ZOoC4KXUpuwcRZRH8OVcGsapZEqOVWS8I8xGkoQaAxL8tsyw5 Pzx8eHg4WET5QZwuDlVB4odvRVL+ohCc7CwfDpY7xjNEdenZ99zqArY2YEOn4C1oeZy1W62T47W1 1E7jONswnCcQF9QJN0x58UO0dTLk7JsNSZtg9JXn50HsfikPAeQA3JTHHGgF4kPxszwR0YqgPJzn vLJHDM1RZSyMIwZ4VrHkg+36i/LYMqGZjfBXHn5wMnfpxYu1zVlc2Yl5FZVzPt/Pl9JHxvDXoxM5 4lGfRZTcdD7aUGjty9spab94wbH1cbFR4JCDEfz3qM1ixLrPxtI78ob8Q5rfWn6zgb9eNcn3i6eI 25uoj/xTpD56dbqdOv26Ttk6bpDWCdJoovuYYR2xQTsVgjb0l1lt3LV69B5blt1F6C5tXE4BSZ7+ ANVlzsnuMuaZDX5uFCMixHNef/GP4JGDmGd2Bol+f1Ea4NTFdZXBfL4+iI0RKNUsj0LbDA0XL0+0 TpFJEBvrQa7+FEd2K2z9qLoAk83G3YoJoVExvBINe3ZAWv9z6/QOTIxDh7tIgA0l/pIWJvPTc+9z +zXZJ0etu3OMhNPzzy0Xnlv+3UFTohMoHafQYIPdgSfXpgesewULX2P9nDfPvaOmYNA8aFGgfINL BTypKIWmwR63x/blpGe969u9/nv7sgPdh/60z55aCgG0cu5Re9vS6e0lhJjBuGZ1r1/bPVgx7UM/ 1OtMPgG/q87tYEb2SKsulEG0r0EAgNBbpLzA+CC/bZFMzO7t1ck/RViib6SJd+fpRTGsgmp9I1Pw PbHZ/lZZVtxkNAK3IlhXc1GSQiD5NYgCKCXsbxqL73DAErD/q7fT0GT1Fct5Ctwk2sv0U+ki8gdF uBqipcDNdeyX/IJHQ0tTN4SAmAN6aAmyJMgXGA16RqQyrLbVXM3IbbKqQvYCMFXtUJundSXpdzOk oQpDAsgsxDg8lmF43jzeEsAYuadHWpqVS47akvsGsz7DonSTSZFJUSelQUuIpCx6LOy5DdM2mrkh FG5omGkIVEHbm5lxAw2WPR2MZtMXh4e/YN/pV8ehrvgbhk0uw4k9GY1mYzx/6M9pdcXteDqb9Ds3 xYqz6ore6MOwtKYFTGjA6Q83a/1ws7UV65vBXnBy9p+KnDaEzufm/smdQLGzA/Fdx4kACT8SwA3Y rMDAEBdHyyCgfLa392TK6ikUo7QUHIrPz03pjQH4E0HYhij0o+1h+NOhiARPZmr7BFK1yMPCMk9A NtSli0rWarp9UmsBdJ6J/7sQXEoC06zapP/WmkWlfaZFT57M639t1FLBl6JswsuXxbr9t/jzAAB0 1QxIIwF1q30mmHPtiXI/szXudRKuxz12LGNksqvPARfmnGha9Ew5B4oeaXVKUA2PWVNEKGkGUlyv KVL31Z1ZZJ6Tf9gFnF785xwrpFGBUagngqPkMDNCmj9OuecFygZRfhAcetn+WxEWmlqny5q9Vx6R FjfsXHHWKnaK+JHNVdNso0ogvaGD2uLAjU6sTBtZ/l/9KkxSPrZKl5pW3tQc/dilW7xZMeYzGG2C iC2uMc3yvQLQqBZzAjCNuMMozuPoGjx+6WN5ARPFgX09xkEKYCxCh8BRxJO33WoDLq/mgc2Go5/k BeI6npcCEOBXdNd2O8jTpjjx7M5RCCnv8wi+0DSigY3vIiIn/DkiN/QCaDmeR4N6pd5PbuQmuR3G Hg30aVWcqVURWNloToP4wT5e2CFVxpIvBMwhSXT1oesvppnworzDKB8ldRUrj6ozgKqb0mcbj+UX KyHd5Ws7AJ+CM0Tx2FeF+ysNc7wSkMrBl+owhMfG4bP2yWs5aHHHSr8qLdRyc/9eyu6pFfkx6OOZ Vw94d6ieIFmnd8omvW6WBmCPIE6SRxvsn6VxEFB1bLOmHa1omrnmPoYJQNM19cKY2/xIISlYAvTl MrxXblWz96FIHDFplGgnCGIXtQRfi3c5Qex4UP3DQ7w8KhpT4UeB9OjtOH1cJZEMhmqQk+cGsPTm MwMXPy83hOLLtVjUKqoIkKJrZW01WlNIJhyPh1L8vbIf3io17/QaMwhgrfBCKDj+XVNgX46U+ibK /belzUorNNdCTBjjNU2+BO9AxDTK/Bqkfax11ZkFfES+CEVemqdeRQoGKyfPExZ/WRU/dhLmGhHk AvKDRSCG9XUMTgBrLQbuUulfTZkqohhGw7O7aQLdbbt2SheMZzS1fZqmwpO4JKWcpvfCbLXWUb1e 5AVTVhE7ee1knm5vqIuTweXE6skrkef2Tuanyubqdtj9YcXefiopVGfQXWH58Z08ELLr25OVtvb9 wqnqJxPY8BrO47spAbOr+lnUXVvMqTonua2CZw3GYKwALjM04ijCN8wYDEkIsJ77vt5CYdfmmEco E/PYTR6hY3PElGKPBmnpONHBk6imCb43yOZI03hpmKp6E2wuXzGpmBjazRqTvSaDTjOa2xFzOTyU u8uh1ZUVAlD1DXkZeXbmzAP6md0ZrS+yBmq5e+RBq0hbzWYTGsXh7WBQPuEyH7pcLDyiyw6db8L/ 1hVWmzp5+8a8pTcF8VVfyjMP0gW2QIufkyyOSYiv4YAEU/d/0Y7R2dFvIFCrLEBF8e1lb80WUCQZ 3soX4iuxAZLIYYkUR34tjawd8z9+/Cj8Dg1tgkWNkzgKHsm0M+sYjaOdMPbtyBavB0q3jEuw8n5L 3GPJe6nrztTu3PaskY4K8WqnQrXCIPOSRweqWPucHqlBzP4RZVnHhA3ptZZdGAs5n9tQMCGmPNPW 0kLn4NsvVNkpoS7zIdWhmSNIlbtLZovUkf2AedIS80tlv+oN7X5rPSQdN2Gb5NAvGXYpTaChAAZ+ tW7iBUH75LR+URZa/z0JR5CPSudA1pZH6l3xaq18hpyNeqNzMo6TXJwhpuMekVsTz8mcA2LyUWHk UXw/jpd6sA52JMDWBt/jy6jSMT80KohYU8bXSi+6R47wldUcUtlIKXFMNfLnbC1N8CP7PdV6SvnL 5y2lE5gSNrNl+a3VoItWNHUhbEO9fdyXy3cqhzZBizoLjBWuqsmVQIg9P+V8B3U4aYI27EnqJEtX tGhqIDR8vodKSz9vYoOWtKPYd1iguBirvleDbQXQpt2Kdkd3p6GTJCxacHN9KXCxYkBQm/VthfLG AuxBojycQ68BVqkVdfR78VIR/1ruxoGEwiCSFSVUj2+UiAfY1cLTjpzfkTl/4FGOTdKO+bcu466e FRX6jXlmVRMI/Njow2T75KTxYssLTiWE1A0n9AFPZGDRQalltZdlMqliiccGvrDq/1BLAwQUAAAA CAB4hQo9V8RK5qgNAABxNQAADQAAAHNjX2xpbmtfcGNpLmPtW/1y2zYS/1t6io07yUiOrC+7qc+K M5Ul2eGcJeskOWkmzXAoErIxoUiFpCz7MnmXe4d7hL7Y7QL8JvXhuL1eZ86T2iSw+8PuYrG7ANha DS6VTm8w7k16v0yKtRr+A4COvXhw+M2tB6VOGZr1+jGcwIXDmDW2dRcuPUPS0c+t5y1OarXValW9 IQoXKaq6Pa9BBdhc4+YJcGtm/5zoDEfqsjtm2gtmwPQBTsLm/m//cvjShb5teb/924LXc43eq3N8 t62fl9q0qmvem0ogxlvNMVaaww7G9syjBxg6tmd7DwvmgmYZMLbNpcdtC4XXpqhVb/hWGZerAX+f 647NTKZ7jm1xXfL0Yu/jB9dj8zF0F14VrttnIaevvs4Wt9yt+oKFekywERaOfeNoc8DHGRoBXF/G akimzODBXsKtdsfAskFbLEyua1OTgUZmmzPLgxX3bpNzUAGP4AM4CYUNyIuWRpN6NsFWhDKEr2sW OMzgrufw6dJjwD3qq9kOzG2Dzx58CA+WlsEchGfgMWfugj0TLxeDa7hgFnM0E4bLKQ4El3Iw0FBP anFvxVxKJOI5J5XDaTm3EVqjmWgBQ41wlDvmuPgOzWAUH7ICtiNhSppH8jtgL4izjEI/gKl5EXO+ KbXHWDKhqphCRIuYFg67Q2eurp3ayKwGOrwAvEXHFmBk0hU3TZgyWLpstjR9z0VyeK9M3l5dT6A9 +ADv26NRezD50BJC2tiL60OC8TmqgthoRkezvAcU1F8rvVHnLTK1z5RLZfIBrQbnymTQG4/h/GoE bRi2RxOlc33ZHsHwejS8GveqAGNGovlOs2FeZ8I5cOYM5qH+bmSAD2hmF2U0DWlth+mM36GEGugY P7b7jG9i07Zu5Jx4MZO2gM9wLXgVWDkcXRV9OeNNEiByqQooll6twI8NpNOszybOwxgRzvkM0c9N 23YqcGa7HpH225K93mw06geNw3oD4Hrc9tXrDbrxwFis7RdhH4YdBe1wh+JjaFuaYlBsrxWLP3BL N5cGg73bVfV2L/a+0NPvPNnwBaEOPD5nDrVHHa9dz+B29fZNssnk01Tbg1sTgS7bjOH2M/OS7Zqz 0GrcSjeT81o3ybalhT5tJNuQkZgRgdrRLmiQ6fIGxp1Lbn3GmUciNMgPBpshHXR7Z9cXKnYqg7+r aD7Sb4Z92Y6AYzhSBhNVdJfQ+eYaTmC1Wi2jY3DLm4VtqvqurbZHF2NVLbfyuOs+aYqTiJmJ3rdt wI2YiGFhxBQG6NhzzEtoB93UHE1mGVo2Gi74yGPcqnSUALSjqJ2rwblyoR6X7NnMZTjsnWYuWRlK 6COqbluzj7LjE5xCSfaV8wAarzII+6Ul6tx4pXqwX36Rg6cvlqpnqyZD5g3Qh8010IfN7dDIHEBT qiySl9Lch/ylfVdXHaYZ6qQcoS5sF0ezubGPgdT1hh6u2rDTZBbOYIBEVBJFRIksTPiGguwKmiOe Mee/l4i5UGvEzMMtsntMVBaElgueWrEe3xrhYyvNJaSIv2W5Q5LwNRpcSogp+b2DKZY5LX8hWDN+ s5RrAPPBDYYPzM/xeDDuqGLFv+sNVKUrK8j6faPxUzND0u29i5HU6/WjDMkoSdLIEHQu25gGA4Lz NqLUCyjpe82xMNydwMD2KCNT5zl1JtZoiKKM1eFkFElSD3A6WHeeYZ2KFDiROFFYkuQhjK/PQpUl AhQIDSHGy6krikusZSwDw4bSzbEXASSMUS8UAmbixb4crk57GBP8SPJ0tIU25Sb3OKNa0fXikqcR KPINlUFoYYGgELWzXCAnj+YZ6gdv0J6gjP7RggY+4wPZ1sjD7SsD9WIwCfURuH1u8flyDjdU4ORy tX9RL9tpLu1ecFFBaOkPwv41mavPrseYkeZzLHITXtjpd1XlatRrdwsQeke9mep/P1ImvTjBYYKg 3+unEV6lCSII7P0p0SsDbIiA/e2c/gTAWbHoeri4dFQKow5GCA/O2iN1hDOADR9fffrYpABc/Fos YA2D8QL3YBSHoFj4igC4kauI4H7WHvfUdrc7wiJRHQ/bnR4JezX68A0Z0CtriFrfzqNcSXoIeBqC pwL1qH3Tj+BpfgfP4XfwHEU8u/wInh+h+K0lShaobzS96jHX22r/o8fY0rd/4zFzJu1/f/yYcf73 7S9LLSiGKRVr1aXu+SXnGKeFqTglaHViRL27skTHuqslmmiy5nNuqxy3tfet4rc4K84vVeAqd75Q ssM9w2eVihoeBDkkEDmckqXGXSZ7nS8lTMZBh2mvcpsjMO6VUDLKFfvTpUt53PcnQVsMiPV4CpWZ V5QKx1RtBbVW2Vc1UeQFnanaLplwy63IvvAuyDdZsLAChZfQLK+BlAmaIBHMtzmCCbSwLMSwdUxr IpmyWxmiv4ki11cVq8xkBn+BXj2bRdKLITum5mLqxR1HBk3biPbmDRwjVoZruo2r8YrY0nxNfQ1f LOsnFKjVMkmfGxlUYzuqVCTDyTZwxmQpRLZ0Q3lyBJlthvOFSLMdHsWn3a9EpKy1WpxOj9NhraFe DoSZMoBGnNAvTLKzeMjiZH6dkbHR4SxBJesKjAxrlqVYw7a/HqMIs2/EynPNMJx0JU+Rh8p2f8GG nVgzqVONzrjoyeX/ZFGkmmqO6oi0corhkssO8Qs3lSV+Wm8Bf03RfXDdx+V0oVwNxtj08mVZUmHy oT/BEKimcfAGhXf8KoF/qpKoOGOCDiNrKaR9DXUfJfhBe+GGf8laWKseNGDONNzdWlg1z6nwF7Ud kRWiQUmbvEGllv6YpdiYp8J0ZXjxIpLkZYj0RvaiWIWYdvQTGQqH461YB25qPvvv34ryt8wNsa3+ 3q/Wc/ckNNMJPK8f//KrtUcnC+fXg84E7aqqlZDAX2c5EIFjRGCn9fvn9zTx8oEUkE/oFPIBZT99 bmwYrkLsFd+phC9F6garLdiblYjqIOIU5NHmrAKNcty1Ay9MezdtBTc79+7uLIT4E3z6L+XST/Po 0IT5LkmzCc/pLN6js6d26IPS9fYqSb+LeVvW0XAuUSD/tCDtbWlH+w6RUstig2xrVoLDvCUdS5DP JWN4bgiflvwDjIX2Zck2xvDA13MzQcAfky251NZKsPrjJGjuJoH5x0lwtHO42TgRwbD+5OZFq+Tg u4e5jeZ/9LjNncfdaPRHj3uUmelMxYIhLR7SqRW3RTKSyyWkWst5hUoj3JaEs18sFGS9JBtlRCce 2oJFcsqDHxxnYTteLBfhVLzC/yq5HjMVo5EYfrDYhtPMx1k9FucoH8fchhMGPVinl3Dlx8Jk1RKe +ViYrFbC0eIwG6LBnM3jIRH2Aw/zNOeGeeri9sFVabSNgSFxQ4Lx3q+rcW/mMLAt8wGOzijmg+dg rp3RmXBJjlp+9uwZVkHlYGNAp3YRl7hXTLPupM7qT1Fn9XR1IF8f8yn6bN12hATUqcpKamPpFq8S T6VMrY0FSajIiZDzBCjd55S/Ig7+vzz0dzyBSE8pEcMpzUgYQQXm2TZ5dOQJZ6QC0TPXlVXb2kIu Z4JDceIVQ6Kg2yrEqNe+DARY60iR2i8zTpVAJ2x0LboOEH51kgHL80f+GlL+eNBoZRwyqYaHpW87 cP698/6E7r3Prsd7MMZpz7RWYYLJNqYgr+S7dU6zTNiZZpG9W+n9Q60mP2ahycU/aGjbefC9vQIr Blja07U3aNDtK6DpOhpe3HtSPW/PsB0DmGVojuF3ViGxP6W7w133qOkqKhsPo7pxazh8SkAXw3xH PC/Ea7jotjCvQszXbcfU9VTdvidXPVm3HdNYJnEJb9nt0GxzjloXXfyveixtTidB7q8etcwsfM4L YriwiFI8CDphxGImPtdb/8UkFuWtMG/8VQ/lfo/ZWJtExClE+jhkTSkSC1jZQ4c/LG4mjlvygmc6 bj5C3e0ZOi9Q73ri0hlev6eI3xe2OF9aOuz7N6a5Ze1HOoXPHi5Eu5JKfvtqTbsp7msjYUao/A6y kI02iOJv73KDdX5zQo78sn7XnbncikN6h75lbx674Nw3ULHulXo97LTHk1KspwJixOR+U/yi675w 40nRmeuaqUoP93fUYYkR3azKDB6qm7otzear8AxeECLRutJLXNC6mCgIKPeSNqwfImPHr2TXj01k Txy5nthiY0fyEMbi3szKmeqcqYJHzlXmZjhKiHxTvSu/ZsdCgT7ZJAGr1WqeBRK3ZjJ94OBVeUnd SqyVvIvr8L46zI4Jd/GvlWMHHIGH5S2pNSse7RIIGqVZyFTngOW5zLVfazXc1dH3mfQ9ce9+AaOr PsXwMAvGP6nhnz7WP8EzTOXBaT1EKS1L28Agcrr+04tyjPlrYtMp8wm3/aUfLPq8yRv59uLWDe7F HHjO6eN7pXZFSuTEc472ieDIeKHFMayX4p5ZNsQuI2OADV+TZE6fMK6VW4VCIdL0G9D3tmS5rMY4 j9+tcr/X/xNUlh/dZL1R6h3TOqfUyQ0g0brCGIMj5xQvZAMFlykYtsXWhaqgLE+FwbWfoPjhRyxS XGD0EQ39PwashJ0VXC8V2JPxZy8R30LbKdbMTmmEDUEOrX4hjaguw6YAqBLrMpiriy4Zi0DEIomc oPMrV/pjz+IRsexTkWLS5KeZb25mVmVzJg79wv9IW+xP4qYhGUKq0ou0vtI0klmaeBN6ufgfUEsD BBQAAAAIANJ9GT2/pJXt7gMAAHUPAAAJAAAAcGNpX2lkcy5ojZbJcts4EIbvfgpUzc2VsQluonID QcRGhZsJUolPKC1UhjUuySU6Kc/bD7iJIETI1k3S9/ff3UA3eX97A25B+0kxBfhlXddfwKo87I4n sD7sQFD+qbYloEEtwJF9Kdd1Cf4ty1dQH09v5e5O/httNiehW4vfwZ/yVFfHAzjuwUt1+P1+/7qt eLWr7/6RFU8kKv6uX8ttta+2YFfuq0P1JmQ12JQvx8MvUB2AELai+5ub+9shsW2TcVm3uda/N8PX BvqrDVM2hXEfMcJxiBjjLE8y9EDA+DHeDaijY5L/SLLvU9qcBp/E5QwzqgQ3jCs8DeRcOh5e4ZP8 kWRT3jPmEupT56QRiC9S/sasIKAsDdEzXz0gJSNrtoKBv8zIms8oKsKcRiSgiKMioMkosEXJcwIS Jdkzz1AElIyc+RLYM8tJpGbUCLz5lBjJKAp5wXzVYWtYep5FfsFU3pkz8DMaiFN7TFiuGLizTe15 ypCakDt7LXq+md4L3j7zE0FAOwGn8TfOCn+4RvroM/105/uJkygqYopRTpN41AnBYl6QZgkmjCWZ kE4dNvZcf0Y+TX6QLMUjb84atDnIZzUI9vt2kXTLrtsgu3ap1HcAMLHVxH7776tYYO0y3Fen+u1L T4BD+f52d7FlViQORGI04CGjPEweKJYNxR6YVhSQFcVk4B0Le0sHjbxhQFNnEBB8WZFYHFBjIHhu QujYkxYbpqszwDTLCnZpYOkE1I+UhDqBrROgKJgTmKamBMHzEMWYyLxpqItgNMipGr81sLc6QZSI HZuESBE4C01GA8+jFEPDHZpq6Co48xkSpn1821N3n9SiNA3JZQXuRteihudi/njM0UN6PmXdtet5 a4B73t7oEmJFPNdSr9QYCJ4TdVVemYOGZzTykcw7H/DIzyZ3Yn3lTuD5S7d0NAaC567tTnlD/KIz yAgKc/JdNSi3GoOe5x60lme++aIz+ElDGv+8qKAstbcaZSsShooArrWHTGKWxPRJEZgLW1NCz3PC oLUw5EPTGHzLCGEYhZNTg0v9nKXYcyxLeWcTbyXaXRTnZFJw21TDczUGLc8907ahXLGlS2jgxegi xJ2ON22ou0UdT5jPl2N801nr5liK7yMOYR/f1o3ZwFsLKDyMLv5CPzZTHg48/BxvDrx2r0g8GvOB n4mP/KE/gtc9zabxzTOvW7xT3jrzysNMFVD8uOyy7w5g+UEBLQ9l/voFanlT5q8X0PKWzFsf87bM 60ZY4h2ZX37MuxJvfaI/Czn+9YFseU/mvesH9mQ5PMKPo2CrfxBk6Hm6IZqRX+x0I9nwHLHpG51Y QdpnN41XKKTB1GAvPjf/A1BLAwQUAAAACAAIiRk9u2wMKFACAAAsBAAAFAAAAHBjaWVfbXNpeF90 ZW1wbGF0ZS5ojZNBc9owEIXv/Io3ySXJpDjNsWQy41CTeAqEgmnCiRH2GmsqJI8kQ/zvuwISLk2n Hh9s7e6n93al6KqDK9S5pKWnTa2Ep2VBW5lTt+JICPZN3Vq5rjwu8kvc3ny9QVxoofGjEopI4U78 Pn4Jj9yKFrnZ3IfSvzz9EE91foRnlXSorVlbsQF/lpYIzpR+Jyz10JoGOW9lqZDOW7lqPEF6CF1E xmJjClm2gcNrjS7IwlcET3bjYMr9z+N4jkfSZIXCpFkpmWPI9rQjCN46rLiKCqz2nFAxCBpmRw0Y GAYLL43ugSTHLbZkHf/j9n2PI/AaxgbIBfeBlVuYOtRdstwWobUfpd1P7J9cFpB6z65MzY4qRrLH nVQKK0LjqGzUdUBwMl7S7Ol5niEeL/AST6fxOFv0ONlXhqO0pQNK8oAlk9mXFdq3LD8QRsm0/8Ql 8UM6TLMFm8AgzcbJbIbB8xQxJvE0S/vzYTzFZD6dPM+SLjCjIIsC4B8tLvdT4jYW5IVU7t34ggfr WJ0qUIkt8YBzklvWJvjw1O3/DE8ZvQ6oYJOTT43sQZbQxl/Dsca7yvv6WxTtdrvuWjddY9eROjBc dB/0RJ1OdBwFv+QcaS+FUi3PDfQmuGsHIwKHq3EYR+MlHx1ymPRTCicSo1n65XXPO5cln8YSP5PR fMnxZMmx12WWjCbDOEuWT51zDktN/8hgiM5VUxDO9vezNtZ3q7POR+mp6nvyK+0nOCZunHw7XmLO 3hpZ4LQutfQXXPnQOFytGnfZYyBpvkfgJnyqJrj6A1BLAwQUAAAACAAIiRk9t1bf1MEMAAABMwAA FAAAAHBjaWVfbXNpeF90ZW1wbGF0ZS5j7Vptc9pIEv58/Io5b8UFDgbszWX3QuwUBuFVnTGUBE5c WZdKoAGmVkhEEsa+lP/7dY9G7yOMc5tcXd25dh2Y6e7pl2d6usfTPKqQI7KeMWoEdLW2zYAaFr1n M9qYwQxOdt31o8cWy4BUZzVy2jppkY7lmA75x9K0KbXJe/MP8ckMyMwzH8nMXZ0jq+Sni/OqEwkf L5lP1p678MwVgY9zj1Liu/Nga3q0TR7dDZnBUh61mB94bLoJKGEBMR2r6Xpk5Vps/ohyYGzjWNQj wZKSgHorn7hz/uXyekIuqUM90yajzdRmM3IF5jk+JSYsjSP+klpkyuUgRx910IUOpO+CYDNgrtMm lMG8R+6p58N3chqtIQTWieuhkCr4ATT3iLtGvhqo+0jQtTFro8T8xEqLMIfLXrprsGgJIsHGLbNt MqVk49P5xq6jCCAmH9Xxb8PJmHSub8nHjqZ1rse3bSAOli7M0nsaimIQYAaSwS7PdIJHUB8lDBSt +xuwdC7UK3V8C0aQvjq+VnSd9Ica6ZBRRxur3clVRyOjiTYa6kqDEJ2iWhQF7HDxnEcJ3GjRwGS2 Hxl+C4H1QTvbIkvznkKAZ5Tdg24mgGf9uE/wbNdZoCg0E4gTR7YJmxPHDerEBx3fL4Ng/a7Z3G63 jYWzabjeommHMvzmOerTrFSaIhTwH/V96gTMtO1HiBuhDyZ4LTTEJOHWCMOxCRhAh/pk1FUpIpIM dPX4E5f3E3Nm9sai5GC5bSwPUt/Xs/x3ZjDLzw6ufPZQIKM4khsykDLeuZwATOnR6WYBfgQqUOYn i86ZQ0lPuZhcGqCsYoCin4yxMhhddcYKypwDTTlBJGGkqddjg5NVwR8rE1zcaDRq4HjmBPN4zDBu OkZHu9QNo9aWcbcEaY4Tiant02cX3CkTZDiQFiopw/vqtQJUN2pXMUbacKQAnhW9avgBeA30nbnO vJYwJC64UXuSHAbzxo1y3RtqhtozulrnVsLaK2UVighWo6OrXQm/rstEtB5OWi2Zooqmq8PrIrWE Fn4Z+mQ0GmpjpWf0rzqXutAMZ/h34+2bC3VcwnutwXLdMaSG6Ock+lDCMez3dWWc1ewXmRm6Dh4v kLcefpUSS6MjF7svpfJpJNX27zLijqJJiTFEuA+BklxMdDwOV5Af/PRu7A56hjrUlE7vLwlj6zQ3 /1FTx0qa4OcMwUAZ5CW8zRMkImD2l8xsd3jdVy9jCTDfkcxnBFxUKrhpIA/DnvHx0PXZwoG8jdk4 /HXR0QyNLuCU8z+/vft8ekfOSOVrBQU0m8SGs6hOgkc41PjQV5B62op+NrZd50i86OiK0en1NDiF DH3Ugf0Ctgy12yfgBjmwSCvPz5kFkiX86hAnE/6TiL9OWsnoqXT0Z+nom8woEaN/I5WndqWCJmJW hQN9MwvSiUG50THvGHpAQrfAXC88WOB8afMhyIpktWKuwaCqeWhXniQCYBFAWRSPe5dZpJIcCsz7 UsWxmvB9Opce/O688t9BuLZI9rtzgDm7P7nujiGH8KQtovWbuyWWy8swz2R46jqoGvW8zTr4UKk8 VfC0K9PANbYeC2g1se/IqpMNCPj51IAizrK81Nd7E8KHdgNEIqXjSY8ujKmJJRZ+8tk/aeKnqekZ HgfcGUSChRORBaof1gZYTcxcD+oM0Bvn4LSosrMWFAvvETDXk4GhKZdgvw5Dr1/XBDwr0d6KNAA0 W8fnYJsnMM7uGmgJRCMihfKjGpO/Jy0hK/qBnRMwZwPlbfPo+ISsqOn4WK+QlbkGzPACIr0qWitb NfFCuGg1tegZ926NHB4mqryOhZ2Hsym9vmY0TDwK67J2ds6j5h/J0FMl+V2Gs0iBd+RV69dPBbjV YwIBPImICEuJsLPWw6sHxEr4AQ0KPwGOwg9gxdkra8dydWSvCxxy+CWG19oJhnou+YiLA9wjrEfA zMIdXGPtRvv++OYagftbO4D+f5D/WJDvQCeGnrzCNjKgfgM65AiOIQoP6lkIpoCXxlwKcRoI/Jb1 cvDfsXAB8ZUwGMHGc3CyncI75nZJap/yI+YIGl3zy4buzO0R5CXnQ8SdUuuktsfq2++1+uk+q9vf a/U3mdXLM81O50dLinAW01R24ZM919zp8heuebrnmjsd/cI13xQim65YVnQlwsJnjvYI7ctLlpgA J40wq5ZmeJ4AUgfKWahSuzQVxSa848q+I5gLJGcg950Q8+efEv/Lh0TIJkJb0DYRmPbVrkBiQ0Uu 0Bakp74fpvfSjC8JdqxOOs8UMv9OJaBLvIoUKAVVYvbrDMAyklEuwAybX46xd7L+I4NL9p7kcHl8 0pYBM2tBAMdjJ9oDB/3BGO+2oB8/IDqEvjDaIGNo2VK2sboc3pLhsGArDGMPKJz7lC8m4+qSXzxi jOEf8LfrPQr418mWkjX18F6LmKQ3UIk5m4H/IVX4AZ7/7hzGIZc5lulZYrKxO6diRHle3JnhynKa KEnre/Zj35bCxCWrY66wY/B/D3Bk7sBnGbbB6UjJP3C6CD2ZrSuq6B+Z6ZLk9l+YxL5fbHZkGl7T 5ivnkrPrOEFgJpF9t12Fcv8T9XjSDfC/MSTl+f7VepzFp7ldH5jeggbGevno87S9Z/GalEkvrdxj zu0P0GRnFR9z2j9AE3lFn688k5r+WW1Kq944ve9T3ssU2DMwL1VAXuvLFNgzHi9VIBuC7mjCD+EB zwz9jTMjR+JGWxJSY+58xlvs0phP67LRrXTU5rfDiR64s59VAw3apQWHTn45Hk7ZYEYFya0tJMrM LRa9DyuCMC8ZzmbFpU43frw5MmcKEQqGBGFthPy8HIrjtmCQbj1Yb+16gdg7Aq1v4f+6pFsGE0GZ +KJit4RTmYTtSyS8kUmwd0ngwOMFsNwKEaf9BeSNEDHdX0DeBh7/SMDuTngfGBTjnkfGMzgo/lkD VgKU94bGZNTt6ONqkYKrn3bCbL0xYkdgpmAz0zbCY1/EM67Qk7+qFB2Q2B/ue/y7MFtkb3QlFhd+ MtU0VA/PXBnwZWfmGl/eUM/LLm7VEymcGRnbWb65vT9T0bznVjBpVrpxP/WYtaAvZixjKI0CUNKg GvpeYOMLRj3BThyXPGTCiTr5wqHyJcFKSrCVKGuEJomJFHk+NglnqdYQjQxkUmEWb6ukUhAUqa3q sGDupOUI9j9h40SSkpbNmyV/T5A16wq+bflEhCqoW6PRKO3bUb6ACphp3FPHciFBWdWwDWiEc/Xs O4tULFK8wl+7eHsxb4bkMzZ4mnKj4rsIQ+3h0Vl4L1HGp08u9Ft9rAySdx53QHiWexpySFoP8/kL hLw+QTWqWSnn5+TX2t6yipqIlyL7CyhoISRkFYkEJQkTmDKZFkuFMMdKqxVp+cPHo42X68lJ4Z6H HJ+IxvwrNELHJ/wZFv6VS3lYE204wA4v3TKn3x6wu8+tO/JX6P1rSV+cPbCKHOiZs/IXBLn7y6+F w6DZZK44IAtzhY2lCUcyfDVheuQVwyeJanOIZkn6XlZrF4TifokDAs1jNZ0yaha/xSp4pfSNhTrM lQtQA+QWfSL4RCvvydB2iPK/afxAGfzHjA8fmORRK/GA9Mq3uOXAYEXTJqOxMVKvccediE3lzeBL nOWrhyFrveyZFS8jW1G2xnumWXjDlL6n4l2QN4suG6OrA/GAQ+U5m5n4YJHvH7SRwLFmTpkNE9T/ EN1HcuXQsfHJx5Us6pg8lQo9it/HtyPFgGQ3GoLx4MnjcyxKv0H1TGmT9dJh5qQtOabTLFGlLAwL 65Iyzydvur5B6wLIVVgF/O/Qsjc2Qk6rvBCgDz+iEMgWbbhmzntcl40j82vcgSeQyQkA23I1EpeD F+8bnxxBf1BLXohFjxmWAN0tjV4JTyn4EXMFzn1ISgU4a+BQ8fFpM62CoDocGXVyoHevmPPHQabA ChvrmwH3QI/6M4/x99nkfsUfghqxenGX3cDrS/jGH9se49RxWJEchH1AQ7znhgIF97cYXIF1q83K 2HPScO00wZxR2/LxgBaq9nHg810afjAz7oyV8LFu+J60mgNyFOpavcClj7VJd4wMDYy27S5kjHW8 58+4JiYWb9rSg5JlIA/AVjKuVIBgTWyT1MVDjGjVmbsZaMDXOABYikdRyOgIJtdTJBbEEwOVFKvR W23xOPogTS3u4PEfdy7ZH7U0MS/Wo9yRtA9pEnAThvCwgCRBxHzYEmtsfFKBFmVu+Pe1tPh0zxQJ wERyVugPxCxut8xsmDTSGq49d81BNYIP1AseOaCikMWxE2+l8ZF0Nn71YgALzVp4EGdeHcJ+/rik +P4fD6GIJNzgYYA+CLpUE4YKJ+IOs9BI9QnP0cHWj/oHTDdFVYHmX1BLAwQUAAAACACgdBk9J8nE BA4MAAA+JQAADQAAAE1ha2VmaWxlLm9ianOtWklz27gSPj/9CtbEh2gmymixHTk1eTW0xNiqaOGI spN5FxYXSMSYJBiClOz8+tcNcJMEei7xQWwQHxrobqAX0G/e/JS/zhvtr5X7D/Gyzncmnj149F60 T9p3GmfvmfadZymNd0j51BNvQsrFcxsyRxAuY+F7dsLhN2DxD2cxAPDRC8kzScsGMCVOVLcTJ+XY UvAgacqwp/NzJEaR3ZB5T8UUlGse84mWc+Jr7ovmsiyASaNc4y88I5EGZOhklMWaE/uip0ejXafT ZPJJ8/KMhhzE8RwvIL2yJeCREwK2bLEEmUErYn4eEpCsyQkkjl0fesVLeDqU4S9Bboz7JCn5eCze 0t3R8Iu3k9Xy8+zONlfW7FsXmSWM0+ceMAF8lKC5VPj5bPnwzdZnKzEmpHEuxqDWJTomoAtfri91 Dignw9/v8rH3Kf5GPq7YCxnDVfoR7hqXeQGufZ+gBvb7bWMNTa7IaVhwHPZSsvVYLjegeOGFOWDT qs1jJ+EBU/OC3QQqJ8ICpTaffOLmu1N4pYGvs+Vo2C3E6x1oPBq2YmvtIlZouBU7eVjPBdTL07DW Z2Vs7eKt4/sJyEufpc1/fwfvjmXqdjudmNRHE2hghW8qqUGBOckJqj8g3hPPIyBxF54CYU4OvM85 QIefR8nxa4XMmdOGEbuoxPTELmpB1tpGZKltFfLWmlY4l/stKGs119czq0JyFjop5S1ovZYEdnnb Gq35bG0KHA9p2ibz49QQmL1PCkRl2qZloUPYtalxtOoWTvS+TZGf5EEXGGAusefntns6mQCK6Y7Y iwl/mg8NqYurs8GtRODOnI/axkl3JNNoDE6KwE+c4UHMNLY9c6XvtU1AkAmLdxoc6wiQ4HbBF2dM 43kCgnCu/QqH+FewkmDLE+LRLfWkt6YxOFpOgDdwOeX+TqPvyXvN0TgErpBofxmLBw2iD/hox4U2 +I089MU8DNYHswCTifnAhYePwIFTUNn7TkfKVh278liKdleeVmmaI6CwRrUTuorOoyB3BFBattXw 57whqvpw8IQfYDEcA0HlKccQqjn8Ja7iUBlXVbPX55Px6niqgLVbAGDpCc9WlXm7XgrenEa4nIDx rAyQZ1CafgcIZWgZJOIkz9rm3swWxlpGOWTdJsxC//ZhNOgLYOQ8I90m9mL84UoCDxGSLbjN1/nN cDiSyOwQikYb1poM+/0rCeUe0i3I+WI8GkplhhGQz20cF+agZBglg1Z+1saYC7doz5bmw0a6soyE wjXar2nWsqb9UX8kR3Af6deQwwZy2IbUp9aH8eW1QDo+R7rdXoPB4FtpL6DbNDG1BqPRWAB9jqRq Qw1xx/PIzXn5tAlJUhYpwLLjZvT8rOjkHqc9n3KR4/hqBgKzIzFJqZgWm2LmM2TOXYzQ3O0FeUld vL1fWRv7wbrtln3UL6iIl9TB8ZRzYx8k0dQJC6BIEwR1toSGAWeF8Wg7wi6CMIB6IgYrcQXGVyzN xYW4WQ+PvqT2gUclFQ49zCiQ5H5BVJ21/G6m5IvQnsdTVWe+3ZKU+PaWCjcY0V3qlLl3Sfcyr8qo ZV6kMpZMuAMHnSh39mSP2dUbmSGfYiMeMSgmAJDwoYqXCBni0YPdlpA0o0Q1qQw2jVV3lHq/Xc91 UxrRdfKozVlWjroWHUPikS7yGP1348UWjdlxch/qhzIKihYm1qykDs6+JCP6TLBYbIxpbJB5sUPC Aq5CrSyriCf8FdRktTb0h2lRsHgM4l47WJ9buvQ7IXdewU2t1cNyWrgTqD78V7CfFyuJ3EIZ9wrO KI4F4a+hTLm+5LXVQUz+qj/KbBNCMij9VZlRORAiwfdLzy8wdpLZoqxvHwLTnIyByc4GiQ0KS4Cj m+UpUaZAdUIqxomEtMEBs9HTQU/kJYK6rtXDVBsIz6Ef2j8YekLt2ZYDW8ZBDWYZVlmFceVh24vE yPGQsz/y2zA9EkP+CWll7wckwFjQNt8F5DkTvub1oRndBSpvRtm+RYLH5cTezKUIyCoTpS1STp4F vT2wTl+StnCOoy3dmlfDxSDu8LBNYavJSi9OlsecFtRstbmHUyj3t/CQWYD5p0KymGV0+wJLJrDU zBZNepqw1Z62zOY6ovKq71iKspZu8SexqYflqqBkMoMkyzOZ2DTH1kWcBgnAVjgqLgK0bJUVMWSq NTOk27hhH8/dtIAVC9ZyGcAYy/CZbbO2LVkXl81zIiYR56QxnajaNHCr4N5hlVCh+Cndk7SsX5T8 vxlLwR2G2cUwPCfQgrBT3Rz928i6hIDW1i05yBQIyRiznJ9bVELoTDsd/C3t3mygJ433xQUk1oww /XH3K7XGMVB9Y/dzhQkOnU5wqASpSDyH6GlC5vhipx/1UMgIMJhKoldZ4Qi1PdjeFi/YEpErwa9d ZFdAkgZtuyn1d6TsyBN591q2I6jZbCifE6idT+cQAJ8d4nLIUe/BybzAZ7uT13Q8HPcHroPHcTgY XF3W/bXrsHR7sSiiN+WOHUUikJ3ijMlERk/PU/Qu9SJaxw5u77N+8/Nct+7tyedZfyDj6zZ0eAB6 o/3B6/jhKR4zubMBi8vx5upGZlWX4+zqRiWCVcrAlUIYC3PzN/iDlQy5YIjsxeahuNk8A1vGeqYX QbDI9M+l0Nf6fG7Mu82bUJUNxsOry25hMaWVzIllfpFsPJ48KRFfbqcF4slV2QBqGfvhfiLzU8zj c5Hbn+E+T6WWtr5KSfqkyHAdL6EiTifUTih9Vi1bNxdySZFdln1OEimA04UMcn7kiIOvmeZEA/9I PcLPwCvTWJozuUhI2uOEqhZqrg0TtCIXC149scXxBN4Lx9M4VBDEb51ioU+KIxE5nvI8TB6mRVzO fUcl+vS2KLBdRS/wt5ePa31RzmHH+9RRaQaR09tSP4j0XamlN9oq9L+yNPQ1kx1ICmKdjb43dMgM Vl/tUl8BAU+dsoOtVtrdWp98mRuV3qD68J5CUqluSQ7/MuPDErLWcjhUMXEx9szixqSC+cSrZhCM wfjGVb8P4dtJ/XPrwObA7tq6iUewLXkgk8ms8omnNpYO04fs+3rU7xOs/TtHfehqS5dc3H0JOmXC EzShEZdVGm/MC6X+gaVPEMsaS5epFxn2+/0TFuKeY3D2OvHkl4Vzv7OY3Awmg4F0ozzybgYetBQa nuvLm8FA3smEToy0ynMbuCgMA916jT0IBFKi2dQ4d1dTQ1R90mP55Hcs+lRuDXB/TY3HCifrbSWu NCXC1BsGUeUqESWXqOQ1K+prwYyKi1ClDIvpdXENJqSI/GtxE6bC1g4BoaVTAAVZE2sG6n2BVKlp v5DTq5E3vrlSLdKwzCIOJaebD4421C5hQlJxR/bCpdcEYYuboyZYXFxAziq/yWqPd/rZVPCu0u1+ 5/TUukVUqVtEqXVboCBdaAJ7kcptPS6+6msDh0hsdAB/a8MQ1Z5eTy77xRVu6iGp8hfmeHQzknb1 E6RVZp1ag+Hw6u/yHhLoF5VPnZnW0tgUdzAJL44aGBOvGjriwqHKFOtWfQZvB3LbcHeAG0YBMazB 6EO/sDKSapg+uflQRNObDy2Q6Xx2W95vJJiq4ncFVw2+e5CV6U7sGvjF70KBuIYsGngtlKoHT6zL 4WhQRDWOtLBWMV05/UfxEQdSs7l+ZyG0N719mM2n9t+w2/uf+uceq7rKgWKqVm33fB/UX1qKfPsm 6RXfihtvQuaJbOunlgdQSWF4W22Mj1oWEA3aDgeFuSGUd/I7F9dYHL6AhyQ+ZA5blmpibTsa7zod yUJsmIJs3HmZ97o9Lb6JOmESOHh3DRKcIdeLGpdGLagJfkcoYV4KXWrcvWnW0wZJ0jbrbDS+rnAU Gm04/fqyxjnXly24xfX4S4WLrsdPbbjZZL26nev/M2o09VLmhs4P0jrGtBropE12yBAqGGQHLShr dNOvYBwabbj7WnAetMltQbpfz8oh3y/n7exPz81vny7e3pnr1eei3W3BWNP5KaLxnyv8vU+2nfpz n7qz+Gio6OycvvkIR9QCIUzIGru/H/UGz53/XLyFkxdq33NKMvwnlgiKvXc8OBoUPGeMhVov0P7Q Lv7Q/qtd/PnuF027M5Ya/ofFRl/fGRvQ0rp78ecvINT/AVBLAQIUABQAAAAIAFWJGT3T/ahh2Q0A ADMoAAAIAAAAAAAAAAEAICAAAAAAAABwY19xMzUuY1BLAQIUABQAAAAIAHiFCj1XxErmqA0AAHE1 AAANAAAAAAAAAAEAICAAAP8NAABzY19saW5rX3BjaS5jUEsBAhQAFAAAAAgA0n0ZPb+kle3uAwAA dQ8AAAkAAAAAAAAAAQAgIAAA0hsAAHBjaV9pZHMuaFBLAQIUABQAAAAIAAiJGT27bAwoUAIAACwE AAAUAAAAAAAAAAEAICAAAOcfAABwY2llX21zaXhfdGVtcGxhdGUuaFBLAQIUABQAAAAIAAiJGT23 Vt/UwQwAAAEzAAAUAAAAAAAAAAEAICAAAGkiAABwY2llX21zaXhfdGVtcGxhdGUuY1BLAQIUABQA AAAIAKB0GT0nycQEDgwAAD4lAAANAAAAAAAAAAEAICAAAFwvAABNYWtlZmlsZS5vYmpzUEsFBgAA AAAGAAYAZwEAAJU7AAAAAA== -------------54b14120bc72690a9f3015af90802d06-- From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from [140.186.70.92] (port=44217 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OoYpZ-0005T0-5K for qemu-devel@nongnu.org; Thu, 26 Aug 2010 05:33:13 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OoYpU-0006Nv-Pi for qemu-devel@nongnu.org; Thu, 26 Aug 2010 05:33:09 -0400 Received: from mail.valinux.co.jp ([210.128.90.3]:49334) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OoYpU-0006NY-74 for qemu-devel@nongnu.org; Thu, 26 Aug 2010 05:33:04 -0400 Date: Thu, 26 Aug 2010 18:43:56 +0900 From: Isaku Yamahata Subject: Re: [Qemu-devel] Template for developing a Qemu device with PCIe?and MSI-X Message-ID: <20100826094356.GF16489@valinux.co.jp> References: <20100825223950.7e664cb4@shadowfax.no-ip.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-2022-jp Content-Disposition: inline In-Reply-To: <20100825223950.7e664cb4@shadowfax.no-ip.com> List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Adnan Khaleel Cc: qemu-devel@nongnu.org On Wed, Aug 25, 2010 at 05:39:50PM -0500, Adnan Khaleel wrote: > Hi Isaku, > > I've made some progress in coding the device template but its no where near > complete. > > I've created some files and am attaching it to this note. Based on what I could > gather from the pcie source files I've made a stab at creating a simple model. > I've also attached a file for a simple pci device that works under regular > Qemu. I would like to duplicate its functionality in your pcie environment for > starters. > > Could you please take a look at the files I've created and tell me if I've > understood your pcie model correctly. Any help will be truly appreciated. pcie_msix_write_config() should call pci_default_write_config() unless you did it so intentionally. You also want to catch up pci api clean up. pci_{set, get}_{byte, word, long, quad}(), pci_config_set_vendor() ... > > Adnan > > The five files I've modified from your git repository are as follows > > hw/pci_ids.h // Added vendor id defines > hw/pc_q35.c // Device instantiation > hw/pcie_msix_template.h // Device header file > hw/pcie_msix_template.c // Device file > Makefile.objs // Added pcie_msix_template.o to list of > objects being built > > Everything should compile without any warnings or errors. > > The last file: > sc_link_pci.c > Is the original PCI device that I'm trying to convert into being PCIe and MSI-X > and is included merely for reference to help you understand what I'd like to > achieve in your environment. > > > > ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ > From: Isaku Yamahata [mailto:yamahata@valinux.co.jp] > To: Adnan Khaleel [mailto:adnan@khaleel.us] > Cc: qemu-devel@nongnu.org > Sent: Wed, 18 Aug 2010 22:19:04 -0500 > Subject: Re: [Qemu-devel] Template for developing a Qemu device with PCIe > and MSI-X > > On Wed, Aug 18, 2010 at 02:10:10PM -0500, Adnan Khaleel wrote: > > Hello Qemu developers, > > > > I'm interested in developing a device model that plugs into Qemu that is > based > > on a PCIe interface and uses MSI-X. My goal is to ultimately attach a GPU > > simulator to this PCIe interface and use the entire platfom (Qemu + GPU > > simulator) for studying cpu, gpu interactions. > > > > I'm not terribly familiar with the Qemu device model and I'm looking for > some > > assistance, perhaps a starting template for pcie and msi-x that would > offer the > > basic functionality that I could then build upon. > > > > I have looked at the various devices that already modelled that are > included > > with Qemu (v0.12.5 at least) and I've noticed several a few pci devices, > eg; > > ne2k and cirrus-pci etc, however only one device truly seems to utilize > both > > the technologies that I'm interested in and that is the virtio-pci.c > > > > I'm not sure what virtio-pci does so I'm not sure if that is a suitable > > starting point for me. > > > > Any help, suggestions etc would be extremely helpful and much > appreciated. > > Qemu doesn't support pcie at the moment. > Only partial patches have been merged, still more patches have to > be merged for pcie to fully work. The following repo is available. > > git clone http://people.valinux.co.jp/~yamahata/qemu/q35/qemu > git clone http://people.valinux.co.jp/~yamahata/qemu/q35/seabios > git clone http://people.valinux.co.jp/~yamahata/qemu/q35/vgabios > > Note: patched seabios and vgabios are needed, you have to pass ACPI DSDT > for q35. > example: > qemu-system-x86_64 -M pc_q35 -acpitable load_header,data=roms/seabios/src/ > q35-acpi-dsdt.aml > > This repo is for those who want to try/develop pcie support, > not for upstream merge. So they include patches unsuitable for upstream. > The repo includes pcie port switch emulator which utilize pcie and > MSI(not MSI-X). > > The difference between PCI device and PCIe device is configuration > space size. > By setting PCIDeviceInfo::is_express = 1, you'll get 4K configuration > space. Helper functions for pcie are found in qemu/hw/pcie.c > For msi-x, see qemu/hw/msix.c. > > Thanks, > -- > yamahata > -- yamahata From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from [140.186.70.92] (port=58840 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1Ooh1H-0003N0-Bo for qemu-devel@nongnu.org; Thu, 26 Aug 2010 14:17:52 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1Ooh1B-0003xC-VT for qemu-devel@nongnu.org; Thu, 26 Aug 2010 14:17:47 -0400 Received: from smtp.mailix.net ([66.11.225.183]:28959) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Ooh1B-0003wj-L4 for qemu-devel@nongnu.org; Thu, 26 Aug 2010 14:17:41 -0400 From: "Adnan Khaleel" Subject: =?iso-8859-1?Q?Re=3A_=5BQemu-devel=5D_Template_for_developing_a_Qe?= =?iso-8859-1?Q?mu_device_with=09PCIe=3Fand_MSI-X?= In-Reply-To: 20100826094356.GF16489@valinux.co.jp Message-ID: <20100826181738.9bdafe6d@shadowfax.no-ip.com> Date: Thu, 26 Aug 2010 13:17:38 -0500 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="-----------f6fee1686365d4888b07072390f9c0e0" Reply-To: adnan@khaleel.us List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Isaku Yamahata Cc: qemu-devel@nongnu.org This is a multi-part message in MIME format. -------------f6fee1686365d4888b07072390f9c0e0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi there. I should have sent a lot of these with my note yesterday but I= was in a hurry to get the files to you first.=20 See my comments below and thanks again. AK =20 pcie=5Fmsix=5Fwrite=5Fconfig() should call pci=5Fdefault=5Fwrite=5Fcon= fig() unless you did it so intentionally.I've made this change. Thanks for t= he pointer. =20 You also want to catch up pci api clean up. pci=5F{set, get}=5F{byte, word, long, quad}(), pci=5Fconfig=5Fset=5Fvendor() ...Are you referring to the setting up o= f the config registers where we pass on the vendor id and device id etc= =3F Would you elaborate a little more. Also, I've got a bunch of questions but let me state my assumptions firs= t so that you have a better idea of what I'm referring to. - The device template is a pcie endpoint - I want to be able to setup 64bit bar addresses, with large apertures - For now, I'd like to be able to do basic MMIO and regular IO reads and= writes.=20 PCIe questions. 1. What does the topology of the bridge with respect to the root look li= ke=3F Is it Root <---> PCIe Bridge 2. If so, where is the slot where I can insert the PCIe device=3F Is it = off the Bridge or would it be better for it to be off the root=3F Root <---> PCIe Bridge <---> PCIe/MSI-X device Or Root <---> PCIe Bridge <---> PCIe/MSI-X Device And hence my confusion about how to do the following: static void pcie=5Fmsix=5Fregister(void) { =20 pci=5Fbridge=5Fqdev=5Fregister(&pcie=5Fmsix=5Finfo); // Is this w= hat I should be doing=3F OR pci=5Fqdev=5Fregister(&pcie=5Fmsix=5Finfo); // Or this } 3. I wasn't sure how to register the device how to do the initializing. = Please see the following section of code: void pcie=5Fmsix=5Finit(PCIBus *bus) { // Is this how we should be doing this=3F pci=5Fcreate=5Fsimple(bus, -1, "pcie=5Fmsix=5Fdevice"); OR pci=5Fbridge=5Fcreate(...); } Or if should I use pci=5Fbridge=5Fcreate(...) in place of the pci=5Fcrea= te=5Fsimple(...) Also, this confusion led me to being unsure what the following device st= ruct should look like typedef struct PCIE=5FMSIX=5FDEVState=5FSt { PCIDevice dev; int mmio=5Findex; } PCIE=5FMSIX=5FDEVState; For the simple device function that I've described above, what is the pu= rpose of this struct=3F What other data should be captured=3F Which include the initializing of the following static structs. Btw, can= you tell me what VMStateDescrption is used for by Qemu=3F Also, what sh= ould the "fields" member contain=3F I couldn't quite make out. static const VMStateDescription vmstate=5Fpcie=5Fmsix =3D { .name =3D "pcie-msix-device", .version=5Fid =3D 1, .minimum=5Fversion=5Fid =3D 1, .minimum=5Fversion=5Fid=5Fold =3D 1, .fields =3D (VMStateField[]) { VMSTATE=5FPCIE=5FDEVICE(dev, PCIE=5FMSIX=5FDEVState), VMSTATE=5FSTRUCT(dev.aer=5Flog, PCIE=5FMSIX=5FDEVState, 0, vmsta= te=5Fpcie=5Faer=5Flog, struct pcie=5Faer=5Flog), VMSTATE=5FEND=5FOF=5FLIST() } }; 4. What is the qdev.props field used for=3F static PCIDeviceInfo pcie=5Fmsix=5Finfo =3D { .qdev.name =3D PCIE=5FMSIX=5FDEVICE, .qdev.desc =3D "PCIE MSIX device template", .qdev.size =3D sizeof(PCIE=5FMSIX=5FDEVState), .qdev.reset =3D pcie=5Fmsix=5Freset, .qdev.vmsd =3D &vmstate=5Fpcie=5Fmsix, .is=5Fexpress =3D 1, .config=5Fwrite =3D pcie=5Fmsix=5Fwrite=5Fconfig, .init =3D pcie=5Fmsix=5Finitfn, .exit =3D pcie=5Fmsix=5Fexitfn, .qdev.props =3D (Property[]) { =20 DEFINE=5FPROP=5FEND=5FOF=5FLIST(), } }; 5. Device instantiation I init the device in pc=5Fq35=5Fbridge=5Finit() in pc=5Fq35.c pcie=5Fmsix=5Finit(root=5Fport=5Fbus); I know I'm doing this incorrectly since I'm not specifying several thing= s. Again, is this the correct place to init the device=3F MSI/MSIX questions 1. How is an interrupt notification passed on to Qemu=3F In the regular = case I'd use qemu=5Fset=5Firq(..) to do so but what is the correct way o= f doing it in the MSIX paradigm=3F For example in the case of a DMA tran= sfer. -------------f6fee1686365d4888b07072390f9c0e0 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi there. I should have sent a lot of these with my note yesterday but I= was in a hurry to get  the files to you first.
See my comments= below and thanks again.

AK

pcie=5Fmsix=5Fwrite=5Fconfig() should call pci=5Fdefault=5Fwrite=5Fconfi= g()
unless you did it so intentionally.
I've made this change. T= hanks for the pointer.
<= span>

You also want to catch up pci api clean up.
pci=5F{set, get}=5F{byte, word, long, quad}(),
pci=5Fconfig=5Fset=5Fvendor() ...
Are you referring to the s= etting up of the config registers where we pass on the vendor id and dev= ice id etc=3F Would you elaborate a little more.

Also, I've got a= bunch of questions but let me state my assumptions first so that you ha= ve a better idea of what I'm referring to.
- The device template is a= pcie endpoint
- I want to be able to setup 64bit bar addresses, with= large apertures
- For now, I'd like to be able to do basic MMIO and = regular IO reads and writes.


PCIe questions.
1. What does the topology of the bridge with respect to= the root look like=3F Is it

Root <---> PCIe Bridge

2. If so, where is the slot = where I can insert the PCIe device=3F Is it off the Bridge or would it b= e better for it to be off the root=3F

Root <---> PCIe Bridge <---> PCIe/MSI-X de= vice

Or

Root <---> PCIe Bridge
 &= nbsp;   <---> PCIe/MSI-X Device

And hence = my confusion about how to do the following:
static void pcie=5Fmsix=5Fregister(void)
{  
    pci=5Fbridge=5Fqdev=5Fregister(&pcie=5Fmsix=5Finf= o);  // Is this what I should be doing=3F
  &nb= sp;     OR
    pci=5Fqdev=5Fregister(&pcie=5Fmsix=5Finfo); =         // Or this
}

3. I wasn't sure how = to register the device how to do the initializing. Please see the follow= ing section of code:

void pcie=5Fmsix=5Finit(PCIBus *bus)
{
  &nbs= p; // Is this how we should be doing this=3F
    = pci=5Fcreate=5Fsimple(bus, -1, "pcie=5Fmsix=5Fdevice");
 =        OR
     = ;   pci=5Fbridge=5Fcreate(...);
}

Or if should I use= pci=5Fbridge=5Fcreate(...) i= n place of the pci=5Fcreate=5Fsimple(...)

Also, this confu= sion led me to being unsure what the following device struct should look= like

typedef struct PCIE=5FMSIX=5FDEVState=5FSt {
  = ;  PCIDevice dev;
    int mmio=5Findex;
} PCIE= =5FMSIX=5FDEVState;

For the simple device function that I've desc= ribed above, what is the purpose of this struct=3F What other data shoul= d be captured=3F

Which include the initializing of the following = static structs. Btw, can you tell me what VMStateDescrption is used for = by Qemu=3F Also, what should the "fields" member contain=3F I couldn't q= uite make out.

static const= VMStateDescription vmstate=5Fpcie=5Fmsix =3D {
  &nbs= p; .name =3D "pcie-msix-device",
    .version=5Fi= d =3D 1,
    .minimum=5Fversion=5Fid =3D 1,
    .minimum=5Fversion=5Fid=5Fold =3D 1,
 = ;   .fields =3D (VMStateField[]) {
   &= nbsp;    VMSTATE=5FPCIE=5FDEVICE(dev, PCIE=5FMSIX=5FDEVSt= ate),
        VMSTATE=5FSTRUC= T(dev.aer=5Flog, PCIE=5FMSIX=5FDEVState, 0, vmstate=5Fpcie=5Faer=5Flog, = struct pcie=5Faer=5Flog),
      &n= bsp; VMSTATE=5FEND=5FOF=5FLIST()
    }
};=

4. What is the qdev.props field used for=3F
static PCIDeviceInfo pcie=5Fmsix=5Finfo = =3D {
    .qdev.name =3D PCIE=5FMSIX=5FDEVICE,
    .qdev.desc =3D "PCIE MSIX device template",
    .qdev.size =3D sizeof(PCIE=5FMSIX=5FDEVState),
    .qdev.reset =3D pcie=5Fmsix=5Freset,&= nbsp;   .qdev.vmsd =3D &vmstate=5Fpcie=5Fmsix,
&nb= sp;   .is=5Fexpress =3D 1,
    .config= =5Fwrite =3D pcie=5Fmsix=5Fwrite=5Fconfig,
    .i= nit =3D pcie=5Fmsix=5Finitfn,
=     .exit =3D pcie= =5Fmsix=5Fexitfn,
    .qdev.props =3D (Property[= ]) {       
  &nbs= p;     DEFINE=5FPROP=5FEND=5FOF=5FLIST(),
&n= bsp;   }
};


5. Device instantiation
I init the device in pc=5Fq= 35=5Fbridge=5Finit() in pc=5Fq35.c

pcie=5Fmsix=5Finit(root=5Fport=5Fbus);

I know I'm = doing this incorrectly since I'm not specifying several things. Again, i= s this the correct place to init the device=3F


MSI/MSIX questions1. How is an interrupt notification passed on to Qemu=3F In the regula= r case I'd use qemu=5Fset=5Firq(..) to do so but what is the correct way= of doing it in the MSIX paradigm=3F For example in the case of a DMA tr= ansfer.

-------------f6fee1686365d4888b07072390f9c0e0-- From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from [140.186.70.92] (port=38049 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1Ootde-0006YH-Sq for qemu-devel@nongnu.org; Fri, 27 Aug 2010 03:46:16 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1Ootdd-0000aF-9K for qemu-devel@nongnu.org; Fri, 27 Aug 2010 03:46:14 -0400 Received: from mail.valinux.co.jp ([210.128.90.3]:52426) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Ootdc-0000Zr-RV for qemu-devel@nongnu.org; Fri, 27 Aug 2010 03:46:13 -0400 Date: Fri, 27 Aug 2010 16:57:18 +0900 From: Isaku Yamahata Subject: Re: [Qemu-devel] Template for developing a Qemu device with?PCIe?and MSI-X Message-ID: <20100827075718.GI16489@valinux.co.jp> References: <20100826181738.9bdafe6d@shadowfax.no-ip.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20100826181738.9bdafe6d@shadowfax.no-ip.com> List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Adnan Khaleel Cc: qemu-devel@nongnu.org On Thu, Aug 26, 2010 at 01:17:38PM -0500, Adnan Khaleel wrote: > You also want to catch up pci api clean up. > pci_{set, get}_{byte, word, long, quad}(), > pci_config_set_vendor() ... > > Are you referring to the setting up of the config registers where we pass on > the vendor id and device id etc? Would you elaborate a little more. Yes. Now there are helper functions to address pci endian. So there is no need to ugly bit operation in pcie_msix_initfn(). > Also, I've got a bunch of questions but let me state my assumptions first so > that you have a better idea of what I'm referring to. > - The device template is a pcie endpoint > - I want to be able to setup 64bit bar addresses, with large apertures Use PCI_BASE_ADDRESS_MEM_TYPE_64. > - For now, I'd like to be able to do basic MMIO and regular IO reads and > writes. > > > PCIe questions. > 1. What does the topology of the bridge with respect to the root look like? Is > it > > Root <---> PCIe Bridge lspci -t would help. Roughly the bus topology looks like root port - upstream port --- downstream port -> end node device | | ... ... > 2. If so, where is the slot where I can insert the PCIe device? Is it off the > Bridge or would it be better for it to be off the root? > > Root <---> PCIe Bridge <---> PCIe/MSI-X device > > Or > > Root <---> PCIe Bridge > <---> PCIe/MSI-X Device On downstream port switch. See above. > And hence my confusion about how to do the following: > static void pcie_msix_register(void) > { > pci_bridge_qdev_register(&pcie_msix_info); // Is this what I should be > doing? > OR > pci_qdev_register(&pcie_msix_info); // Or this > } pci_qdev_register() > 3. I wasn't sure how to register the device how to do the initializing. Please > see the following section of code: > > void pcie_msix_init(PCIBus *bus) > { > // Is this how we should be doing this? > pci_create_simple(bus, -1, "pcie_msix_device"); > OR > pci_bridge_create(...); > } > Or if should I use pci_bridge_create(...) in place of the pci_create_simple > (...) pci_create_simple() > Also, this confusion led me to being unsure what the following device struct > should look like > > typedef struct PCIE_MSIX_DEVState_St { > PCIDevice dev; > int mmio_index; > } PCIE_MSIX_DEVState; > > For the simple device function that I've described above, what is the purpose > of this struct? What other data should be captured? > > Which include the initializing of the following static structs. Btw, can you > tell me what VMStateDescrption is used for by Qemu? Also, what should the > "fields" member contain? I couldn't quite make out. > > static const VMStateDescription vmstate_pcie_msix = { > .name = "pcie-msix-device", > .version_id = 1, > .minimum_version_id = 1, > .minimum_version_id_old = 1, > .fields = (VMStateField[]) { > VMSTATE_PCIE_DEVICE(dev, PCIE_MSIX_DEVState), > VMSTATE_STRUCT(dev.aer_log, PCIE_MSIX_DEVState, 0, > vmstate_pcie_aer_log, struct pcie_aer_log), > VMSTATE_END_OF_LIST() > } > }; > > 4. What is the qdev.props field used for? > static PCIDeviceInfo pcie_msix_info = { > .qdev.name = PCIE_MSIX_DEVICE, > .qdev.desc = "PCIE MSIX device template", > .qdev.size = sizeof(PCIE_MSIX_DEVState), > .qdev.reset = pcie_msix_reset, > .qdev.vmsd = &vmstate_pcie_msix, > .is_express = 1, > .config_write = pcie_msix_write_config, > .init = pcie_msix_initfn, > .exit = pcie_msix_exitfn, > .qdev.props = (Property[]) { > DEFINE_PROP_END_OF_LIST(), > } > }; Perhaps at first it would be better to get a idea of what qdev device model is. The following slide would be a good start point. http://www.linux-kvm.org/wiki/images/f/fe/2010-forum-armbru-qdev.pdf > 5. Device instantiation > I init the device in pc_q35_bridge_init() in pc_q35.c > > pcie_msix_init(root_port_bus); > > I know I'm doing this incorrectly since I'm not specifying several things. > Again, is this the correct place to init the device? Yes if you really wanted the end node device to be on root port. Otherwise downstream port is the place on which the end node device is. > MSI/MSIX questions > 1. How is an interrupt notification passed on to Qemu? In the regular case I'd > use qemu_set_irq(..) to do so but what is the correct way of doing it in the > MSIX paradigm? For example in the case of a DMA transfer. Use msix_notify()/msi_notify(). virtio_pci_notify() in virtio-pci.c is an example. pcie_notify() in pcie.c is also another example. -- yamahata From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from [140.186.70.92] (port=47499 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1Op1IN-0002K4-QK for qemu-devel@nongnu.org; Fri, 27 Aug 2010 11:56:53 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1Op1Ai-0000G3-DN for qemu-devel@nongnu.org; Fri, 27 Aug 2010 11:51:20 -0400 Received: from mail-vw0-f45.google.com ([209.85.212.45]:48080) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Op1Ai-0000FJ-Ak for qemu-devel@nongnu.org; Fri, 27 Aug 2010 11:48:52 -0400 Received: by vws19 with SMTP id 19so3089124vws.4 for ; Fri, 27 Aug 2010 08:48:50 -0700 (PDT) MIME-Version: 1.0 Sender: camm@ualberta.ca In-Reply-To: <20100825223950.7e664cb4@shadowfax.no-ip.com> References: <20100825223950.7e664cb4@shadowfax.no-ip.com> Date: Fri, 27 Aug 2010 09:48:48 -0600 Message-ID: Subject: Re: [Qemu-devel] Template for developing a Qemu device with PCIe and MSI-X From: Cam Macdonell Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: adnan@khaleel.us Cc: Isaku Yamahata , qemu-devel@nongnu.org On Wed, Aug 25, 2010 at 4:39 PM, Adnan Khaleel wrote: > Hi Isaku, > > I've made some progress in coding the device template but its no where ne= ar > complete. > > I've created some files and am attaching it to this note. Based on what I > could gather from the pcie source files I've made a stab at creating a > simple model. I've also attached a file for a simple pci device that work= s > under regular Qemu. I would like to duplicate its functionality in your p= cie > environment for starters. > > Could you please take a look at the files I've created and tell me if I'v= e > understood your pcie model correctly. Any help will be truly appreciated. > > Adnan Hi Adnan, There is a fairly simple device I've created called "ivshmem" that is in the qemu git tree. It is a regular PCI device that exports a shared memory object via a BAR and supports a few registers and optional MSI-X interrupts (I had to pick through the virtio code to get MSI-X working, so looking at ivshmem might save you some effort). My device is somewhat similar to a graphics card actually which I recall is your goal. The purpose of ivshmem is to support sharing memory between multiple guests running on the same host. It follows the qdev model which you will need to do. Cam > > The five files I've modified from your git repository are as follows > > hw/pci_ids.h=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 // = Added vendor id defines > hw/pc_q35.c=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 // D= evice instantiation > hw/pcie_msix_template.h=A0 // Device header file > hw/pcie_msix_template.c=A0 // Device file > Makefile.objs=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 // Ad= ded pcie_msix_template.o to list of > objects being built > > Everything should compile without any warnings or errors. > > The last file: > sc_link_pci.c > Is the original PCI device that I'm trying to convert into being PCIe and > MSI-X and is included merely for reference to help you understand what I'= d > like to achieve in your environment. > > > ________________________________ > From: Isaku Yamahata [mailto:yamahata@valinux.co.jp] > To: Adnan Khaleel [mailto:adnan@khaleel.us] > Cc: qemu-devel@nongnu.org > Sent: Wed, 18 Aug 2010 22:19:04 -0500 > Subject: Re: [Qemu-devel] Template for developing a Qemu device with PCIe > and MSI-X > > On Wed, Aug 18, 2010 at 02:10:10PM -0500, Adnan Khaleel wrote: >> Hello Qemu developers, >> >> I'm interested in developing a device model that plugs into Qemu that is >> based >> on a PCIe interface and uses MSI-X. My goal is to ultimately attach a GP= U >> simulator to this PCIe interface and use the entire platfom (Qemu + GPU >> simulator) for studying cpu, gpu interactions. >> >> I'm not terribly familiar with the Qemu device model and I'm looking for >> some >> assistance, perhaps a starting template for pcie and msi-x that would >> offer the >> basic functionality that I could then build upon. >> >> I have looked at the various devices that already modelled that are >> included >> with Qemu (v0.12.5 at least) and I've noticed several a few pci devices, >> eg; >> ne2k and cirrus-pci etc, however only one device truly seems to utilize >> both >> the technologies that I'm interested in and that is the virtio-pci.c >> >> I'm not sure what virtio-pci does so I'm not sure if that is a suitable >> starting point for me. >> >> Any help, suggestions etc would be extremely helpful and much appreciate= d. > > Qemu doesn't support pcie at the moment. > Only partial patches have been merged, still more patches have to > be merged for pcie to fully work. The following repo is available. > > git clone http://people.valinux.co.jp/~yamahata/qemu/q35/qemu > git clone http://people.valinux.co.jp/~yamahata/qemu/q35/seabios > git clone http://people.valinux.co.jp/~yamahata/qemu/q35/vgabios > > Note: patched seabios and vgabios are needed, you have to pass ACPI DSDT > for q35. > example: > qemu-system-x86_64 -M pc_q35 -acpitable > load_header,data=3Droms/seabios/src/q35-acpi-dsdt.aml > > This repo is for those who want to try/develop pcie support, > not for upstream merge. So they include patches unsuitable for upstream. > The repo includes pcie port switch emulator which utilize pcie and > MSI(not MSI-X). > > The difference between PCI device and PCIe device is configuration > space size. > By setting PCIDeviceInfo::is_express =3D 1, you'll get 4K configuration > space. Helper functions for pcie are found in qemu/hw/pcie.c > For msi-x, see qemu/hw/msix.c. > > Thanks, > -- > yamahata > From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from [140.186.70.92] (port=52690 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1Oqset-0002ze-NU for qemu-devel@nongnu.org; Wed, 01 Sep 2010 15:07:45 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1Oqser-0003VE-6w for qemu-devel@nongnu.org; Wed, 01 Sep 2010 15:07:43 -0400 Received: from smtp.mailix.net ([66.11.225.183]:31570) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Oqseq-0003Uu-Lm for qemu-devel@nongnu.org; Wed, 01 Sep 2010 15:07:41 -0400 From: "Adnan Khaleel" Subject: Re: [Qemu-devel] Template for developing a Qemu device with PCIe and MSI-X In-Reply-To: AANLkTimmReFQzgQKL1b1mrCjvRwyGt2fynib_xbHX-B_@mail.gmail.com Message-ID: <20100901190733.5ea9ca03@shadowfax.no-ip.com> Date: Wed, 01 Sep 2010 14:07:33 -0500 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="-----------b0469c9ed3f58912faffa2921ef5b58d" Reply-To: adnan@khaleel.us List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Cam Macdonell Cc: Isaku Yamahata , qemu-devel@nongnu.org This is a multi-part message in MIME format. -------------b0469c9ed3f58912faffa2921ef5b58d Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Yamahata, Cam, Thank you both very much for pointers about Qemu coding for PCIe and MSI= -X. I'm at a point where I can see my device when I do an lspci -t -v as sho= wn below. linux-an84:~ # lspci -t -v -[0000:00]-+-00.0 Intel Corporation 82G33/G31/P35/P31 Express DRAM Cont= roller +-01.0 Cirrus Logic GD 5446 +-04.0-[0000:20]-- +-18.0-[0000:21]-- +-18.1-[0000:22]-- +-18.2-[0000:23]-- +-18.3-[0000:24]-- +-18.4-[0000:25]-- +-18.5-[0000:26]-- +-19.0-[0000:36-bf]--+-00.0-[0000:37-47]--+-00.0-[0000:38]-- | | +-01.0-[0000:39]-- | | +-02.0-[0000:3a]-- | | +-03.0-[0000:3b]-- | | +-04.0-[0000:3c]-- | | +-05.0-[0000:3d]-- | | +-06.0-[0000:3e]-- | | +-07.0-[0000:3f]-- | | +-08.0-[0000:40]---= -00.0 Cray Inc Device 0301 <- The device that I've included However, I'm having a bit of an issue with the MSI-X. I'm following the code examples in virtio-pci.c and ivshmem.c that Cam p= ointed out to. I've got bar 0&1 already occupied so I assign the msix=5F= mmio=5Fmap to bar 2. However, when I do that, Qemu fails to boot and fai= ls with the following assertion fail: unused outb: port=3D0x00f1 data=3D0x00 qemu: fatal: Trying to execute code outside RAM or ROM at 0x00000000= 00100000 Couple of thins I'm unsure about: 1. Am I registering the 64bit bar addresses correctly=3F pci=5Fregister=5Fbar(&d->dev, i, BAR=5FRegions[i][0], PCI=5FBASE=5FADD= RESS=5FSPACE=5FIO | PCI=5FBASE=5FADDRESS=5FMEM=5FTYPE=5F64, pcie=5Fmsi= x=5Fio=5Fmap); 2. In the function int msix=5Finit(PCIDevice *pdev, unsigned short nentr= ies, unsigned bar=5Fnr, unsigned bar=5Fsize); I'm not sure what bar=5Fnr is, however setting it to 1 (as in code examp= les above) or to 2 (bar that I want to register the msix=5Fmmio=5Fmap to= ) both fail with the same error. rc =3D msix=5Finit(&d->dev, d->vectors, 2, 0); : : pci=5Fregister=5Fbar(&d->dev, 2, msix=5Fbar=5Fsize(&d->dev), PCI=5FBAS= E=5FADDRESS=5FSPACE=5FMEMORY | PCI=5FBASE=5FADDRESS=5FMEM=5FTYPE=5F64, m= six=5Fmmio=5Fmap); Here is my init function code in its entirety: static const unsigned long long BAR=5FRegions[6][2] =3D=20 { // len , type=20 { 0x2000000000ull, PCI=5FBASE=5FADDRESS=5FSPACE=5FMEMORY} , //BAR0 { 0x2000000ull, PCI=5FBASE=5FADDRESS=5FSPACE=5FIO } , //BAR1 { 0, 0} , //BAR2 { 0, 0} , //BAR3 { 0, 0} , //BAR4 { 0, 0} //BAR5=20 }; static int pcie=5Fmsix=5Finitfn(PCIDevice *pci=5Fdev) { PCIE=5FMSIX=5FDEVState *d =3D DO=5FUPCAST(PCIE=5FMSIX=5FDEVState, de= v, pci=5Fdev); PCIBridge *br =3D DO=5FUPCAST(PCIBridge, dev, pci=5Fdev); PCIEPort *p =3D DO=5FUPCAST(PCIEPort, br, br); int rc, i; PRINT=5FDEBUG("%s: PCIE MSIX Device init...\n", =5F=5FFUNCTION=5F=5F= ); pci=5Fconfig=5Fset=5Fvendor=5Fid(d->dev.config, PCIE=5FMSIX=5FVID); pci=5Fconfig=5Fset=5Fdevice=5Fid(d->dev.config, PCIE=5FMSIX=5FDID); d->dev.config[PCI=5FREVISION=5FID] =3D PCIE=5FMSIX=5FVERSION; d->dev.config[PCI=5FSUBSYSTEM=5FVENDOR=5FID] =3D PCIE=5FMSIX=5FVID= & 0xff; d->dev.config[PCI=5FSUBSYSTEM=5FVENDOR=5FID+1] =3D (PCIE=5FMSIX=5FVI= D >> 8) & 0xff; d->dev.config[PCI=5FSUBSYSTEM=5FID] =3D PCIE=5FMSIX=5FSS=5FDID & 0= xff; d->dev.config[PCI=5FSUBSYSTEM=5FID+1] =3D (PCIE=5FMSIX=5FSS=5FDID >>= 8) & 0xff; d->mmio=5Findex =3D cpu=5Fregister=5Fio=5Fmemory(pcie=5Fmsix=5Fmem= =5Fread=5Ffn, pcie=5Fmsix=5Fmem=5Fwrite=5Ffn, d); =20 for(i=3D0; i < PCI=5FNUM=5FREGIONS -1; i++) { //-1 for the Exp ROM B= AR if(BAR=5FRegions[i][0] !=3D 0) {=20 if(BAR=5FRegions[i][1] =3D=3D PCI=5FBASE=5FADDRESS=5FSPACE= =5FIO)=20 { //io region PRINT=5FDEBUG("%s: Registering Bar %i as I/O BAR\n", =5F= =5FFUNCTION=5F=5F, i); pci=5Fregister=5Fbar(&d->dev, i, BAR=5FRegions[i][0], PC= I=5FBASE=5FADDRESS=5FSPACE=5FIO | PCI=5FBASE=5FADDRESS=5FMEM=5FTYPE=5F64= , pcie=5Fmsix=5Fio=5Fmap); } else {=20 //mem region PRINT=5FDEBUG("%s: Registering Bar %i as MEM BAR\n", =5F= =5FFUNCTION=5F=5F, i); pci=5Fregister=5Fbar(&d->dev, i, BAR=5FRegions[i][0], PC= I=5FBASE=5FADDRESS=5FSPACE=5FMEMORY | PCI=5FBASE=5FADDRESS=5FMEM=5FTYPE= =5F64, pcie=5Fmsix=5Fmem=5Fmap); } } } d->dev.config[PCI=5FINTERRUPT=5FPIN] =3D 1; rc =3D msix=5Finit(&d->dev, d->vectors, 2, 0); =20 if (!rc) { PRINT=5FDEBUG("%s: Registering Bar %i as I/O BAR\n", =5F=5FFUNCT= ION=5F=5F, i); pci=5Fregister=5Fbar(&d->dev, 2, msix=5Fbar=5Fsize(&d->dev), PCI= =5FBASE=5FADDRESS=5FSPACE=5FMEMORY | PCI=5FBASE=5FADDRESS=5FMEM=5FTYPE= =5F64, msix=5Fmmio=5Fmap); PRINT=5FDEBUG("%s: MSI-X initialized (%d vectors)\n", =5F=5FFUNC= TION=5F=5F, d->vectors); } else { PRINT=5FDEBUG("%s: MSI-X initialization failed!\n", =5F=5FFUNCTI= ON=5F=5F); exit(1); } =20 // Activate the vectors for (i =3D 0; i < d->vectors; i++) { msix=5Fvector=5Fuse(&d->dev, i); } rc =3D pci=5Fpcie=5Fcap=5Finit(&d->dev, PCIE=5FMSIX=5FEXP=5FOFFSET, = PCI=5FEXP=5FTYPE=5FENDPOINT, p->port); if (rc < 0) { return rc; } pcie=5Fcap=5Fflr=5Finit(&d->dev, &pcie=5Fmsix=5Fflr); pcie=5Fcap=5Fdeverr=5Finit(&d->dev); pcie=5Fcap=5Fari=5Finit(&d->dev); rc =3D pcie=5Faer=5Finit(&d->dev, PCIE=5FMSIX=5FAER=5FOFFSET); if (rc < 0) { return rc; } PRINT=5FDEBUG("%s: Init done\n", =5F=5FFUNCTION=5F=5F); return 0; } Thanks AK =5F=5F=5F=5F=5F =20 From: Cam Macdonell [mailto:cam@cs.ualberta.ca] To: adnan@khaleel.us Cc: Isaku Yamahata [mailto:yamahata@valinux.co.jp], qemu-devel@nongnu.or= g Sent: Fri, 27 Aug 2010 10:48:48 -0500 Subject: Re: [Qemu-devel] Template for developing a Qemu device with PCI= e and MSI-X On Wed, Aug 25, 2010 at 4:39 PM, Adnan Khaleel wrote: > Hi Isaku, > > I've made some progress in coding the device template but its no whe= re near > complete. > > I've created some files and am attaching it to this note. Based on w= hat I > could gather from the pcie source files I've made a stab at creating= a > simple model. I've also attached a file for a simple pci device that= works > under regular Qemu. I would like to duplicate its functionality in y= our pcie > environment for starters. > > Could you please take a look at the files I've created and tell me i= f I've > understood your pcie model correctly. Any help will be truly appreci= ated. > > Adnan =20 Hi Adnan, =20 There is a fairly simple device I've created called "ivshmem" that is in the qemu git tree. It is a regular PCI device that exports a shared memory object via a BAR and supports a few registers and optional MSI-X interrupts (I had to pick through the virtio code to get MSI-X working, so looking at ivshmem might save you some effort). My device is somewhat similar to a graphics card actually which I recall is your goal. The purpose of ivshmem is to support sharing memory between multiple guests running on the same host. It follows the qdev model which you will need to do. =20 Cam =20 > > The five files I've modified from your git repository are as follows > > hw/pci=5Fids.h // Added vendor id defines > hw/pc=5Fq35.c // Device instantiation > hw/pcie=5Fmsix=5Ftemplate.h // Device header file > hw/pcie=5Fmsix=5Ftemplate.c // Device file > Makefile.objs // Added pcie=5Fmsix=5Ftemplate.o to= list of > objects being built > > Everything should compile without any warnings or errors. > > The last file: > sc=5Flink=5Fpci.c > Is the original PCI device that I'm trying to convert into being PCI= e and > MSI-X and is included merely for reference to help you understand wh= at I'd > like to achieve in your environment. > > > =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F > From: Isaku Yamahata [mailto:yamahata@valinux.co.jp] > To: Adnan Khaleel [mailto:adnan@khaleel.us] > Cc: qemu-devel@nongnu.org > Sent: Wed, 18 Aug 2010 22:19:04 -0500 > Subject: Re: [Qemu-devel] Template for developing a Qemu device with= PCIe > and MSI-X > > On Wed, Aug 18, 2010 at 02:10:10PM -0500, Adnan Khaleel wrote: >> Hello Qemu developers, >> >> I'm interested in developing a device model that plugs into Qemu th= at is >> based >> on a PCIe interface and uses MSI-X. My goal is to ultimately attach= a GPU >> simulator to this PCIe interface and use the entire platfom (Qemu += GPU >> simulator) for studying cpu, gpu interactions. >> >> I'm not terribly familiar with the Qemu device model and I'm lookin= g for >> some >> assistance, perhaps a starting template for pcie and msi-x that wou= ld >> offer the >> basic functionality that I could then build upon. >> >> I have looked at the various devices that already modelled that are >> included >> with Qemu (v0.12.5 at least) and I've noticed several a few pci dev= ices, >> eg; >> ne2k and cirrus-pci etc, however only one device truly seems to uti= lize >> both >> the technologies that I'm interested in and that is the virtio-pci.= c >> >> I'm not sure what virtio-pci does so I'm not sure if that is a suit= able >> starting point for me. >> >> Any help, suggestions etc would be extremely helpful and much appre= ciated. > > Qemu doesn't support pcie at the moment. > Only partial patches have been merged, still more patches have to > be merged for pcie to fully work. The following repo is available. > > git clone http://people.valinux.co.jp/~yamahata/qemu/q35/qemu > git clone http://people.valinux.co.jp/~yamahata/qemu/q35/seabios > git clone http://people.valinux.co.jp/~yamahata/qemu/q35/vgabios > > Note: patched seabios and vgabios are needed, you have to pass ACPI = DSDT > for q35. > example: > qemu-system-x86=5F64 -M pc=5Fq35 -acpitable > load=5Fheader,data=3Droms/seabios/src/q35-acpi-dsdt.aml > > This repo is for those who want to try/develop pcie support, > not for upstream merge. So they include patches unsuitable for upstr= eam. > The repo includes pcie port switch emulator which utilize pcie and > MSI(not MSI-X). > > The difference between PCI device and PCIe device is configuration > space size. > By setting PCIDeviceInfo::is=5Fexpress =3D 1, you'll get 4K configur= ation > space. Helper functions for pcie are found in qemu/hw/pcie.c > For msi-x, see qemu/hw/msix.c. > > Thanks, > -- > yamahata > =20 -------------b0469c9ed3f58912faffa2921ef5b58d Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
Yamahata, Cam,

Thank you both ve= ry much for pointers about Qemu coding for PCIe and MSI-X.

I'm at= a point where I can see my device when I do an lspci -t -v as shown bel= ow.

linux-= an84:~ # lspci -t -v
-[0000:00]-+-00.0  Intel Corporation 8= 2G33/G31/P35/P31 Express DRAM Controller
    = ;       +-01.0  Cirrus Logic GD 5446<= /span>
           +-0= 4.0-[0000:20]--
        =    +-18.0-[0000:21]--
     &n= bsp;     +-18.1-[0000:22]--
  &nbs= p;        +-18.2-[0000:23]--=            +-18.3-[000= 0:24]--
         &n= bsp; +-18.4-[0000:25]--
      &nbs= p;    +-18.5-[0000:26]--
    =        +-19.0-[0000:36-bf]--+-00.0-[0000:3= 7-47]--+-00.0-[0000:38]--
      &n= bsp;    |        =             |&nbs= p;           &nbs= p;       +-01.0-[0000:39]--
 =           |  &nbs= p;           &nbs= p;     |       &n= bsp;            += -02.0-[0000:3a]--
       &nb= sp;   |         &= nbsp;          |  = ;            = ;      +-03.0-[0000:3b]--
  &= nbsp;        |    = ;            = ;    |        &nb= sp;           +-04.0-[= 0000:3c]--
         = ;  |          &nb= sp;         |   &= nbsp;           &= nbsp;    +-05.0-[0000:3d]--
   &nb= sp;       |     &= nbsp;           &= nbsp;  |          = ;          +-06.0-[0000:3e]= --
           = |            = ;        |    &nb= sp;           &nb= sp;   +-07.0-[0000:3f]--
     = ;      |      &nb= sp;           &nb= sp; |           &= nbsp;        +-08.0-[0000:40]----00.0=   Cray Inc Device 0301  <- The device that I'= ve included

However, I'm having a bit of an issue with the MSI-X.=

I'm following the code examples in virtio-pci.c and ivshmem.c th= at Cam pointed out to. I've got bar 0&1 already occupied so I assign= the msix=5Fmmio=5Fmap to bar 2. However, when I do that, Qemu fails to = boot and fails with the following assertion fail:

unused outb: port=3D0x00f1 data=3D= 0x00
qemu: fatal: Trying to execute cod= e=20 outside RAM or ROM at 0x0000000000100000

Couple of thins I= 'm unsure about:
1. Am I registering the 64bit bar addresses correctl= y=3F
pci=5Fregister=5Fbar(&= d->dev, i,=20 BAR=5FRegions[i][0], PCI=5FBASE=5FADDRESS=5FSPACE=5FIO |=20 PCI=5FBASE=5FADDRESS=5FMEM=5FTYPE=5F64, pcie=5Fmsix=5Fio=5Fmap);<= br>
2. In the function int msix=5Finit(PCIDevice *pdev, unsigned shor= t nentries, unsigned bar=5Fnr, unsigned bar=5Fsize);
I'm not sure wha= t bar=5Fnr is, however setting it to 1 (as in code examples above) or to= 2 (bar that I want to register the msix=5Fmmio=5Fmap to) both fail with= the same error.

rc =3D msi= x=5Finit(&d->dev,=20 d->vectors, 2, 0);
:
:
pci=5Fregister=5Fbar(&d->dev, 2,=20 msix=5Fbar=5Fsize(&d->dev), PCI=5FBASE=5FADDRESS=5FSPACE=5FMEMORY= | PCI=5FBASE=5FADDRESS=5FM= EM=5FTYPE=5F64, msix=5Fmmio= =5Fmap);


Here is my init function code in its entirety= :

static const unsigned lon= g long BAR=5FRegions[6][2] =3D
{
    // = len , type
    { 0x2000000000ull, PCI=5FBASE=5FA= DDRESS=5FSPACE=5FMEMORY} ,  //BAR0
    { 0x2= 000000ull,    PCI=5FBASE=5FADDRESS=5FSPACE=5FIO &nbs= p;  } ,  //BAR1
    { 0, 0} ,  //B= AR2
    { 0, 0} ,  //BAR3
 &nbs= p;  { 0, 0} ,  //BAR4
    { 0, 0} =    //BAR5
};

static int pcie=5Fmsix=5Fini= tfn(PCIDevice *pci=5Fdev)
{
    PCIE=5FMS= IX=5FDEVState *d =3D DO=5FUPCAST(PCIE=5FMSIX=5FDEVState, dev, pci=5Fdev)= ;
    PCIBridge *br =3D DO=5FUPCAST(PCIBridge, de= v, pci=5Fdev);
    PCIEPort *p =3D DO=5FUPCAST(PC= IEPort, br, br);
    int rc, i;

    PRINT=5FDEBUG("%s: PCIE = MSIX Device init...\n", =5F=5FFUNCTION=5F=5F);

    pci=5Fconfig=5Fset=5Fvendor=5Fid(d= ->dev.config, PCIE=5FMSIX=5FVID);
    pci=5Fco= nfig=5Fset=5Fdevice=5Fid(d->dev.config, PCIE=5FMSIX=5FDID);&= nbsp;   d->dev.config[PCI=5FREVISION=5FID] =3D PCIE=5FMSIX= =5FVERSION;
    d->dev.config[PCI=5FSUBSYSTEM= =5FVENDOR=5FID]   =3D PCIE=5FMSIX=5FVID & 0xff;
&n= bsp;   d->dev.config[PCI=5FSUBSYSTEM=5FVENDOR=5FID+1] =3D (= PCIE=5FMSIX=5FVID >> 8) & 0xff;
    d-&= gt;dev.config[PCI=5FSUBSYSTEM=5FID]   =3D PCIE=5FMSIX=5FSS=5FD= ID & 0xff;
    d->dev.config[PCI=5FSUBSYST= EM=5FID+1] =3D (PCIE=5FMSIX=5FSS=5FDID >> 8) & 0xff;
    d->mmio=5Findex= =3D cpu=5Fregister=5Fio=5Fmemory(pcie=5Fmsix=5Fmem=5Fread=5Ffn, pcie=5F= msix=5Fmem=5Fwrite=5Ffn, d);
<= span style=3D"font-family: Courier;">   
 &= nbsp;  for(i=3D0; i < PCI=5FNUM=5FREGIONS -1; i++) { //-1 for th= e Exp ROM BAR
        if(BAR= =5FRegions[i][0] !=3D 0)
      &nb= sp; {
         &nb= sp;  if(BAR=5FRegions[i][1] =3D=3D PCI=5FBASE=5FADDRESS=5FSPACE=5FI= O)
          =   {
         &= nbsp;      //io region
  &nbs= p;           &nbs= p; PRINT=5FDEBUG("%s: Registering Bar %i as I/O BAR\n", =5F=5FFUNCTION= =5F=5F, i);
        &nbs= p;       pci=5Fregister=5Fbar(&d->d= ev, i, BAR=5FRegions[i][0], PCI=5FBASE=5FADDRESS=5FSPACE=5FIO | PCI=5FBA= SE=5FADDRESS=5FMEM=5FTYPE=5F64, pcie=5Fmsix=5Fio=5Fmap);
 &= nbsp;          } else {
           =      //mem region
    &n= bsp;           PRINT= =5FDEBUG("%s: Registering Bar %i as MEM BAR\n", =5F=5FFUNCTION=5F=5F, i)= ;
          &n= bsp;     pci=5Fregister=5Fbar(&d->dev, i, BAR= =5FRegions[i][0], PCI=5FBASE=5FADDRESS=5FSPACE=5FMEMORY | PCI=5FBASE=5FA= DDRESS=5FMEM=5FTYPE=5F64, pcie=5Fmsix=5Fmem=5Fmap);
  =           }
 &= nbsp;      }
    }
    d->dev.config[PCI=5FINTERRUPT=5FPIN] =3D 1;

    rc =3D msix= =5Finit(&d->dev, d->vectors, 2, 0);
   =
    if (!rc) {
    =     PRINT=5FDEBUG("%s: Registering Bar %i as I/O BAR\n", = =5F=5FFUNCTION=5F=5F, i);
      &n= bsp; pci=5Fregister=5Fbar(&d->dev, 2, msix=5Fbar=5Fsize(&d-&g= t;dev), PCI=5FBASE=5FADDRESS=5FSPACE=5FMEMORY | PCI=5FBASE=5FADDRESS=5FMEM=5FTYPE=5F64, msix=5Fmmio=5Fmap);
 &nbs= p;      PRINT=5FDEBUG("%s: MSI-X initialized (%= d vectors)\n", =5F=5FFUNCTION=5F=5F, d->vectors);
  = ;  }
    else {
   &n= bsp;    PRINT=5FDEBUG("%s: MSI-X initialization failed!\n= ", =5F=5FFUNCTION=5F=5F);
      &n= bsp; exit(1);
    }
   
    // Activate the vectors
  &n= bsp; for (i =3D 0; i < d->vectors; i++) {
  &nbs= p;     msix=5Fvector=5Fuse(&d->dev, i);
    }

&nbs= p;   rc =3D pci=5Fpcie=5Fcap=5Finit(&d->dev, PCIE=5FMSI= X=5FEXP=5FOFFSET, PCI=5FEXP=5FTYPE=5FENDPOINT, p->port);
&nbs= p;   if (rc < 0) {
     &n= bsp;  return rc;
    }

    pcie=5Fcap=5Fflr=5Finit(&d= ->dev, &pcie=5Fmsix=5Fflr);
    pcie=5Fcap= =5Fdeverr=5Finit(&d->dev);
    pcie=5Fcap= =5Fari=5Finit(&d->dev);
    rc =3D pcie=5F= aer=5Finit(&d->dev, PCIE=5FMSIX=5FAER=5FOFFSET);
 &= nbsp;  if (rc < 0) {
<= span style=3D"font-family: Courier;">      = ;  return rc;
    }

    PRINT=5FDEBUG("%s: Init done\n", = =5F=5FFUNCTION=5F=5F);
    return 0;
}


Thanks

AK



From: Cam Macdonell [mailto:cam@cs.ualbert= a.ca]
To: adnan@khaleel.us
Cc: Isaku Yamahata [mailt= o:yamahata@valinux.co.jp], qemu-devel@nongnu.org
Sent: Fri, 27= Aug 2010 10:48:48 -0500
Subject: Re: [Qemu-devel] Template fo= r developing a Qemu device with PCIe and MSI-X

On Wed, Aug 25, 20= 10 at 4:39 PM, Adnan Khaleel <adn= an@khaleel.us> wrote:
> Hi Isaku,
>
> I've made some progress in coding the device template but its no wh= ere near
> complete.
>
> I've created some files and am attaching it to this note. Based on = what I
> could gather from the pcie source files I've made a stab at creatin= g a
> simple model. I've also attached a file for a simple pci device tha= t works
> under regular Qemu. I would like to duplicate its functionality in = your pcie
> environment for starters.
>
> Could you please take a look at the files I've created and tell me = if I've
> understood your pcie model correctly. Any help will be truly apprec= iated.
>
> Adnan

Hi Adnan,

There is a fairly simple device I've created called "ivshmem" that is in the qemu git tree. It is a regular PCI device that exports a
shared memory object via a BAR and supports a few registers and
optional MSI-X interrupts (I had to pick through the virtio code to
get MSI-X working, so looking at ivshmem might save you some effort). My device is somewhat similar to a graphics card actually which I
recall is your goal. The purpose of ivshmem is to support sharing
memory between multiple guests running on the same host. It follows
the qdev model which you will need to do.

Cam

>
> The five files I've modified from your git repository are as follow= s
>
> hw/pci=5Fids.h         = ;           // Added v= endor id defines
> hw/pc=5Fq35.c         =            // Device i= nstantiation
> hw/pcie=5Fmsix=5Ftemplate.h  // Device header file
> hw/pcie=5Fmsix=5Ftemplate.c  // Device file
> Makefile.objs         =           // Added pcie=5Fm= six=5Ftemplate.o to list of
> objects being built
>
> Everything should compile without any warnings or errors.
>
> The last file:
> sc=5Flink=5Fpci.c
> Is the original PCI device that I'm trying to convert into being PC= Ie and
> MSI-X and is included merely for reference to help you understand w= hat I'd
> like to achieve in your environment.
>
>
> =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F
> From: Isaku Yamahata [mailto:yamahata@valinux.co.jp]
> To: Adnan Khaleel [mailto:adnan= @khaleel.us]
> Cc: qemu-devel@nongnu.org<= /a>
> Sent: Wed, 18 Aug 2010 22:19:04 -0500
> Subject: Re: [Qemu-devel] Template for developing a Qemu device wit= h PCIe
> and MSI-X
>
> On Wed, Aug 18, 2010 at 02:10:10PM -0500, Adnan Khaleel wrote:
>> Hello Qemu developers,
>>
>> I'm interested in developing a device model that plugs into Qem= u that is
>> based
>> on a PCIe interface and uses MSI-X. My goal is to ultimately at= tach a GPU
>> simulator to this PCIe interface and use the entire platfom (Qe= mu + GPU
>> simulator) for studying cpu, gpu interactions.
>>
>> I'm not terribly familiar with the Qemu device model and I'm lo= oking for
>> some
>> assistance, perhaps a starting template for pcie and msi-x that= would
>> offer the
>> basic functionality that I could then build upon.
>>
>> I have looked at the various devices that already modelled that= are
>> included
>> with Qemu (v0.12.5 at least) and I've noticed several a few pci= devices,
>> eg;
>> ne2k and cirrus-pci etc, however only one device truly seems to= utilize
>> both
>> the technologies that I'm interested in and that is the virtio-= pci.c
>>
>> I'm not sure what virtio-pci does so I'm not sure if that is a = suitable
>> starting point for me.
>>
>> Any help, suggestions etc would be extremely helpful and much a= ppreciated.
>
> Qemu doesn't support pcie at the moment.
> Only partial patches have been merged, still more patches have to > be merged for pcie to fully work. The following repo is available.<= br> >
> git clone http://people.valinux.co.jp/~yamahata/qemu/= q35/qemu
> git clone http://people.valinux.co.jp/~yamahata/qe= mu/q35/seabios
> git clone http://people.valinux.co.jp/~yamahata/qe= mu/q35/vgabios
>
> Note: patched seabios and vgabios are needed, you have to pass ACPI= DSDT
> for q35.
> example:
> qemu-system-x86=5F64 -M pc=5Fq35 -acpitable
> load=5Fheader,data=3Droms/seabios/src/q35-acpi-dsdt.aml
>
> This repo is for those who want to try/develop pcie support,
> not for upstream merge. So they include patches unsuitable for upst= ream.
> The repo includes pcie port switch emulator which utilize pcie and<= br> > MSI(not MSI-X).
>
> The difference between PCI device and PCIe device is configuration<= br> > space size.
> By setting PCIDeviceInfo::is=5Fexpress =3D 1, you'll get 4K configu= ration
> space. Helper functions for pcie are found in qemu/hw/pcie.c
> For msi-x, see qemu/hw/msix.c.
>
> Thanks,
> --
> yamahata
>
-------------b0469c9ed3f58912faffa2921ef5b58d-- From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from [140.186.70.92] (port=55833 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OqzV9-000205-Lg for qemu-devel@nongnu.org; Wed, 01 Sep 2010 22:26:09 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OqzV7-0005cr-KY for qemu-devel@nongnu.org; Wed, 01 Sep 2010 22:26:07 -0400 Received: from mail.valinux.co.jp ([210.128.90.3]:36845) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OqzV6-0005cF-Nn for qemu-devel@nongnu.org; Wed, 01 Sep 2010 22:26:05 -0400 Date: Thu, 2 Sep 2010 11:38:31 +0900 From: Isaku Yamahata Subject: Re: [Qemu-devel] Template for developing a Qemu device with PCIe and MSI-X Message-ID: <20100902023831.GA20106@valinux.co.jp> References: <20100901190733.5ea9ca03@shadowfax.no-ip.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-2022-jp Content-Disposition: inline In-Reply-To: <20100901190733.5ea9ca03@shadowfax.no-ip.com> List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Adnan Khaleel Cc: Cam Macdonell , qemu-devel@nongnu.org On Wed, Sep 01, 2010 at 02:07:33PM -0500, Adnan Khaleel wrote: > Yamahata, Cam, > > Thank you both very much for pointers about Qemu coding for PCIe and MSI-X. > > I'm at a point where I can see my device when I do an lspci -t -v as shown > below. > > linux-an84:~ # lspci -t -v > -[0000:00]-+-00.0 Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller > +-01.0 Cirrus Logic GD 5446 > +-04.0-[0000:20]-- > +-18.0-[0000:21]-- > +-18.1-[0000:22]-- > +-18.2-[0000:23]-- > +-18.3-[0000:24]-- > +-18.4-[0000:25]-- > +-18.5-[0000:26]-- > +-19.0-[0000:36-bf]--+-00.0-[0000:37-47]--+-00.0-[0000:38]-- > | | +-01.0-[0000:39]-- > | | +-02.0-[0000:3a]-- > | | +-03.0-[0000:3b]-- > | | +-04.0-[0000:3c]-- > | | +-05.0-[0000:3d]-- > | | +-06.0-[0000:3e]-- > | | +-07.0-[0000:3f]-- > | | +-08.0-[0000:40]----00.0 > Cray Inc Device 0301 <- The device that I've included > > However, I'm having a bit of an issue with the MSI-X. > > I'm following the code examples in virtio-pci.c and ivshmem.c that Cam pointed > out to. I've got bar 0&1 already occupied so I assign the msix_mmio_map to bar > 2. However, when I do that, Qemu fails to boot and fails with the following > assertion fail: > > unused outb: port=0x00f1 data=0x00 > qemu: fatal: Trying to execute code outside RAM or ROM at 0x0000000000100000 > > Couple of thins I'm unsure about: > 1. Am I registering the 64bit bar addresses correctly? > pci_register_bar(&d->dev, i, BAR_Regions[i][0], PCI_BASE_ADDRESS_SPACE_IO | > PCI_BASE_ADDRESS_MEM_TYPE_64, pcie_msix_io_map); No. PCI_BASE_ADDRESS_SPACE_IO | PCI_BASE_ADDRESS_MEM_TYPE_64 is invalid. PCI doesn't support 64bit io space, but 32bit. And x86 supports 64k io sparce. > 2. In the function int msix_init(PCIDevice *pdev, unsigned short nentries, > unsigned bar_nr, unsigned bar_size); > I'm not sure what bar_nr is, however setting it to 1 (as in code examples > above) or to 2 (bar that I want to register the msix_mmio_map to) both fail > with the same error. It should be 2. I'm not sure why it failed. > rc = msix_init(&d->dev, d->vectors, 2, 0); > : > : > pci_register_bar(&d->dev, 2, msix_bar_size(&d->dev), > PCI_BASE_ADDRESS_SPACE_MEMORY | PCI_BASE_ADDRESS_MEM_TYPE_64, msix_mmio_map); > > > Here is my init function code in its entirety: > > static const unsigned long long BAR_Regions[6][2] = > { > // len , type > { 0x2000000000ull, PCI_BASE_ADDRESS_SPACE_MEMORY} , //BAR0 > { 0x2000000ull, PCI_BASE_ADDRESS_SPACE_IO } , //BAR1 > { 0, 0} , //BAR2 > { 0, 0} , //BAR3 > { 0, 0} , //BAR4 > { 0, 0} //BAR5 > }; Probably what you want is something like { 0x2000000000ull, PCI_BASE_ADDRESS_SPACE_MEMORY | PCI_BASE_ADDRESS_MEM_TYPE_64} , //BAR0 { 0, 0} , //BAR1 // 64bit BAR occupies 2 BAR entries so that BAR1 can't be used. { 0x2000000ull, PCI_BASE_ADDRESS_SPACE_IO } , //BAR2 { 0, 0} , //BAR3 // for MSI-X { 0, 0} , //BAR4 { 0, 0} //BAR5 > > static int pcie_msix_initfn(PCIDevice *pci_dev) > { > PCIE_MSIX_DEVState *d = DO_UPCAST(PCIE_MSIX_DEVState, dev, pci_dev); > PCIBridge *br = DO_UPCAST(PCIBridge, dev, pci_dev); > PCIEPort *p = DO_UPCAST(PCIEPort, br, br); > int rc, i; > > PRINT_DEBUG("%s: PCIE MSIX Device init...\n", __FUNCTION__); > > pci_config_set_vendor_id(d->dev.config, PCIE_MSIX_VID); > pci_config_set_device_id(d->dev.config, PCIE_MSIX_DID); > d->dev.config[PCI_REVISION_ID] = PCIE_MSIX_VERSION; > d->dev.config[PCI_SUBSYSTEM_VENDOR_ID] = PCIE_MSIX_VID & 0xff; > d->dev.config[PCI_SUBSYSTEM_VENDOR_ID+1] = (PCIE_MSIX_VID >> 8) & 0xff; > d->dev.config[PCI_SUBSYSTEM_ID] = PCIE_MSIX_SS_DID & 0xff; > d->dev.config[PCI_SUBSYSTEM_ID+1] = (PCIE_MSIX_SS_DID >> 8) & 0xff; Use pci_set_word(). > d->mmio_index = cpu_register_io_memory(pcie_msix_mem_read_fn, > pcie_msix_mem_write_fn, d); > > for(i=0; i < PCI_NUM_REGIONS -1; i++) { //-1 for the Exp ROM BAR > if(BAR_Regions[i][0] != 0) > { > if(BAR_Regions[i][1] == PCI_BASE_ADDRESS_SPACE_IO) > { > //io region > PRINT_DEBUG("%s: Registering Bar %i as I/O BAR\n", > __FUNCTION__, i); > pci_register_bar(&d->dev, i, BAR_Regions[i][0], > PCI_BASE_ADDRESS_SPACE_IO | PCI_BASE_ADDRESS_MEM_TYPE_64, pcie_msix_io_map); > } else { > //mem region > PRINT_DEBUG("%s: Registering Bar %i as MEM BAR\n", > __FUNCTION__, i); > pci_register_bar(&d->dev, i, BAR_Regions[i][0], > PCI_BASE_ADDRESS_SPACE_MEMORY | PCI_BASE_ADDRESS_MEM_TYPE_64, > pcie_msix_mem_map); > } > } > } > d->dev.config[PCI_INTERRUPT_PIN] = 1; > > rc = msix_init(&d->dev, d->vectors, 2, 0); > > if (!rc) { > PRINT_DEBUG("%s: Registering Bar %i as I/O BAR\n", __FUNCTION__, i); > pci_register_bar(&d->dev, 2, msix_bar_size(&d->dev), > PCI_BASE_ADDRESS_SPACE_MEMORY | PCI_BASE_ADDRESS_MEM_TYPE_64, msix_mmio_map); > PRINT_DEBUG("%s: MSI-X initialized (%d vectors)\n", __FUNCTION__, d-> > vectors); > } > else { > PRINT_DEBUG("%s: MSI-X initialization failed!\n", __FUNCTION__); > exit(1); > } > > // Activate the vectors > for (i = 0; i < d->vectors; i++) { > msix_vector_use(&d->dev, i); > } > > rc = pci_pcie_cap_init(&d->dev, PCIE_MSIX_EXP_OFFSET, > PCI_EXP_TYPE_ENDPOINT, p->port); > if (rc < 0) { > return rc; > } > > pcie_cap_flr_init(&d->dev, &pcie_msix_flr); > pcie_cap_deverr_init(&d->dev); > pcie_cap_ari_init(&d->dev); > rc = pcie_aer_init(&d->dev, PCIE_MSIX_AER_OFFSET); > if (rc < 0) { > return rc; > } > > PRINT_DEBUG("%s: Init done\n", __FUNCTION__); > return 0; > } > > > Thanks > > AK > > > > > ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ > From: Cam Macdonell [mailto:cam@cs.ualberta.ca] > To: adnan@khaleel.us > Cc: Isaku Yamahata [mailto:yamahata@valinux.co.jp], qemu-devel@nongnu.org > Sent: Fri, 27 Aug 2010 10:48:48 -0500 > Subject: Re: [Qemu-devel] Template for developing a Qemu device with PCIe > and MSI-X > > On Wed, Aug 25, 2010 at 4:39 PM, Adnan Khaleel wrote: > > Hi Isaku, > > > > I've made some progress in coding the device template but its no where > near > > complete. > > > > I've created some files and am attaching it to this note. Based on what I > > could gather from the pcie source files I've made a stab at creating a > > simple model. I've also attached a file for a simple pci device that > works > > under regular Qemu. I would like to duplicate its functionality in your > pcie > > environment for starters. > > > > Could you please take a look at the files I've created and tell me if > I've > > understood your pcie model correctly. Any help will be truly appreciated. > > > > Adnan > > Hi Adnan, > > There is a fairly simple device I've created called "ivshmem" that is > in the qemu git tree. It is a regular PCI device that exports a > shared memory object via a BAR and supports a few registers and > optional MSI-X interrupts (I had to pick through the virtio code to > get MSI-X working, so looking at ivshmem might save you some effort). > My device is somewhat similar to a graphics card actually which I > recall is your goal. The purpose of ivshmem is to support sharing > memory between multiple guests running on the same host. It follows > the qdev model which you will need to do. > > Cam > > > > > The five files I've modified from your git repository are as follows > > > > hw/pci_ids.h // Added vendor id defines > > hw/pc_q35.c // Device instantiation > > hw/pcie_msix_template.h // Device header file > > hw/pcie_msix_template.c // Device file > > Makefile.objs // Added pcie_msix_template.o to list of > > objects being built > > > > Everything should compile without any warnings or errors. > > > > The last file: > > sc_link_pci.c > > Is the original PCI device that I'm trying to convert into being PCIe and > > MSI-X and is included merely for reference to help you understand what > I'd > > like to achieve in your environment. > > > > > > ________________________________ > > From: Isaku Yamahata [mailto:yamahata@valinux.co.jp] > > To: Adnan Khaleel [mailto:adnan@khaleel.us] > > Cc: qemu-devel@nongnu.org > > Sent: Wed, 18 Aug 2010 22:19:04 -0500 > > Subject: Re: [Qemu-devel] Template for developing a Qemu device with PCIe > > and MSI-X > > > > On Wed, Aug 18, 2010 at 02:10:10PM -0500, Adnan Khaleel wrote: > >> Hello Qemu developers, > >> > >> I'm interested in developing a device model that plugs into Qemu that is > >> based > >> on a PCIe interface and uses MSI-X. My goal is to ultimately attach a > GPU > >> simulator to this PCIe interface and use the entire platfom (Qemu + GPU > >> simulator) for studying cpu, gpu interactions. > >> > >> I'm not terribly familiar with the Qemu device model and I'm looking for > >> some > >> assistance, perhaps a starting template for pcie and msi-x that would > >> offer the > >> basic functionality that I could then build upon. > >> > >> I have looked at the various devices that already modelled that are > >> included > >> with Qemu (v0.12.5 at least) and I've noticed several a few pci devices, > >> eg; > >> ne2k and cirrus-pci etc, however only one device truly seems to utilize > >> both > >> the technologies that I'm interested in and that is the virtio-pci.c > >> > >> I'm not sure what virtio-pci does so I'm not sure if that is a suitable > >> starting point for me. > >> > >> Any help, suggestions etc would be extremely helpful and much > appreciated. > > > > Qemu doesn't support pcie at the moment. > > Only partial patches have been merged, still more patches have to > > be merged for pcie to fully work. The following repo is available. > > > > git clone http://people.valinux.co.jp/~yamahata/qemu/q35/qemu > > git clone http://people.valinux.co.jp/~yamahata/qemu/q35/seabios > > git clone http://people.valinux.co.jp/~yamahata/qemu/q35/vgabios > > > > Note: patched seabios and vgabios are needed, you have to pass ACPI DSDT > > for q35. > > example: > > qemu-system-x86_64 -M pc_q35 -acpitable > > load_header,data=roms/seabios/src/q35-acpi-dsdt.aml > > > > This repo is for those who want to try/develop pcie support, > > not for upstream merge. So they include patches unsuitable for upstream. > > The repo includes pcie port switch emulator which utilize pcie and > > MSI(not MSI-X). > > > > The difference between PCI device and PCIe device is configuration > > space size. > > By setting PCIDeviceInfo::is_express = 1, you'll get 4K configuration > > space. Helper functions for pcie are found in qemu/hw/pcie.c > > For msi-x, see qemu/hw/msix.c. > > > > Thanks, > > -- > > yamahata > > > -- yamahata From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from [140.186.70.92] (port=42778 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OrDoH-0000DR-VR for qemu-devel@nongnu.org; Thu, 02 Sep 2010 13:42:51 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OrDoG-0000ph-AM for qemu-devel@nongnu.org; Thu, 02 Sep 2010 13:42:49 -0400 Received: from smtp.mailix.net ([66.11.225.183]:29288) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OrDoF-0000pa-SB for qemu-devel@nongnu.org; Thu, 02 Sep 2010 13:42:48 -0400 From: "Adnan Khaleel" Subject: =?iso-8859-1?Q?Re=3A_=5BQemu-devel=5D_Template_for_developing_a_Qe?= =?iso-8859-1?Q?mu_device_with_PCIe=09and_MSI-X?= In-Reply-To: 20100902023831.GA20106@valinux.co.jp Message-ID: <20100902174242.4cb1fb0f@shadowfax.no-ip.com> Date: Thu, 02 Sep 2010 12:42:42 -0500 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="-----------9ff798cb7af029a1bad9e6189ec420e7" Reply-To: adnan@khaleel.us List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Isaku Yamahata Cc: Cam Macdonell , qemu-devel@nongnu.org This is a multi-part message in MIME format. -------------9ff798cb7af029a1bad9e6189ec420e7 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable I've tried everything you mentioned and I still get the same problem. Th= e only thing that seems to avoid that issue is if I reduce the aperture = size from 0x2000000000ull to 0x2000000ull. Here is the relevant section of code: static const unsigned long long BAR=5FRegions[6][2] =3D=20 { // len , type=20 { 0x2000000ull, PCI=5FBASE=5FADDRESS=5FSPACE=5FMEMORY | PCI=5FBASE= =5FADDRESS=5FMEM=5FTYPE=5F64} , //BAR0, =20 { 0, 0} , // BAR1 { 0x2000000ull, PCI=5FBASE=5FADDRESS=5FSPACE=5FIO } , //BAR2, { 0, 0} , // BAR3 for MSI-X { 0, 0} , // BAR4 { 0, 0} , // BAR5 =20 }; static int pcie=5Fmsix=5Finitfn(PCIDevice *pci=5Fdev) { PCIE=5FMSIX=5FDEVState *d =3D DO=5FUPCAST(PCIE=5FMSIX=5FDEVState, de= v, pci=5Fdev); PCIBridge *br =3D DO=5FUPCAST(PCIBridge, dev, pci=5Fdev); PCIEPort *p =3D DO=5FUPCAST(PCIEPort, br, br); int rc, i; PRINT=5FDEBUG("%s: PCIE MSIX Device init...\n", =5F=5FFUNCTION=5F=5F= ); pci=5Fconfig=5Fset=5Fvendor=5Fid(d->dev.config, PCIE=5FMSIX=5FVID); pci=5Fconfig=5Fset=5Fdevice=5Fid(d->dev.config, PCIE=5FMSIX=5FDID); memcpy(d->dev.config, g=5Fcfg=5Finit, sizeof(g=5Fcfg=5Finit[0x20])); d->mmio=5Findex =3D cpu=5Fregister=5Fio=5Fmemory(pcie=5Fmsix=5Fmem= =5Fread=5Ffn, pcie=5Fmsix=5Fmem=5Fwrite=5Ffn, d); int msix=5Fmem=5Fbar =3D 0; // Since its a 64bit BAR, we take up BA= R0 & BAR1 int msix=5Fio=5Fbar =3D 2; int msix=5Fmmio=5Fbar =3D 3; pci=5Fregister=5Fbar(&d->dev, msix=5Fmem=5Fbar, BAR=5FRegions[msix= =5Fmem=5Fbar][0], BAR=5FRegions[msix=5Fmem=5Fbar][1], pcie=5Fmsix=5Fmem= =5Fmap); pci=5Fregister=5Fbar(&d->dev, msix=5Fio=5Fbar, BAR=5FRegions[msix=5F= io=5Fbar][0], BAR=5FRegions[msix=5Fio=5Fbar][1], pcie=5Fmsix=5Fio=5Fmap)= ; rc =3D msix=5Finit(&d->dev, d->vectors, msix=5Fmmio=5Fbar, 0); =20 if (!rc) { PRINT=5FDEBUG("%s: Registering Bar %i as I/O BAR\n", =5F=5FFUNCT= ION=5F=5F, msix=5Fmmio=5Fbar); pci=5Fregister=5Fbar(&d->dev, msix=5Fmmio=5Fbar, msix=5Fbar=5Fsi= ze(&d->dev), PCI=5FBASE=5FADDRESS=5FSPACE=5FMEMORY, msix=5Fmmio=5Fmap); PRINT=5FDEBUG("%s: MSI-X initialized (%d vectors)\n", =5F=5FFUNC= TION=5F=5F, d->vectors); } else { PRINT=5FDEBUG("%s: MSI-X initialization failed!\n", =5F=5FFUNCTI= ON=5F=5F); return rc; } =20 // Activate the vectors for (i =3D 0; i < d->vectors; i++) { msix=5Fvector=5Fuse(&d->dev, i); } rc =3D pci=5Fpcie=5Fcap=5Finit(&d->dev, PCIE=5FMSIX=5FEXP=5FOFFSET, = PCI=5FEXP=5FTYPE=5FENDPOINT, p->port); if (rc < 0) { return rc; } pcie=5Fcap=5Fflr=5Finit(&d->dev, &pcie=5Fmsix=5Fflr); pcie=5Fcap=5Fdeverr=5Finit(&d->dev); pcie=5Fcap=5Fari=5Finit(&d->dev); rc =3D pcie=5Faer=5Finit(&d->dev, PCIE=5FMSIX=5FAER=5FOFFSET); if (rc < 0) { return rc; } PRINT=5FDEBUG("%s: Init done\n", =5F=5FFUNCTION=5F=5F); return 0; } Another question I have is why doesn't the device show up when I try a c= at /proc/interrupts. linux-an84:~/AriesKernelModules/gni/aries/ghal # cat /proc/interrupts CPU0 0: 694 IO-APIC-edge timer 1: 6 IO-APIC-edge i8042 4: 753 IO-APIC-edge serial 8: 1 IO-APIC-edge rtc0 9: 0 IO-APIC-fasteoi acpi 12: 89 IO-APIC-edge i8042 14: 3522 IO-APIC-edge ata=5Fpiix 15: 785 IO-APIC-edge ata=5Fpiix 16: 162 IO-APIC-fasteoi eth0 4344: 0 PCI-MSI-edge aerdrv 4345: 0 PCI-MSI-edge aerdrv 4346: 0 PCI-MSI-edge aerdrv 4347: 0 PCI-MSI-edge aerdrv 4348: 0 PCI-MSI-edge aerdrv 4349: 0 PCI-MSI-edge aerdrv 4350: 0 PCI-MSI-edge aerdrv 4351: 0 PCI-MSI-edge aerdrv NMI: 0 Non-maskable interrupts LOC: 107095 Local timer interrupts RES: 0 Rescheduling interrupts CAL: 0 function call interrupts TLB: 0 TLB shootdowns TRM: 0 Thermal event interrupts THR: 0 Threshold APIC interrupts SPU: 0 Spurious interrupts ERR: 0 Shouldn't there be an entry for the MSI-X device=3F Thanks for all your input. AK =5F=5F=5F=5F=5F =20 Probably what you want is something like =20 { 0x2000000000ull, PCI=5FBASE=5FADDRESS=5FSPACE=5FMEMORY | PCI=5FBASE= =5FADDRESS=5FMEM=5FTYPE=5F64} , //BAR0 { 0, 0} , //BAR1=20 // 64bit BAR occupies 2 BAR entries so that BAR1 can't be used. { 0x2000000ull, PCI=5FBASE=5FADDRESS=5FSPACE=5FIO } , //BAR2 { 0, 0} , //BAR3 // for MSI-X { 0, 0} , //BAR4 { 0, 0} //BAR5 =20 =20 =20 -------------9ff798cb7af029a1bad9e6189ec420e7 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
I've tried everything you mentioned and= I still get the same problem. The only thing that seems to avoid that i= ssue is if I reduce the aperture size from 0x2000000000ull to 0x2000000u= ll.

Here is the relevant section of code:

static const unsigned long long BAR=5FRegions[6][2= ] =3D
{
    // len , type
&nb= sp;   { 0x2000000ull, PCI=5FBASE=5FADDRESS=5FSPACE=5FMEMORY | = PCI=5FBASE=5FADDRESS=5FMEM=5FTYPE=5F64} ,  //BAR0,   = ;    
    { 0, 0} , // BAR1<= /span>
    { 0x2000000ull, PCI=5FBASE=5FADDRESS=5FSPACE= =5FIO    } ,  //BAR2,
    { 0= , 0} , // BAR3 for MSI-X
    { 0, 0} , // BAR4
    { 0, 0} , // BAR5   
};=

static int pcie=5Fmsix=5Fin= itfn(PCIDevice *pci=5Fdev)
{
    PCIE=5FM= SIX=5FDEVState *d =3D DO=5FUPCAST(PCIE=5FMSIX=5FDEVState, dev, pci=5Fdev= );
    PCIBridge *br =3D DO=5FUPCAST(PCIBridge, d= ev, pci=5Fdev);
    PCIEPort *p =3D DO=5FUPCAST(P= CIEPort, br, br);
    int rc, i;

    PRINT=5FDEBUG("%s: PCIE = MSIX Device init...\n", =5F=5FFUNCTION=5F=5F);

    pci=5Fconfig=5Fset=5Fvendor=5Fid(d= ->dev.config, PCIE=5FMSIX=5FVID);
    pci=5Fco= nfig=5Fset=5Fdevice=5Fid(d->dev.config, PCIE=5FMSIX=5FDID);
    memcpy(d->dev.c= onfig, g=5Fcfg=5Finit, sizeof(g=5Fcfg=5Finit[0x20]));
 &nbs= p;  d->mmio=5Findex =3D cpu=5Fregister=5Fio=5Fmemory(pcie=5Fmsix= =5Fmem=5Fread=5Ffn, pcie=5Fmsix=5Fmem=5Fwrite=5Ffn, d);

&nbs= p;   int msix=5Fmem=5Fbar  =3D 0; // Since its a 64bit BA= R, we take up BAR0 & BAR1
=     int msix=5Fio= =5Fbar   =3D 2;
    int msix=5Fmmio=5Fb= ar =3D 3;

   = pci=5Fregister=5Fbar(&d->dev, msix=5Fmem=5Fbar, BAR=5FRegions[ms= ix=5Fmem=5Fbar][0], BAR=5FRegions[msix=5Fmem=5Fbar][1], pcie=5Fmsix=5Fme= m=5Fmap);
    pci=5Fregister=5Fbar(&d->dev= , msix=5Fio=5Fbar, BAR=5FRegions[msix=5Fio=5Fbar][0], BAR=5FRegions[msix= =5Fio=5Fbar][1], pcie=5Fmsix=5Fio=5Fmap);

    rc =3D msix=5Finit(&d->dev, d-&g= t;vectors, msix=5Fmmio=5Fbar, 0);
   
&n= bsp;   if (!rc) {
      =   PRINT=5FDEBUG("%s: Registering Bar %i as I/O BAR\n", =5F=5FFUNCTI= ON=5F=5F, msix=5Fmmio=5Fbar);
=      &nbs= p;  pci=5Fregister=5Fbar(&d->dev, msix=5Fmmio=5Fbar, msix=5F= bar=5Fsize(&d->dev), PCI=5FBASE=5FADDRESS=5FSPACE=5FMEMORY, msix= =5Fmmio=5Fmap);
        PRINT= =5FDEBUG("%s: MSI-X initialized (%d vectors)\n", =5F=5FFUNCTION=5F=5F, d= ->vectors);
    }
    e= lse {
        PRINT=5FDEBUG("= %s: MSI-X initialization failed!\n", =5F=5FFUNCTION=5F=5F);
&nbs= p;       return rc;
  &n= bsp; }
   
    // Activat= e the vectors
    for (i =3D 0; i < d->vect= ors; i++) {
        msix=5Fve= ctor=5Fuse(&d->dev, i);
    }

    rc =3D pci=5Fpcie=5Fc= ap=5Finit(&d->dev, PCIE=5FMSIX=5FEXP=5FOFFSET, PCI=5FEXP=5FTYPE= =5FENDPOINT, p->port);
    if (rc < 0) {
        return rc;
&nbs= p;   }

  = ;  pcie=5Fcap=5Fflr=5Finit(&d->dev, &pcie=5Fmsix=5Fflr);=
    pcie=5Fcap=5Fdeverr=5Finit(&d->dev);<= /span>
    pcie=5Fcap=5Fari=5Finit(&d->dev);
    rc =3D pcie=5Faer=5Finit(&d->dev, PCIE=5FM= SIX=5FAER=5FOFFSET);
    if (rc < 0) {<= br style=3D"font-family: Courier;">        return rc;
 &= nbsp;  }

  &n= bsp; PRINT=5FDEBUG("%s: Init done\n", =5F=5FFUNCTION=5F=5F);
&nb= sp;   return 0;
}

Another question I hav= e is why doesn't the device show up when I try a cat /proc/interrupts.
linux-an84:~/AriesKernelModules/gni/aries/ghal # cat /proc= /interrupts
        &nbs= p;  CPU0
  0:       = ; 694   IO-APIC-edge      timer
  1:          6&nbs= p;  IO-APIC-edge      i8042
 = 4:        753   IO-APIC-ed= ge      serial
  8:  &nb= sp;       1   IO-APIC-edge =      rtc0
=   9:    &= nbsp;     0   IO-APIC-fasteoi   = acpi
 12:         8= 9   IO-APIC-edge      i8042&= nbsp;14:       3522   IO-APIC-ed= ge      ata=5Fpiix
 15:  = ;      785   IO-APIC-edge  =     ata=5Fpiix
=  16:    &= nbsp;   162   IO-APIC-fasteoi   eth0
4344:          0 &n= bsp; PCI-MSI-edge      aerdrv
4345:&n= bsp;         0   PCI-M= SI-edge      aerdrv
4346:  &n= bsp;       0   PCI-MSI-edge = ;     aerdrv
4347:    &n= bsp;     0   PCI-MSI-edge   = ;   aerdrv
4348:      &n= bsp;   0   PCI-MSI-edge     = ; aerdrv
4349:        &n= bsp; 0   PCI-MSI-edge      aerdrv
4350:          0 =   PCI-MSI-edge      aerdrv
4351:&n= bsp;         0   PCI-M= SI-edge      aerdrv
NMI:  &nb= sp;       0   Non-maskable inter= rupts
LOC:     107095   Local time= r interrupts
RES:        = ;  0   Rescheduling interrupts
CAL:  &n= bsp;       0   function call int= errupts
TLB:        &nbs= p; 0   TLB shootdowns
TRM:    &nbs= p;     0   Thermal event interrupts=
<= span style=3D"font-family: Courier;">SPU:     &= nbsp;    0   Spurious interrupts
ERR:&n= bsp;         0

Sho= uldn't there be an entry for the MSI-X device=3F

Thanks for all y= our input.

AK


Probably what you want is something like

{ 0x2000000000ull, PCI=5FBASE=5FADDRESS=5FSPACE=5FMEMORY | PCI=5FBASE=5F= ADDRESS=5FMEM=5FTYPE=5F64} , //BAR0
{ 0, 0} , //BAR1=09
=09// 64bit BAR occupies 2 BAR entries so that BAR1 can't be used.
{ 0x2000000ull, PCI=5FBASE=5FADDRESS=5FSPACE=5FIO } , //BAR2
{ 0, 0} , //BAR3
=09// for MSI-X
{ 0, 0} , //BAR4
{ 0, 0} //BAR5


-------------9ff798cb7af029a1bad9e6189ec420e7-- From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from [140.186.70.92] (port=37950 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OrIiG-0003TB-V8 for qemu-devel@nongnu.org; Thu, 02 Sep 2010 18:56:59 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OrIiE-0005Mx-Jm for qemu-devel@nongnu.org; Thu, 02 Sep 2010 18:56:56 -0400 Received: from smtp.mailix.net ([66.11.225.183]:56846) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OrIiE-0005Md-3h for qemu-devel@nongnu.org; Thu, 02 Sep 2010 18:56:54 -0400 From: "Adnan Khaleel" Subject: =?iso-8859-1?Q?Re=3A_=5BQemu-devel=5D_Template_for_developing_a_Qe?= =?iso-8859-1?Q?mu_device_with_PCIe=09and_MSI-X?= In-Reply-To: 20100902023831.GA20106@valinux.co.jp Message-ID: <20100902225648.c36436e0@shadowfax.no-ip.com> Date: Thu, 02 Sep 2010 17:56:48 -0500 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="-----------b7433c2fe550da2c0f6e0df25183cf8f" Reply-To: adnan@khaleel.us List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Isaku Yamahata Cc: qemu-devel@nongnu.org This is a multi-part message in MIME format. -------------b7433c2fe550da2c0f6e0df25183cf8f Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable During bootup, I also see the following error message: pci 0000:40:00.0: BAR 0 bad alignment 2000000000: 0x00000000000000-0x000= 01fffffffff Any idea whats causing this=3F AK =5F=5F=5F=5F=5F =20 From: Isaku Yamahata [mailto:yamahata@valinux.co.jp] To: Adnan Khaleel [mailto:adnan@khaleel.us] Cc: Cam Macdonell [mailto:cam@cs.ualberta.ca], qemu-devel@nongnu.org Sent: Wed, 01 Sep 2010 21:38:31 -0500 Subject: Re: [Qemu-devel] Template for developing a Qemu device with PCI= e and MSI-X On Wed, Sep 01, 2010 at 02:07:33PM -0500, Adnan Khaleel wrote: > Yamahata, Cam, >=20 > Thank you both very much for pointers about Qemu coding for PCIe and= MSI-X. >=20 > I'm at a point where I can see my device when I do an lspci -t -v as= shown > below. >=20 > linux-an84:~ # lspci -t -v > -[0000:00]-+-00.0 Intel Corporation 82G33/G31/P35/P31 Express DRAM = Controller > +-01.0 Cirrus Logic GD 5446 > +-04.0-[0000:20]-- > +-18.0-[0000:21]-- > +-18.1-[0000:22]-- > +-18.2-[0000:23]-- > +-18.3-[0000:24]-- > +-18.4-[0000:25]-- > +-18.5-[0000:26]-- > +-19.0-[0000:36-bf]--+-00.0-[0000:37-47]--+-00.0-[0000:38= ]-- > | | +-01.0-[0000:39= ]-- > | | +-02.0-[0000:3a= ]-- > | | +-03.0-[0000:3b= ]-- > | | +-04.0-[0000:3c= ]-- > | | +-05.0-[0000:3d= ]-- > | | +-06.0-[0000:3e= ]-- > | | +-07.0-[0000:3f= ]-- > | | +-08.0-[0000:40= ]----00.0=20 > Cray Inc Device 0301 <- The device that I've included >=20 > However, I'm having a bit of an issue with the MSI-X. >=20 > I'm following the code examples in virtio-pci.c and ivshmem.c that C= am pointed > out to. I've got bar 0&1 already occupied so I assign the msix=5Fmmi= o=5Fmap to bar > 2. However, when I do that, Qemu fails to boot and fails with the fo= llowing > assertion fail: >=20 > unused outb: port=3D0x00f1 data=3D0x00 > qemu: fatal: Trying to execute code outside RAM or ROM at 0x00000000= 00100000 >=20 > Couple of thins I'm unsure about: > 1. Am I registering the 64bit bar addresses correctly=3F > pci=5Fregister=5Fbar(&d->dev, i, BAR=5FRegions[i][0], PCI=5FBASE=5FA= DDRESS=5FSPACE=5FIO | > PCI=5FBASE=5FADDRESS=5FMEM=5FTYPE=5F64, pcie=5Fmsix=5Fio=5Fmap); =20 No. PCI=5FBASE=5FADDRESS=5FSPACE=5FIO | PCI=5FBASE=5FADDRESS=5FMEM=5FT= YPE=5F64 is invalid. PCI doesn't support 64bit io space, but 32bit. And x86 supports 64k io sparce. =20 =20 > 2. In the function int msix=5Finit(PCIDevice *pdev, unsigned short n= entries, > unsigned bar=5Fnr, unsigned bar=5Fsize); > I'm not sure what bar=5Fnr is, however setting it to 1 (as in code e= xamples > above) or to 2 (bar that I want to register the msix=5Fmmio=5Fmap to= ) both fail > with the same error. =20 It should be 2. I'm not sure why it failed. =20 =20 > rc =3D msix=5Finit(&d->dev, d->vectors, 2, 0); > : > : > pci=5Fregister=5Fbar(&d->dev, 2, msix=5Fbar=5Fsize(&d->dev), > PCI=5FBASE=5FADDRESS=5FSPACE=5FMEMORY | PCI=5FBASE=5FADDRESS=5FMEM= =5FTYPE=5F64, msix=5Fmmio=5Fmap); >=20 >=20 > Here is my init function code in its entirety: >=20 > static const unsigned long long BAR=5FRegions[6][2] =3D > { > // len , type > { 0x2000000000ull, PCI=5FBASE=5FADDRESS=5FSPACE=5FMEMORY} , //B= AR0 > { 0x2000000ull, PCI=5FBASE=5FADDRESS=5FSPACE=5FIO } , //B= AR1 > { 0, 0} , //BAR2 > { 0, 0} , //BAR3 > { 0, 0} , //BAR4 > { 0, 0} //BAR5 > }; =20 Probably what you want is something like =20 { 0x2000000000ull, PCI=5FBASE=5FADDRESS=5FSPACE=5FMEMORY | PCI=5FBASE= =5FADDRESS=5FMEM=5FTYPE=5F64} , //BAR0 { 0, 0} , //BAR1=20 // 64bit BAR occupies 2 BAR entries so that BAR1 can't be used. { 0x2000000ull, PCI=5FBASE=5FADDRESS=5FSPACE=5FIO } , //BAR2 { 0, 0} , //BAR3 // for MSI-X { 0, 0} , //BAR4 { 0, 0} //BAR5 =20 >=20 > static int pcie=5Fmsix=5Finitfn(PCIDevice *pci=5Fdev) > { > PCIE=5FMSIX=5FDEVState *d =3D DO=5FUPCAST(PCIE=5FMSIX=5FDEVState= , dev, pci=5Fdev); > PCIBridge *br =3D DO=5FUPCAST(PCIBridge, dev, pci=5Fdev); > PCIEPort *p =3D DO=5FUPCAST(PCIEPort, br, br); > int rc, i; >=20 > PRINT=5FDEBUG("%s: PCIE MSIX Device init...\n", =5F=5FFUNCTION= =5F=5F); >=20 > pci=5Fconfig=5Fset=5Fvendor=5Fid(d->dev.config, PCIE=5FMSIX=5FVI= D); > pci=5Fconfig=5Fset=5Fdevice=5Fid(d->dev.config, PCIE=5FMSIX=5FDI= D); > d->dev.config[PCI=5FREVISION=5FID] =3D PCIE=5FMSIX=5FVERSION; > d->dev.config[PCI=5FSUBSYSTEM=5FVENDOR=5FID] =3D PCIE=5FMSIX= =5FVID & 0xff; > d->dev.config[PCI=5FSUBSYSTEM=5FVENDOR=5FID+1] =3D (PCIE=5FMSIX= =5FVID >> 8) & 0xff; > d->dev.config[PCI=5FSUBSYSTEM=5FID] =3D PCIE=5FMSIX=5FSS=5FDID= & 0xff; > d->dev.config[PCI=5FSUBSYSTEM=5FID+1] =3D (PCIE=5FMSIX=5FSS=5FDI= D >> 8) & 0xff; =20 Use pci=5Fset=5Fword(). =20 > d->mmio=5Findex =3D cpu=5Fregister=5Fio=5Fmemory(pcie=5Fmsix=5Fm= em=5Fread=5Ffn, > pcie=5Fmsix=5Fmem=5Fwrite=5Ffn, d); > =20 > for(i=3D0; i < PCI=5FNUM=5FREGIONS -1; i++) { //-1 for the Exp R= OM BAR > if(BAR=5FRegions[i][0] !=3D 0) > { > if(BAR=5FRegions[i][1] =3D=3D PCI=5FBASE=5FADDRESS=5FSPA= CE=5FIO) > { > //io region > PRINT=5FDEBUG("%s: Registering Bar %i as I/O BAR\n", > =5F=5FFUNCTION=5F=5F, i); > pci=5Fregister=5Fbar(&d->dev, i, BAR=5FRegions[i][0]= , > PCI=5FBASE=5FADDRESS=5FSPACE=5FIO | PCI=5FBASE=5FADDRESS=5FMEM=5FTYP= E=5F64, pcie=5Fmsix=5Fio=5Fmap); > } else { > //mem region > PRINT=5FDEBUG("%s: Registering Bar %i as MEM BAR\n", > =5F=5FFUNCTION=5F=5F, i); > pci=5Fregister=5Fbar(&d->dev, i, BAR=5FRegions[i][0]= , > PCI=5FBASE=5FADDRESS=5FSPACE=5FMEMORY | PCI=5FBASE=5FADDRESS=5FMEM= =5FTYPE=5F64, > pcie=5Fmsix=5Fmem=5Fmap); > } > } > } > d->dev.config[PCI=5FINTERRUPT=5FPIN] =3D 1; >=20 > rc =3D msix=5Finit(&d->dev, d->vectors, 2, 0); > =20 > if (!rc) { > PRINT=5FDEBUG("%s: Registering Bar %i as I/O BAR\n", =5F=5FF= UNCTION=5F=5F, i); > pci=5Fregister=5Fbar(&d->dev, 2, msix=5Fbar=5Fsize(&d->dev), > PCI=5FBASE=5FADDRESS=5FSPACE=5FMEMORY | PCI=5FBASE=5FADDRESS=5FMEM= =5FTYPE=5F64, msix=5Fmmio=5Fmap); > PRINT=5FDEBUG("%s: MSI-X initialized (%d vectors)\n", =5F=5F= FUNCTION=5F=5F, d-> > vectors); > } > else { > PRINT=5FDEBUG("%s: MSI-X initialization failed!\n", =5F=5FFU= NCTION=5F=5F); > exit(1); > } > =20 > // Activate the vectors > for (i =3D 0; i < d->vectors; i++) { > msix=5Fvector=5Fuse(&d->dev, i); > } >=20 > rc =3D pci=5Fpcie=5Fcap=5Finit(&d->dev, PCIE=5FMSIX=5FEXP=5FOFFS= ET, > PCI=5FEXP=5FTYPE=5FENDPOINT, p->port); > if (rc < 0) { > return rc; > } >=20 > pcie=5Fcap=5Fflr=5Finit(&d->dev, &pcie=5Fmsix=5Fflr); > pcie=5Fcap=5Fdeverr=5Finit(&d->dev); > pcie=5Fcap=5Fari=5Finit(&d->dev); > rc =3D pcie=5Faer=5Finit(&d->dev, PCIE=5FMSIX=5FAER=5FOFFSET); > if (rc < 0) { > return rc; > } >=20 > PRINT=5FDEBUG("%s: Init done\n", =5F=5FFUNCTION=5F=5F); > return 0; > } >=20 >=20 > Thanks >=20 > AK >=20 >=20 >=20 >=20 > =E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81= =E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81= =E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81= =E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81= =E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81 > From: Cam Macdonell [mailto:cam@cs.ualberta.ca] > To: adnan@khaleel.us > Cc: Isaku Yamahata [mailto:yamahata@valinux.co.jp], qemu-devel@n= ongnu.org > Sent: Fri, 27 Aug 2010 10:48:48 -0500 > Subject: Re: [Qemu-devel] Template for developing a Qemu device = with PCIe > and MSI-X >=20 > On Wed, Aug 25, 2010 at 4:39 PM, Adnan Khaleel wrote: > > Hi Isaku, > > > > I've made some progress in coding the device template but its = no where > near > > complete. > > > > I've created some files and am attaching it to this note. Base= d on what I > > could gather from the pcie source files I've made a stab at cr= eating a > > simple model. I've also attached a file for a simple pci devic= e that > works > > under regular Qemu. I would like to duplicate its functionalit= y in your > pcie > > environment for starters. > > > > Could you please take a look at the files I've created and tel= l me if > I've > > understood your pcie model correctly. Any help will be truly a= ppreciated. > > > > Adnan >=20 > Hi Adnan, >=20 > There is a fairly simple device I've created called "ivshmem" th= at is > in the qemu git tree. It is a regular PCI device that exports a > shared memory object via a BAR and supports a few registers and > optional MSI-X interrupts (I had to pick through the virtio code= to > get MSI-X working, so looking at ivshmem might save you some eff= ort). > My device is somewhat similar to a graphics card actually which = I > recall is your goal. The purpose of ivshmem is to support sharin= g > memory between multiple guests running on the same host. It foll= ows > the qdev model which you will need to do. >=20 > Cam >=20 > > > > The five files I've modified from your git repository are as f= ollows > > > > hw/pci=5Fids.h // Added vendor id defines > > hw/pc=5Fq35.c // Device instantiation > > hw/pcie=5Fmsix=5Ftemplate.h // Device header file > > hw/pcie=5Fmsix=5Ftemplate.c // Device file > > Makefile.objs // Added pcie=5Fmsix=5Ftemplat= e.o to list of > > objects being built > > > > Everything should compile without any warnings or errors. > > > > The last file: > > sc=5Flink=5Fpci.c > > Is the original PCI device that I'm trying to convert into bei= ng PCIe and > > MSI-X and is included merely for reference to help you underst= and what > I'd > > like to achieve in your environment. > > > > > > =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F > > From: Isaku Yamahata [mailto:yamahata@valinux.co.jp] > > To: Adnan Khaleel [mailto:adnan@khaleel.us] > > Cc: qemu-devel@nongnu.org > > Sent: Wed, 18 Aug 2010 22:19:04 -0500 > > Subject: Re: [Qemu-devel] Template for developing a Qemu devic= e with PCIe > > and MSI-X > > > > On Wed, Aug 18, 2010 at 02:10:10PM -0500, Adnan Khaleel wrote: > >> Hello Qemu developers, > >> > >> I'm interested in developing a device model that plugs into Q= emu that is > >> based > >> on a PCIe interface and uses MSI-X. My goal is to ultimately = attach a > GPU > >> simulator to this PCIe interface and use the entire platfom (= Qemu + GPU > >> simulator) for studying cpu, gpu interactions. > >> > >> I'm not terribly familiar with the Qemu device model and I'm = looking for > >> some > >> assistance, perhaps a starting template for pcie and msi-x th= at would > >> offer the > >> basic functionality that I could then build upon. > >> > >> I have looked at the various devices that already modelled th= at are > >> included > >> with Qemu (v0.12.5 at least) and I've noticed several a few p= ci devices, > >> eg; > >> ne2k and cirrus-pci etc, however only one device truly seems = to utilize > >> both > >> the technologies that I'm interested in and that is the virti= o-pci.c > >> > >> I'm not sure what virtio-pci does so I'm not sure if that is = a suitable > >> starting point for me. > >> > >> Any help, suggestions etc would be extremely helpful and much > appreciated. > > > > Qemu doesn't support pcie at the moment. > > Only partial patches have been merged, still more patches have= to > > be merged for pcie to fully work. The following repo is availa= ble. > > > > git clone http://people.valinux.co.jp/~yamahata/qemu/q35/qemu > > git clone http://people.valinux.co.jp/~yamahata/qemu/q35/seabi= os > > git clone http://people.valinux.co.jp/~yamahata/qemu/q35/vgabi= os > > > > Note: patched seabios and vgabios are needed, you have to pass= ACPI DSDT > > for q35. > > example: > > qemu-system-x86=5F64 -M pc=5Fq35 -acpitable > > load=5Fheader,data=3Droms/seabios/src/q35-acpi-dsdt.aml > > > > This repo is for those who want to try/develop pcie support, > > not for upstream merge. So they include patches unsuitable for= upstream. > > The repo includes pcie port switch emulator which utilize pcie= and > > MSI(not MSI-X). > > > > The difference between PCI device and PCIe device is configura= tion > > space size. > > By setting PCIDeviceInfo::is=5Fexpress =3D 1, you'll get 4K co= nfiguration > > space. Helper functions for pcie are found in qemu/hw/pcie.c > > For msi-x, see qemu/hw/msix.c. > > > > Thanks, > > -- > > yamahata > > >=20 =20 --=20 yamahata =20 -------------b7433c2fe550da2c0f6e0df25183cf8f Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable
During bootup, I also see the following= error message:

pci 0000:40= :00.0: BAR 0 bad alignment 2000000000: 0x00000000000000-0x00001fffffffff=

Any idea whats causing this=3F

AK

From: Isaku Yamahat= a [mailto:yamahata@valinux.co.jp]
To: Adnan Khaleel [mailto:ad= nan@khaleel.us]
Cc: Cam Macdonell [mailto:cam@cs.ualberta.ca],= qemu-devel@nongnu.org
Sent: Wed, 01 Sep 2010 21:38:31 -0500Subject: Re: [Qemu-devel] Template for developing a Qemu device= with PCIe=09and MSI-X

On Wed, Sep 01, 2010 at 02:07:33PM -0500, = Adnan Khaleel wrote:
> Yamahata, Cam,
>
> Thank you both very much for pointers about Qemu coding for PCIe an= d MSI-X.
>
> I'm at a point where I can see my device when I do an lspci -t -v a= s shown
> below.
>
> linux-an84:~ # lspci -t -v
> -[0000:00]-+-00.0 Intel Corporation 82G33/G31/P35/P31 Express DRAM= Controller
> +-01.0 Cirrus Logic GD 5446
> +-04.0-[0000:20]--
> +-18.0-[0000:21]--
> +-18.1-[0000:22]--
> +-18.2-[0000:23]--
> +-18.3-[0000:24]--
> +-18.4-[0000:25]--
> +-18.5-[0000:26]--
> +-19.0-[0000:36-bf]--+-00.0-[0000:37-47]--+-00.0-[0000:3= 8]--
> | | +-01.0-[0000:3= 9]--
> | | +-02.0-[0000:3= a]--
> | | +-03.0-[0000:3= b]--
> | | +-04.0-[0000:3= c]--
> | | +-05.0-[0000:3= d]--
> | | +-06.0-[0000:3= e]--
> | | +-07.0-[0000:3= f]--
> | | +-08.0-[0000:4= 0]----00.0
> Cray Inc Device 0301 <- The device that I've included
>
> However, I'm having a bit of an issue with the MSI-X.
>
> I'm following the code examples in virtio-pci.c and ivshmem.c that = Cam pointed
> out to. I've got bar 0&1 already occupied so I assign the msix= =5Fmmio=5Fmap to bar
> 2. However, when I do that, Qemu fails to boot and fails with the f= ollowing
> assertion fail:
>
> unused outb: port=3D0x00f1 data=3D0x00
> qemu: fatal: Trying to execute code outside RAM or ROM at 0x0000000= 000100000
>
> Couple of thins I'm unsure about:
> 1. Am I registering the 64bit bar addresses correctly=3F
> pci=5Fregister=5Fbar(&d->dev, i, BAR=5FRegions[i][0], PCI=5F= BASE=5FADDRESS=5FSPACE=5FIO |
> PCI=5FBASE=5FADDRESS=5FMEM=5FTYPE=5F64, pcie=5Fmsix=5Fio=5Fmap);
No. PCI=5FBASE=5FADDRESS=5FSPACE=5FIO | PCI=5FBASE=5FADDRESS=5FMEM=5FTYP= E=5F64 is invalid.
PCI doesn't support 64bit io space, but 32bit. And x86 supports 64k
io sparce.


> 2. In the function int msix=5Finit(PCIDevice *pdev, unsigned short = nentries,
> unsigned bar=5Fnr, unsigned bar=5Fsize);
> I'm not sure what bar=5Fnr is, however setting it to 1 (as in code = examples
> above) or to 2 (bar that I want to register the msix=5Fmmio=5Fmap t= o) both fail
> with the same error.

It should be 2. I'm not sure why it failed.


> rc =3D msix=5Finit(&d->dev, d->vectors, 2, 0);
> :
> :
> pci=5Fregister=5Fbar(&d->dev, 2, msix=5Fbar=5Fsize(&d-&g= t;dev),
> PCI=5FBASE=5FADDRESS=5FSPACE=5FMEMORY | PCI=5FBASE=5FADDRESS=5FMEM= =5FTYPE=5F64, msix=5Fmmio=5Fmap);
>
>
> Here is my init function code in its entirety:
>
> static const unsigned long long BAR=5FRegions[6][2] =3D
> {
> // len , type
> { 0x2000000000ull, PCI=5FBASE=5FADDRESS=5FSPACE=5FMEMORY} , //= BAR0
> { 0x2000000ull, PCI=5FBASE=5FADDRESS=5FSPACE=5FIO } , //= BAR1
> { 0, 0} , //BAR2
> { 0, 0} , //BAR3
> { 0, 0} , //BAR4
> { 0, 0} //BAR5
> };

Probably what you want is something like

{ 0x2000000000ull, PCI=5FBASE=5FADDRESS=5FSPACE=5FMEMORY | PCI=5FBASE=5F= ADDRESS=5FMEM=5FTYPE=5F64} , //BAR0
{ 0, 0} , //BAR1=09
=09// 64bit BAR occupies 2 BAR entries so that BAR1 can't be used.
{ 0x2000000ull, PCI=5FBASE=5FADDRESS=5FSPACE=5FIO } , //BAR2
{ 0, 0} , //BAR3
=09// for MSI-X
{ 0, 0} , //BAR4
{ 0, 0} //BAR5

>
> static int pcie=5Fmsix=5Finitfn(PCIDevice *pci=5Fdev)
> {
> PCIE=5FMSIX=5FDEVState *d =3D DO=5FUPCAST(PCIE=5FMSIX=5FDEVStat= e, dev, pci=5Fdev);
> PCIBridge *br =3D DO=5FUPCAST(PCIBridge, dev, pci=5Fdev);
> PCIEPort *p =3D DO=5FUPCAST(PCIEPort, br, br);
> int rc, i;
>
> PRINT=5FDEBUG("%s: PCIE MSIX Device init...\n", =5F=5FFUNCTION= =5F=5F);
>
> pci=5Fconfig=5Fset=5Fvendor=5Fid(d->dev.config, PCIE=5FMSIX= =5FVID);
> pci=5Fconfig=5Fset=5Fdevice=5Fid(d->dev.config, PCIE=5FMSIX= =5FDID);
> d->dev.config[PCI=5FREVISION=5FID] =3D PCIE=5FMSIX=5FVERSION= ;
> d->dev.config[PCI=5FSUBSYSTEM=5FVENDOR=5FID] =3D PCIE=5FMS= IX=5FVID & 0xff;
> d->dev.config[PCI=5FSUBSYSTEM=5FVENDOR=5FID+1] =3D (PCIE=5FM= SIX=5FVID >> 8) & 0xff;
> d->dev.config[PCI=5FSUBSYSTEM=5FID] =3D PCIE=5FMSIX=5FSS= =5FDID & 0xff;
> d->dev.config[PCI=5FSUBSYSTEM=5FID+1] =3D (PCIE=5FMSIX=5FSS= =5FDID >> 8) & 0xff;

Use pci=5Fset=5Fword().

> d->mmio=5Findex =3D cpu=5Fregister=5Fio=5Fmemory(pcie=5Fmsix= =5Fmem=5Fread=5Ffn,
> pcie=5Fmsix=5Fmem=5Fwrite=5Ffn, d);
>
> for(i=3D0; i < PCI=5FNUM=5FREGIONS -1; i++) { //-1 for the E= xp ROM BAR
> if(BAR=5FRegions[i][0] !=3D 0)
> {
> if(BAR=5FRegions[i][1] =3D=3D PCI=5FBASE=5FADDRESS=5FSP= ACE=5FIO)
> {
> //io region
> PRINT=5FDEBUG("%s: Registering Bar %i as I/O BAR\n"= ,
> =5F=5FFUNCTION=5F=5F, i);
> pci=5Fregister=5Fbar(&d->dev, i, BAR=5FRegio= ns[i][0],
> PCI=5FBASE=5FADDRESS=5FSPACE=5FIO | PCI=5FBASE=5FADDRESS=5FMEM=5FTY= PE=5F64, pcie=5Fmsix=5Fio=5Fmap);
> } else {
> //mem region
> PRINT=5FDEBUG("%s: Registering Bar %i as MEM BAR\n"= ,
> =5F=5FFUNCTION=5F=5F, i);
> pci=5Fregister=5Fbar(&d->dev, i, BAR=5FRegio= ns[i][0],
> PCI=5FBASE=5FADDRESS=5FSPACE=5FMEMORY | PCI=5FBASE=5FADDRESS=5FMEM= =5FTYPE=5F64,
> pcie=5Fmsix=5Fmem=5Fmap);
> }
> }
> }
> d->dev.config[PCI=5FINTERRUPT=5FPIN] =3D 1;
>
> rc =3D msix=5Finit(&d->dev, d->vectors, 2, 0);
>
> if (!rc) {
> PRINT=5FDEBUG("%s: Registering Bar %i as I/O BAR\n", =5F=5F= FUNCTION=5F=5F, i);
> pci=5Fregister=5Fbar(&d->dev, 2, msix=5Fbar=5Fsize(&= amp;d->dev),
> PCI=5FBASE=5FADDRESS=5FSPACE=5FMEMORY | PCI=5FBASE=5FADDRESS=5FMEM= =5FTYPE=5F64, msix=5Fmmio=5Fmap);
> PRINT=5FDEBUG("%s: MSI-X initialized (%d vectors)\n", =5F= =5FFUNCTION=5F=5F, d->
> vectors);
> }
> else {
> PRINT=5FDEBUG("%s: MSI-X initialization failed!\n", =5F=5FF= UNCTION=5F=5F);
> exit(1);
> }
>
> // Activate the vectors
> for (i =3D 0; i < d->vectors; i++) {
> msix=5Fvector=5Fuse(&d->dev, i);
> }
>
> rc =3D pci=5Fpcie=5Fcap=5Finit(&d->dev, PCIE=5FMSIX=5FEX= P=5FOFFSET,
> PCI=5FEXP=5FTYPE=5FENDPOINT, p->port);
> if (rc < 0) {
> return rc;
> }
>
> pcie=5Fcap=5Fflr=5Finit(&d->dev, &pcie=5Fmsix=5Fflr)= ;
> pcie=5Fcap=5Fdeverr=5Finit(&d->dev);
> pcie=5Fcap=5Fari=5Finit(&d->dev);
> rc =3D pcie=5Faer=5Finit(&d->dev, PCIE=5FMSIX=5FAER=5FOF= FSET);
> if (rc < 0) {
> return rc;
> }
>
> PRINT=5FDEBUG("%s: Init done\n", =5F=5FFUNCTION=5F=5F);
> return 0;
> }
>
>
> Thanks
>
> AK
>
>
>
>
> =E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81= =E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81= =E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81= =E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81= =E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81=E2=94=81
> From: Cam Macdonell [mailto:cam@cs.ualberta.ca]
> To: adnan@khaleel.us > Cc: Isaku Yamahata [mailto:yamahata@valinux.co.jp], qemu-devel@nongnu.org
> Sent: Fri, 27 Aug 2010 10:48:48 -0500
> Subject: Re: [Qemu-devel] Template for developing a Qemu device= with PCIe
> and MSI-X
>
> On Wed, Aug 25, 2010 at 4:39 PM, Adnan Khaleel <adnan@khaleel.us> wrote:
> > Hi Isaku,
> >
> > I've made some progress in coding the device template but = its no where
> near
> > complete.
> >
> > I've created some files and am attaching it to this note. = Based on what I
> > could gather from the pcie source files I've made a stab a= t creating a
> > simple model. I've also attached a file for a simple pci d= evice that
> works
> > under regular Qemu. I would like to duplicate its function= ality in your
> pcie
> > environment for starters.
> >
> > Could you please take a look at the files I've created and= tell me if
> I've
> > understood your pcie model correctly. Any help will be tru= ly appreciated.
> >
> > Adnan
>
> Hi Adnan,
>
> There is a fairly simple device I've created called "ivshmem" t= hat is
> in the qemu git tree. It is a regular PCI device that exports a=
> shared memory object via a BAR and supports a few registers and=
> optional MSI-X interrupts (I had to pick through the virtio cod= e to
> get MSI-X working, so looking at ivshmem might save you some ef= fort).
> My device is somewhat similar to a graphics card actually which= I
> recall is your goal. The purpose of ivshmem is to support shari= ng
> memory between multiple guests running on the same host. It fol= lows
> the qdev model which you will need to do.
>
> Cam
>
> >
> > The five files I've modified from your git repository are = as follows
> >
> > hw/pci=5Fids.h // Added vendor id defin= es
> > hw/pc=5Fq35.c // Device instantiation > > hw/pcie=5Fmsix=5Ftemplate.h // Device header file
> > hw/pcie=5Fmsix=5Ftemplate.c // Device file
> > Makefile.objs // Added pcie=5Fmsix=5Ftem= plate.o to list of
> > objects being built
> >
> > Everything should compile without any warnings or errors.<= br> > >
> > The last file:
> > sc=5Flink=5Fpci.c
> > Is the original PCI device that I'm trying to convert into= being PCIe and
> > MSI-X and is included merely for reference to help you und= erstand what
> I'd
> > like to achieve in your environment.
> >
> >
> > =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F
> > From: Isaku Yamahata [mailto:yamahata@valinux.co.jp]
> > To: Adnan Khaleel [mailto:adnan@khaleel.us]
> > Cc: qemu-devel@no= ngnu.org
> > Sent: Wed, 18 Aug 2010 22:19:04 -0500
> > Subject: Re: [Qemu-devel] Template for developing a Qemu d= evice with PCIe
> > and MSI-X
> >
> > On Wed, Aug 18, 2010 at 02:10:10PM -0500, Adnan Khaleel wr= ote:
> >> Hello Qemu developers,
> >>
> >> I'm interested in developing a device model that plugs= into Qemu that is
> >> based
> >> on a PCIe interface and uses MSI-X. My goal is to ulti= mately attach a
> GPU
> >> simulator to this PCIe interface and use the entire pl= atfom (Qemu + GPU
> >> simulator) for studying cpu, gpu interactions.
> >>
> >> I'm not terribly familiar with the Qemu device model a= nd I'm looking for
> >> some
> >> assistance, perhaps a starting template for pcie and m= si-x that would
> >> offer the
> >> basic functionality that I could then build upon.
> >>
> >> I have looked at the various devices that already mode= lled that are
> >> included
> >> with Qemu (v0.12.5 at least) and I've noticed several = a few pci devices,
> >> eg;
> >> ne2k and cirrus-pci etc, however only one device truly= seems to utilize
> >> both
> >> the technologies that I'm interested in and that is th= e virtio-pci.c
> >>
> >> I'm not sure what virtio-pci does so I'm not sure if t= hat is a suitable
> >> starting point for me.
> >>
> >> Any help, suggestions etc would be extremely helpful a= nd much
> appreciated.
> >
> > Qemu doesn't support pcie at the moment.
> > Only partial patches have been merged, still more patches = have to
> > be merged for pcie to fully work. The following repo is av= ailable.
> >
> > git clone http://people.valinux.co.jp/~yamah= ata/qemu/q35/qemu
> > git clone http://people.valinux.co.jp/~ya= mahata/qemu/q35/seabios
> > git clone http://people.valinux.co.jp/~ya= mahata/qemu/q35/vgabios
> >
> > Note: patched seabios and vgabios are needed, you have to = pass ACPI DSDT
> > for q35.
> > example:
> > qemu-system-x86=5F64 -M pc=5Fq35 -acpitable
> > load=5Fheader,data=3Droms/seabios/src/q35-acpi-dsdt.aml > >
> > This repo is for those who want to try/develop pcie suppor= t,
> > not for upstream merge. So they include patches unsuitable= for upstream.
> > The repo includes pcie port switch emulator which utilize = pcie and
> > MSI(not MSI-X).
> >
> > The difference between PCI device and PCIe device is confi= guration
> > space size.
> > By setting PCIDeviceInfo::is=5Fexpress =3D 1, you'll get 4= K configuration
> > space. Helper functions for pcie are found in qemu/hw/pcie= .c
> > For msi-x, see qemu/hw/msix.c.
> >
> > Thanks,
> > --
> > yamahata
> >
>

--
yamahata
-------------b7433c2fe550da2c0f6e0df25183cf8f-- From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from [140.186.70.92] (port=47470 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OrLgg-0007QG-KR for qemu-devel@nongnu.org; Thu, 02 Sep 2010 22:07:31 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OrLgf-0004lF-83 for qemu-devel@nongnu.org; Thu, 02 Sep 2010 22:07:30 -0400 Received: from mail.valinux.co.jp ([210.128.90.3]:46307) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OrLge-0004kr-OJ for qemu-devel@nongnu.org; Thu, 02 Sep 2010 22:07:29 -0400 Date: Fri, 3 Sep 2010 11:20:12 +0900 From: Isaku Yamahata Subject: Re: [Qemu-devel] Template for developing a Qemu device with PCIe?and MSI-X Message-ID: <20100903022012.GA25721@valinux.co.jp> References: <20100902174242.4cb1fb0f@shadowfax.no-ip.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-2022-jp Content-Disposition: inline In-Reply-To: <20100902174242.4cb1fb0f@shadowfax.no-ip.com> List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Adnan Khaleel Cc: Cam Macdonell , qemu-devel@nongnu.org On Thu, Sep 02, 2010 at 12:42:42PM -0500, Adnan Khaleel wrote: > I've tried everything you mentioned and I still get the same problem. The only > thing that seems to avoid that issue is if I reduce the aperture size from > 0x2000000000ull to 0x2000000ull. I suppose that Cam is seeing the same issue. Right now seabios can't handle too huge BAR due to overflow. There is a rejected patch floating around, but I haven't created a revised patch yet. > > Here is the relevant section of code: > > static const unsigned long long BAR_Regions[6][2] = > { > // len , type > { 0x2000000ull, PCI_BASE_ADDRESS_SPACE_MEMORY | > PCI_BASE_ADDRESS_MEM_TYPE_64} , //BAR0, > { 0, 0} , // BAR1 > { 0x2000000ull, PCI_BASE_ADDRESS_SPACE_IO } , //BAR2, > { 0, 0} , // BAR3 for MSI-X > { 0, 0} , // BAR4 > { 0, 0} , // BAR5 > }; > > static int pcie_msix_initfn(PCIDevice *pci_dev) > { > PCIE_MSIX_DEVState *d = DO_UPCAST(PCIE_MSIX_DEVState, dev, pci_dev); > PCIBridge *br = DO_UPCAST(PCIBridge, dev, pci_dev); > PCIEPort *p = DO_UPCAST(PCIEPort, br, br); > int rc, i; > > PRINT_DEBUG("%s: PCIE MSIX Device init...\n", __FUNCTION__); > > pci_config_set_vendor_id(d->dev.config, PCIE_MSIX_VID); > pci_config_set_device_id(d->dev.config, PCIE_MSIX_DID); > > memcpy(d->dev.config, g_cfg_init, sizeof(g_cfg_init[0x20])); > d->mmio_index = cpu_register_io_memory(pcie_msix_mem_read_fn, > pcie_msix_mem_write_fn, d); > > int msix_mem_bar = 0; // Since its a 64bit BAR, we take up BAR0 & BAR1 > int msix_io_bar = 2; > int msix_mmio_bar = 3; > > pci_register_bar(&d->dev, msix_mem_bar, BAR_Regions[msix_mem_bar][0], > BAR_Regions[msix_mem_bar][1], pcie_msix_mem_map); > pci_register_bar(&d->dev, msix_io_bar, BAR_Regions[msix_io_bar][0], > BAR_Regions[msix_io_bar][1], pcie_msix_io_map); > > rc = msix_init(&d->dev, d->vectors, msix_mmio_bar, 0); > > if (!rc) { > PRINT_DEBUG("%s: Registering Bar %i as I/O BAR\n", __FUNCTION__, > msix_mmio_bar); > pci_register_bar(&d->dev, msix_mmio_bar, msix_bar_size(&d->dev), > PCI_BASE_ADDRESS_SPACE_MEMORY, msix_mmio_map); > PRINT_DEBUG("%s: MSI-X initialized (%d vectors)\n", __FUNCTION__, d-> > vectors); > } > else { > PRINT_DEBUG("%s: MSI-X initialization failed!\n", __FUNCTION__); > return rc; > } > > // Activate the vectors > for (i = 0; i < d->vectors; i++) { > msix_vector_use(&d->dev, i); > } > > rc = pci_pcie_cap_init(&d->dev, PCIE_MSIX_EXP_OFFSET, > PCI_EXP_TYPE_ENDPOINT, p->port); > if (rc < 0) { > return rc; > } > > pcie_cap_flr_init(&d->dev, &pcie_msix_flr); > pcie_cap_deverr_init(&d->dev); > pcie_cap_ari_init(&d->dev); > rc = pcie_aer_init(&d->dev, PCIE_MSIX_AER_OFFSET); > if (rc < 0) { > return rc; > } > > PRINT_DEBUG("%s: Init done\n", __FUNCTION__); > return 0; > } > > Another question I have is why doesn't the device show up when I try a cat / > proc/interrupts. > > linux-an84:~/AriesKernelModules/gni/aries/ghal # cat /proc/interrupts > CPU0 > 0: 694 IO-APIC-edge timer > 1: 6 IO-APIC-edge i8042 > 4: 753 IO-APIC-edge serial > 8: 1 IO-APIC-edge rtc0 > 9: 0 IO-APIC-fasteoi acpi > 12: 89 IO-APIC-edge i8042 > 14: 3522 IO-APIC-edge ata_piix > 15: 785 IO-APIC-edge ata_piix > 16: 162 IO-APIC-fasteoi eth0 > 4344: 0 PCI-MSI-edge aerdrv > 4345: 0 PCI-MSI-edge aerdrv > 4346: 0 PCI-MSI-edge aerdrv > 4347: 0 PCI-MSI-edge aerdrv > 4348: 0 PCI-MSI-edge aerdrv > 4349: 0 PCI-MSI-edge aerdrv > 4350: 0 PCI-MSI-edge aerdrv > 4351: 0 PCI-MSI-edge aerdrv > NMI: 0 Non-maskable interrupts > LOC: 107095 Local timer interrupts > RES: 0 Rescheduling interrupts > CAL: 0 function call interrupts > TLB: 0 TLB shootdowns > TRM: 0 Thermal event interrupts > THR: 0 Threshold APIC interrupts > SPU: 0 Spurious interrupts > ERR: 0 > > Shouldn't there be an entry for the MSI-X device? > > Thanks for all your input. > > AK > > > > ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ > Probably what you want is something like > > { 0x2000000000ull, PCI_BASE_ADDRESS_SPACE_MEMORY | > PCI_BASE_ADDRESS_MEM_TYPE_64} , //BAR0 > { 0, 0} , //BAR1 > // 64bit BAR occupies 2 BAR entries so that BAR1 can't be used. > { 0x2000000ull, PCI_BASE_ADDRESS_SPACE_IO } , //BAR2 > { 0, 0} , //BAR3 > // for MSI-X > { 0, 0} , //BAR4 > { 0, 0} //BAR5 > > > -- yamahata From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from [140.186.70.92] (port=59609 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OttGe-0002nH-6i for qemu-devel@nongnu.org; Thu, 09 Sep 2010 22:23:09 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OttGc-0000N8-P0 for qemu-devel@nongnu.org; Thu, 09 Sep 2010 22:23:08 -0400 Received: from mail.valinux.co.jp ([210.128.90.3]:57767) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OttGc-0000MK-7P for qemu-devel@nongnu.org; Thu, 09 Sep 2010 22:23:06 -0400 Date: Fri, 10 Sep 2010 11:00:23 +0900 From: Isaku Yamahata Subject: Re: [Qemu-devel] Template for developing a Qemu device with?PCIe?and MSI-X Message-ID: <20100910020023.GB2535@valinux.co.jp> References: <20100909190713.d8dc99ce@shadowfax.no-ip.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-2022-jp Content-Disposition: inline In-Reply-To: <20100909190713.d8dc99ce@shadowfax.no-ip.com> List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Adnan Khaleel Cc: qemu-devel@nongnu.org http://www.seabios.org/pipermail/seabios/2010-July/000796.html I haven't found my time to respin to check PMM stuff yet. If you give it a try, it would be appreciated. thanks, On Thu, Sep 09, 2010 at 02:07:13PM -0500, Adnan Khaleel wrote: > Can you point me to this patch? I found one for BAR overflow checking that you > wrote which isn't merged with the seabios git source I downloaded from you. I'm > assuming this is not the one you're talking about correct? > > > ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ > From: Isaku Yamahata [mailto:yamahata@valinux.co.jp] > To: Adnan Khaleel [mailto:adnan@khaleel.us] > Cc: Cam Macdonell [mailto:cam@cs.ualberta.ca], qemu-devel@nongnu.org > Sent: Thu, 02 Sep 2010 21:20:12 -0500 > Subject: Re: [Qemu-devel] Template for developing a Qemu device with PCIe? > and MSI-X > > On Thu, Sep 02, 2010 at 12:42:42PM -0500, Adnan Khaleel wrote: > > I've tried everything you mentioned and I still get the same problem. The > only > > thing that seems to avoid that issue is if I reduce the aperture size > from > > 0x2000000000ull to 0x2000000ull. > > I suppose that Cam is seeing the same issue. > > Right now seabios can't handle too huge BAR due > to overflow. > There is a rejected patch floating around, > but I haven't created a revised patch yet. > > > > > Here is the relevant section of code: > > > > static const unsigned long long BAR_Regions[6][2] = > > { > > // len , type > > { 0x2000000ull, PCI_BASE_ADDRESS_SPACE_MEMORY | > > PCI_BASE_ADDRESS_MEM_TYPE_64} , //BAR0, > > { 0, 0} , // BAR1 > > { 0x2000000ull, PCI_BASE_ADDRESS_SPACE_IO } , //BAR2, > > { 0, 0} , // BAR3 for MSI-X > > { 0, 0} , // BAR4 > > { 0, 0} , // BAR5 > > }; > > > > static int pcie_msix_initfn(PCIDevice *pci_dev) > > { > > PCIE_MSIX_DEVState *d = DO_UPCAST(PCIE_MSIX_DEVState, dev, pci_dev); > > PCIBridge *br = DO_UPCAST(PCIBridge, dev, pci_dev); > > PCIEPort *p = DO_UPCAST(PCIEPort, br, br); > > int rc, i; > > > > PRINT_DEBUG("%s: PCIE MSIX Device init...\n", __FUNCTION__); > > > > pci_config_set_vendor_id(d->dev.config, PCIE_MSIX_VID); > > pci_config_set_device_id(d->dev.config, PCIE_MSIX_DID); > > > > memcpy(d->dev.config, g_cfg_init, sizeof(g_cfg_init[0x20])); > > d->mmio_index = cpu_register_io_memory(pcie_msix_mem_read_fn, > > pcie_msix_mem_write_fn, d); > > > > int msix_mem_bar = 0; // Since its a 64bit BAR, we take up BAR0 & BAR1 > > int msix_io_bar = 2; > > int msix_mmio_bar = 3; > > > > pci_register_bar(&d->dev, msix_mem_bar, BAR_Regions[msix_mem_bar][0], > > BAR_Regions[msix_mem_bar][1], pcie_msix_mem_map); > > pci_register_bar(&d->dev, msix_io_bar, BAR_Regions[msix_io_bar][0], > > BAR_Regions[msix_io_bar][1], pcie_msix_io_map); > > > > rc = msix_init(&d->dev, d->vectors, msix_mmio_bar, 0); > > > > if (!rc) { > > PRINT_DEBUG("%s: Registering Bar %i as I/O BAR\n", __FUNCTION__, > > msix_mmio_bar); > > pci_register_bar(&d->dev, msix_mmio_bar, msix_bar_size(&d->dev), > > PCI_BASE_ADDRESS_SPACE_MEMORY, msix_mmio_map); > > PRINT_DEBUG("%s: MSI-X initialized (%d vectors)\n", __FUNCTION__, d-> > > vectors); > > } > > else { > > PRINT_DEBUG("%s: MSI-X initialization failed!\n", __FUNCTION__); > > return rc; > > } > > > > // Activate the vectors > > for (i = 0; i < d->vectors; i++) { > > msix_vector_use(&d->dev, i); > > } > > > > rc = pci_pcie_cap_init(&d->dev, PCIE_MSIX_EXP_OFFSET, > > PCI_EXP_TYPE_ENDPOINT, p->port); > > if (rc < 0) { > > return rc; > > } > > > > pcie_cap_flr_init(&d->dev, &pcie_msix_flr); > > pcie_cap_deverr_init(&d->dev); > > pcie_cap_ari_init(&d->dev); > > rc = pcie_aer_init(&d->dev, PCIE_MSIX_AER_OFFSET); > > if (rc < 0) { > > return rc; > > } > > > > PRINT_DEBUG("%s: Init done\n", __FUNCTION__); > > return 0; > > } > > > > Another question I have is why doesn't the device show up when I try a > cat / > > proc/interrupts. > > > > linux-an84:~/AriesKernelModules/gni/aries/ghal # cat /proc/interrupts > > CPU0 > > 0: 694 IO-APIC-edge timer > > 1: 6 IO-APIC-edge i8042 > > 4: 753 IO-APIC-edge serial > > 8: 1 IO-APIC-edge rtc0 > > 9: 0 IO-APIC-fasteoi acpi > > 12: 89 IO-APIC-edge i8042 > > 14: 3522 IO-APIC-edge ata_piix > > 15: 785 IO-APIC-edge ata_piix > > 16: 162 IO-APIC-fasteoi eth0 > > 4344: 0 PCI-MSI-edge aerdrv > > 4345: 0 PCI-MSI-edge aerdrv > > 4346: 0 PCI-MSI-edge aerdrv > > 4347: 0 PCI-MSI-edge aerdrv > > 4348: 0 PCI-MSI-edge aerdrv > > 4349: 0 PCI-MSI-edge aerdrv > > 4350: 0 PCI-MSI-edge aerdrv > > 4351: 0 PCI-MSI-edge aerdrv > > NMI: 0 Non-maskable interrupts > > LOC: 107095 Local timer interrupts > > RES: 0 Rescheduling interrupts > > CAL: 0 function call interrupts > > TLB: 0 TLB shootdowns > > TRM: 0 Thermal event interrupts > > THR: 0 Threshold APIC interrupts > > SPU: 0 Spurious interrupts > > ERR: 0 > > > > Shouldn't there be an entry for the MSI-X device? > > > > Thanks for all your input. > > > > AK > > > > > > > > ????????????????????????????????????? > > Probably what you want is something like > > > > { 0x2000000000ull, PCI_BASE_ADDRESS_SPACE_MEMORY | > > PCI_BASE_ADDRESS_MEM_TYPE_64} , //BAR0 > > { 0, 0} , //BAR1 > > // 64bit BAR occupies 2 BAR entries so that BAR1 can't be used. > > { 0x2000000ull, PCI_BASE_ADDRESS_SPACE_IO } , //BAR2 > > { 0, 0} , //BAR3 > > // for MSI-X > > { 0, 0} , //BAR4 > > { 0, 0} //BAR5 > > > > > > > > -- > yamahata > -- yamahata