RE: PCI device order problem

RE: PCI device order problem

[Leech, Christopher]

If you're that concerned about possible ordering changes due to future BIOS
upgrades, I'd suggest setting up an /etc/mactab and using nameif to control
interface naming from userspace.

-- Chris Leech

> -----Original Message-----
> From: H. J. Lu [mailto:hjl@lucon.org] 
> We can use eth1. It is just very confusing since Linux and 
> hardware manual don't agree which one is the first NIC. Also, 
> when we upgrade the BIOS, the BIOS order may change. As for 
> other schemes, we don't want to change every software which 
> access ethX.

Re: PCI device order problem

[H. J. Lu]

On Thu, Oct 24, 2002 at 05:25:01PM -0700, Leech, Christopher wrote:
> 
> If you're that concerned about possible ordering changes due to future BIOS
> upgrades, I'd suggest setting up an /etc/mactab and using nameif to control
> interface naming from userspace.
> 

That is not enough. I use PXE to install RedHat. The RedHat 7.3 installer only
deals with ethX.


H.J.

Re: PCI device order problem

[H. J. Lu]

On Fri, Oct 25, 2002 at 11:00:31AM +0100, Alan Cox wrote:
> On Fri, 2002-10-25 at 00:56, H. J. Lu wrote:
> > It is different from the hardware documentation. The hardware manual says
> > it has 2 NICs, NIC 1 (03:07.0) and NIC2 (03:07.1), which makes senses
> > to me. NIC 1 is a special one which supports IPMI over LAN. Since we
> > only use one NIC now, we'd like to use NIC 1 and call it eth0.
> 
> SIOCSIFNAME ioctl. You can call them "haddock" and "chips" if you really
> want, or swap the eth%d names about. RH 8.0 allows you to bind an
> interface to a mac address too
> 

It doesn't help RedHat installer over network.


H.J.

Re: PCI device order problem

[Alan Cox]

On Fri, 2002-10-25 at 00:56, H. J. Lu wrote:
> It is different from the hardware documentation. The hardware manual says
> it has 2 NICs, NIC 1 (03:07.0) and NIC2 (03:07.1), which makes senses
> to me. NIC 1 is a special one which supports IPMI over LAN. Since we
> only use one NIC now, we'd like to use NIC 1 and call it eth0.

SIOCSIFNAME ioctl. You can call them "haddock" and "chips" if you really
want, or swap the eth%d names about. RH 8.0 allows you to bind an
interface to a mac address too

Re: PCI device order problem

[H. J. Lu]

On Thu, Oct 24, 2002 at 05:21:10PM -0700, Leech, Christopher wrote:
> I've heard some grumbling about this with specific motherboards and 82546
> LOMs. While I understand what's happening, and that using nameif to manage
> this is the correct answer, I am a bit surprised that function 1 would be
> placed on the global PCI device list before function 0 for a multi-function
> device.
> 

You can tell it from the BIOS, which lists 0339 before 0338 by default.


H.J.

Re: PCI device order problem

[Jeff Garzik]

There is also the pci=nosort kernel command line parameter.

RE: PCI device order problem

[Leech, Christopher]

I've heard some grumbling about this with specific motherboards and 82546
LOMs. While I understand what's happening, and that using nameif to manage
this is the correct answer, I am a bit surprised that function 1 would be
placed on the global PCI device list before function 0 for a multi-function
device.

-- Chris Leech


> >>Without answering your specific question, but addressing $subject, 
> >>what
> >>problem is caused by the PCI device order you see?
> >>    
> >>
> >
> >It is different from the hardware documentation. The hardware manual 
> >says it has 2 NICs, NIC 1 (03:07.0) and NIC2 (03:07.1), which makes 
> >senses to me. NIC 1 is a special one which supports IPMI over LAN. 
> >Since we only use one NIC now, we'd like to use NIC 1 and 
> call it eth0.
> >  
> >
> 
> Well, overall, depending on ordering is error-prone (as you see).  I 
> would suggest migrating to a less-fragile scheme.  nameif and ethtool 
> together should get you exactly the device you need...  
> (though I wonder 
> why simply using eth1 is so awful, if this situation is constant...)
> 
>     Jeff

Re: PCI device order problem

[H. J. Lu]

On Thu, Oct 24, 2002 at 08:14:09PM -0400, Jeff Garzik wrote:
> H. J. Lu wrote:
> 
> >On Thu, Oct 24, 2002 at 07:49:41PM -0400, Jeff Garzik wrote:
> >  
> >
> >>H. J. Lu wrote:
> >>
> >>    
> >>
> >>>In arch/i386/kernel/pci-pc.c, there are
> >>>
> >>>/*
> >>>* Sort the device list according to PCI BIOS. Nasty hack, but since some
> >>>* fool forgot to define the `correct' device order in the PCI BIOS specs
> >>>* and we want to be (possibly bug-to-bug ;-]) compatible with older kernels 
> >>>* which used BIOS ordering, we are bound to do this... 
> >>>*/
> >>>
> >>>static void __devinit pcibios_sort(void)
> >>>
> >>>The problem is on my MB:
> >>>
> >>>00:00.0 Host bridge: Intel Corp. e7500 [Plumas] DRAM Controller (rev 03)
> >>>00:00.1 Class ff00: Intel Corp. e7500 [Plumas] DRAM Controller Error Reporting ( rev 03)
> >>>00:03.0 PCI bridge: Intel Corp. e7500 [Plumas] HI_C Virtual PCI Bridge (F0) (rev 03)
> >>>00:03.1 Class ff00: Intel Corp. e7500 [Plumas] HI_C Virtual PCI Bridge (F1) (rev 03)
> >>>00:1d.0 USB Controller: Intel Corp. 82801CA/CAM USB (Hub  (rev 02)
> >>>00:1d.1 USB Controller: Intel Corp. 82801CA/CAM USB (Hub  (rev 02)
> >>>00:1e.0 PCI bridge: Intel Corp. 82801BA/CA/DB PCI Bridge (rev 42)
> >>>00:1f.0 ISA bridge: Intel Corp. 82801CA ISA Bridge (LPC) (rev 02)
> >>>00:1f.1 IDE interface: Intel Corp. 82801CA IDE U100 (rev 02)
> >>>00:1f.3 SMBus: Intel Corp. 82801CA/CAM SMBus (rev 02)
> >>>01:0c.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27)
> >>>02:1c.0 PIC: Intel Corp. 82870P2 P64H2 I/OxAPIC (rev 03)
> >>>02:1d.0 PCI bridge: Intel Corp. 82870P2 P64H2 Hub PCI Bridge (rev 03)
> >>>02:1e.0 PIC: Intel Corp. 82870P2 P64H2 I/OxAPIC (rev 03)
> >>>02:1f.0 PCI bridge: Intel Corp. 82870P2 P64H2 Hub PCI Bridge (rev 03)
> >>>03:07.0 Ethernet controller: Intel Corp. 82546EB Gigabit Ethernet Controller (rev 01)
> >>>03:07.1 Ethernet controller: Intel Corp. 82546EB Gigabit Ethernet Controller (rev 01)
> >>>03:08.0 RAID bus controller: 3ware Inc 3ware 7000-series ATA-RAID (rev 01)
> >>>
> >>>Eth1 becomes:
> >>>03:07.0 Ethernet controller: Intel Corp. 82546EB Gigabit Ethernet Controller (rev 01)
> >>>
> >>>and eth0 becomes:
> >>>03:07.1 Ethernet controller: Intel Corp. 82546EB Gigabit Ethernet Controller (rev 01)
> >>>
> >>>Is that a good idea to have an option to sort the PCI device by PCI bus and
> >>>slot numbers?
> >>> 
> >>>
> >>>      
> >>>
> >>Without answering your specific question, but addressing $subject, what 
> >>problem is caused by the PCI device order you see?
> >>    
> >>
> >
> >It is different from the hardware documentation. The hardware manual says
> >it has 2 NICs, NIC 1 (03:07.0) and NIC2 (03:07.1), which makes senses
> >to me. NIC 1 is a special one which supports IPMI over LAN. Since we
> >only use one NIC now, we'd like to use NIC 1 and call it eth0.
> >  
> >
> 
> Well, overall, depending on ordering is error-prone (as you see).  I 
> would suggest migrating to a less-fragile scheme.  nameif and ethtool 
> together should get you exactly the device you need...  (though I wonder 
> why simply using eth1 is so awful, if this situation is constant...)

We can use eth1. It is just very confusing since Linux and hardware
manual don't agree which one is the first NIC. Also, when we upgrade
the BIOS, the BIOS order may change. As for other schemes, we don't want
to change every software which access ethX.


H.J.

Re: PCI device order problem

[Jeff Garzik]

H. J. Lu wrote:

>On Thu, Oct 24, 2002 at 07:49:41PM -0400, Jeff Garzik wrote:
>  
>
>>H. J. Lu wrote:
>>
>>    
>>
>>>In arch/i386/kernel/pci-pc.c, there are
>>>
>>>/*
>>>* Sort the device list according to PCI BIOS. Nasty hack, but since some
>>>* fool forgot to define the `correct' device order in the PCI BIOS specs
>>>* and we want to be (possibly bug-to-bug ;-]) compatible with older kernels 
>>>* which used BIOS ordering, we are bound to do this... 
>>>*/
>>>
>>>static void __devinit pcibios_sort(void)
>>>
>>>The problem is on my MB:
>>>
>>>00:00.0 Host bridge: Intel Corp. e7500 [Plumas] DRAM Controller (rev 03)
>>>00:00.1 Class ff00: Intel Corp. e7500 [Plumas] DRAM Controller Error Reporting ( rev 03)
>>>00:03.0 PCI bridge: Intel Corp. e7500 [Plumas] HI_C Virtual PCI Bridge (F0) (rev 03)
>>>00:03.1 Class ff00: Intel Corp. e7500 [Plumas] HI_C Virtual PCI Bridge (F1) (rev 03)
>>>00:1d.0 USB Controller: Intel Corp. 82801CA/CAM USB (Hub  (rev 02)
>>>00:1d.1 USB Controller: Intel Corp. 82801CA/CAM USB (Hub  (rev 02)
>>>00:1e.0 PCI bridge: Intel Corp. 82801BA/CA/DB PCI Bridge (rev 42)
>>>00:1f.0 ISA bridge: Intel Corp. 82801CA ISA Bridge (LPC) (rev 02)
>>>00:1f.1 IDE interface: Intel Corp. 82801CA IDE U100 (rev 02)
>>>00:1f.3 SMBus: Intel Corp. 82801CA/CAM SMBus (rev 02)
>>>01:0c.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27)
>>>02:1c.0 PIC: Intel Corp. 82870P2 P64H2 I/OxAPIC (rev 03)
>>>02:1d.0 PCI bridge: Intel Corp. 82870P2 P64H2 Hub PCI Bridge (rev 03)
>>>02:1e.0 PIC: Intel Corp. 82870P2 P64H2 I/OxAPIC (rev 03)
>>>02:1f.0 PCI bridge: Intel Corp. 82870P2 P64H2 Hub PCI Bridge (rev 03)
>>>03:07.0 Ethernet controller: Intel Corp. 82546EB Gigabit Ethernet Controller (rev 01)
>>>03:07.1 Ethernet controller: Intel Corp. 82546EB Gigabit Ethernet Controller (rev 01)
>>>03:08.0 RAID bus controller: 3ware Inc 3ware 7000-series ATA-RAID (rev 01)
>>>
>>>Eth1 becomes:
>>>03:07.0 Ethernet controller: Intel Corp. 82546EB Gigabit Ethernet Controller (rev 01)
>>>
>>>and eth0 becomes:
>>>03:07.1 Ethernet controller: Intel Corp. 82546EB Gigabit Ethernet Controller (rev 01)
>>>
>>>Is that a good idea to have an option to sort the PCI device by PCI bus and
>>>slot numbers?
>>> 
>>>
>>>      
>>>
>>Without answering your specific question, but addressing $subject, what 
>>problem is caused by the PCI device order you see?
>>    
>>
>
>It is different from the hardware documentation. The hardware manual says
>it has 2 NICs, NIC 1 (03:07.0) and NIC2 (03:07.1), which makes senses
>to me. NIC 1 is a special one which supports IPMI over LAN. Since we
>only use one NIC now, we'd like to use NIC 1 and call it eth0.
>  
>

Well, overall, depending on ordering is error-prone (as you see).  I 
would suggest migrating to a less-fragile scheme.  nameif and ethtool 
together should get you exactly the device you need...  (though I wonder 
why simply using eth1 is so awful, if this situation is constant...)

    Jeff

Re: PCI device order problem

[H. J. Lu]

On Thu, Oct 24, 2002 at 07:49:41PM -0400, Jeff Garzik wrote:
> H. J. Lu wrote:
> 
> >In arch/i386/kernel/pci-pc.c, there are
> >
> >/*
> > * Sort the device list according to PCI BIOS. Nasty hack, but since some
> > * fool forgot to define the `correct' device order in the PCI BIOS specs
> > * and we want to be (possibly bug-to-bug ;-]) compatible with older kernels 
> > * which used BIOS ordering, we are bound to do this... 
> > */
> >
> >static void __devinit pcibios_sort(void)
> >
> >The problem is on my MB:
> >
> >00:00.0 Host bridge: Intel Corp. e7500 [Plumas] DRAM Controller (rev 03)
> >00:00.1 Class ff00: Intel Corp. e7500 [Plumas] DRAM Controller Error Reporting ( rev 03)
> >00:03.0 PCI bridge: Intel Corp. e7500 [Plumas] HI_C Virtual PCI Bridge (F0) (rev 03)
> >00:03.1 Class ff00: Intel Corp. e7500 [Plumas] HI_C Virtual PCI Bridge (F1) (rev 03)
> >00:1d.0 USB Controller: Intel Corp. 82801CA/CAM USB (Hub  (rev 02)
> >00:1d.1 USB Controller: Intel Corp. 82801CA/CAM USB (Hub  (rev 02)
> >00:1e.0 PCI bridge: Intel Corp. 82801BA/CA/DB PCI Bridge (rev 42)
> >00:1f.0 ISA bridge: Intel Corp. 82801CA ISA Bridge (LPC) (rev 02)
> >00:1f.1 IDE interface: Intel Corp. 82801CA IDE U100 (rev 02)
> >00:1f.3 SMBus: Intel Corp. 82801CA/CAM SMBus (rev 02)
> >01:0c.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27)
> >02:1c.0 PIC: Intel Corp. 82870P2 P64H2 I/OxAPIC (rev 03)
> >02:1d.0 PCI bridge: Intel Corp. 82870P2 P64H2 Hub PCI Bridge (rev 03)
> >02:1e.0 PIC: Intel Corp. 82870P2 P64H2 I/OxAPIC (rev 03)
> >02:1f.0 PCI bridge: Intel Corp. 82870P2 P64H2 Hub PCI Bridge (rev 03)
> >03:07.0 Ethernet controller: Intel Corp. 82546EB Gigabit Ethernet Controller (rev 01)
> >03:07.1 Ethernet controller: Intel Corp. 82546EB Gigabit Ethernet Controller (rev 01)
> >03:08.0 RAID bus controller: 3ware Inc 3ware 7000-series ATA-RAID (rev 01)
> >
> >Eth1 becomes:
> >03:07.0 Ethernet controller: Intel Corp. 82546EB Gigabit Ethernet Controller (rev 01)
> >
> >and eth0 becomes:
> >03:07.1 Ethernet controller: Intel Corp. 82546EB Gigabit Ethernet Controller (rev 01)
> >
> >Is that a good idea to have an option to sort the PCI device by PCI bus and
> >slot numbers?
> >  
> >
> 
> Without answering your specific question, but addressing $subject, what 
> problem is caused by the PCI device order you see?

It is different from the hardware documentation. The hardware manual says
it has 2 NICs, NIC 1 (03:07.0) and NIC2 (03:07.1), which makes senses
to me. NIC 1 is a special one which supports IPMI over LAN. Since we
only use one NIC now, we'd like to use NIC 1 and call it eth0.


H.J.

Re: PCI device order problem

[Jeff Garzik]

H. J. Lu wrote:

>In arch/i386/kernel/pci-pc.c, there are
>
>/*
> * Sort the device list according to PCI BIOS. Nasty hack, but since some
> * fool forgot to define the `correct' device order in the PCI BIOS specs
> * and we want to be (possibly bug-to-bug ;-]) compatible with older kernels 
> * which used BIOS ordering, we are bound to do this... 
> */
>
>static void __devinit pcibios_sort(void)
>
>The problem is on my MB:
>
>00:00.0 Host bridge: Intel Corp. e7500 [Plumas] DRAM Controller (rev 03)
>00:00.1 Class ff00: Intel Corp. e7500 [Plumas] DRAM Controller Error Reporting ( rev 03)
>00:03.0 PCI bridge: Intel Corp. e7500 [Plumas] HI_C Virtual PCI Bridge (F0) (rev 03)
>00:03.1 Class ff00: Intel Corp. e7500 [Plumas] HI_C Virtual PCI Bridge (F1) (rev 03)
>00:1d.0 USB Controller: Intel Corp. 82801CA/CAM USB (Hub  (rev 02)
>00:1d.1 USB Controller: Intel Corp. 82801CA/CAM USB (Hub  (rev 02)
>00:1e.0 PCI bridge: Intel Corp. 82801BA/CA/DB PCI Bridge (rev 42)
>00:1f.0 ISA bridge: Intel Corp. 82801CA ISA Bridge (LPC) (rev 02)
>00:1f.1 IDE interface: Intel Corp. 82801CA IDE U100 (rev 02)
>00:1f.3 SMBus: Intel Corp. 82801CA/CAM SMBus (rev 02)
>01:0c.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27)
>02:1c.0 PIC: Intel Corp. 82870P2 P64H2 I/OxAPIC (rev 03)
>02:1d.0 PCI bridge: Intel Corp. 82870P2 P64H2 Hub PCI Bridge (rev 03)
>02:1e.0 PIC: Intel Corp. 82870P2 P64H2 I/OxAPIC (rev 03)
>02:1f.0 PCI bridge: Intel Corp. 82870P2 P64H2 Hub PCI Bridge (rev 03)
>03:07.0 Ethernet controller: Intel Corp. 82546EB Gigabit Ethernet Controller (rev 01)
>03:07.1 Ethernet controller: Intel Corp. 82546EB Gigabit Ethernet Controller (rev 01)
>03:08.0 RAID bus controller: 3ware Inc 3ware 7000-series ATA-RAID (rev 01)
>
>Eth1 becomes:
>03:07.0 Ethernet controller: Intel Corp. 82546EB Gigabit Ethernet Controller (rev 01)
>
>and eth0 becomes:
>03:07.1 Ethernet controller: Intel Corp. 82546EB Gigabit Ethernet Controller (rev 01)
>
>Is that a good idea to have an option to sort the PCI device by PCI bus and
>slot numbers?
>  
>

Without answering your specific question, but addressing $subject, what 
problem is caused by the PCI device order you see?

You can use ETHTOOL_GDRVINFO to get the PCI bus info for an ethernet 
interface, so you know at all times what device is associated with what 
PCI device.  And among other ways you are notified of device 
addition/removal by the execution of /sbin/hotplug for each ethernet 
interface.

    Jeff

PCI device order problem

[H. J. Lu]

In arch/i386/kernel/pci-pc.c, there are

/*
 * Sort the device list according to PCI BIOS. Nasty hack, but since some
 * fool forgot to define the `correct' device order in the PCI BIOS specs
 * and we want to be (possibly bug-to-bug ;-]) compatible with older kernels 
 * which used BIOS ordering, we are bound to do this... 
 */

static void __devinit pcibios_sort(void)

The problem is on my MB:

00:00.0 Host bridge: Intel Corp. e7500 [Plumas] DRAM Controller (rev 03)
00:00.1 Class ff00: Intel Corp. e7500 [Plumas] DRAM Controller Error Reporting ( rev 03)
00:03.0 PCI bridge: Intel Corp. e7500 [Plumas] HI_C Virtual PCI Bridge (F0) (rev 03)
00:03.1 Class ff00: Intel Corp. e7500 [Plumas] HI_C Virtual PCI Bridge (F1) (rev 03)
00:1d.0 USB Controller: Intel Corp. 82801CA/CAM USB (Hub  (rev 02)
00:1d.1 USB Controller: Intel Corp. 82801CA/CAM USB (Hub  (rev 02)
00:1e.0 PCI bridge: Intel Corp. 82801BA/CA/DB PCI Bridge (rev 42)
00:1f.0 ISA bridge: Intel Corp. 82801CA ISA Bridge (LPC) (rev 02)
00:1f.1 IDE interface: Intel Corp. 82801CA IDE U100 (rev 02)
00:1f.3 SMBus: Intel Corp. 82801CA/CAM SMBus (rev 02)
01:0c.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27)
02:1c.0 PIC: Intel Corp. 82870P2 P64H2 I/OxAPIC (rev 03)
02:1d.0 PCI bridge: Intel Corp. 82870P2 P64H2 Hub PCI Bridge (rev 03)
02:1e.0 PIC: Intel Corp. 82870P2 P64H2 I/OxAPIC (rev 03)
02:1f.0 PCI bridge: Intel Corp. 82870P2 P64H2 Hub PCI Bridge (rev 03)
03:07.0 Ethernet controller: Intel Corp. 82546EB Gigabit Ethernet Controller (rev 01)
03:07.1 Ethernet controller: Intel Corp. 82546EB Gigabit Ethernet Controller (rev 01)
03:08.0 RAID bus controller: 3ware Inc 3ware 7000-series ATA-RAID (rev 01)

Eth1 becomes:
03:07.0 Ethernet controller: Intel Corp. 82546EB Gigabit Ethernet Controller (rev 01)

and eth0 becomes:
03:07.1 Ethernet controller: Intel Corp. 82546EB Gigabit Ethernet Controller (rev 01)

Is that a good idea to have an option to sort the PCI device by PCI bus and
slot numbers?


H.J.