Decode dimms on dual socket machines

Decode dimms on dual socket machines

[Michael Fuckner]

Hi all,

I have a Dual-Socket Machine with Supermicro Motherboard (X8DTH-iF) and 
I'd like to use decode-dimms, but it just decodes the modules on one 
socket (dmidecode -t 17 and ipmitool sdr show all 6 modules on both 
sockets).


Does anybody have a clue how to access those other 3 modules?

Regards,
  Michael!


this is what I can see right now:
test24:/media/i2c-tools # ./tools/i2cdetect 0
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-0.
I will probe address range 0x03-0x77.
Continue? [Y/n]
      0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:          -- -- -- -- -- 08 -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- UU -- UU -- UU -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 2e 2f
30: 30 -- 32 -- 34 -- -- -- -- -- -- -- -- -- 3e --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
50: UU -- UU -- UU -- -- -- -- -- -- -- -- -- -- --
60: -- 61 62 -- -- -- -- -- -- 69 -- -- -- -- -- --
70: 70 -- -- -- -- -- -- --

18,1a,1c: jc42 temperature sensors
50,52,54: eeproms


test24:~ # ll /sys/bus/i2c/devices/
total 0
lrwxrwxrwx 1 root root 0 Apr 10 13:45 0-0018 -> 
../../../devices/pci0000:00/0000:00:1f.3/i2c-0/0-0018
lrwxrwxrwx 1 root root 0 Apr 10 13:45 0-001a -> 
../../../devices/pci0000:00/0000:00:1f.3/i2c-0/0-001a
lrwxrwxrwx 1 root root 0 Apr 10 13:45 0-001c -> 
../../../devices/pci0000:00/0000:00:1f.3/i2c-0/0-001c
lrwxrwxrwx 1 root root 0 Apr 10 13:48 0-0050 -> 
../../../devices/pci0000:00/0000:00:1f.3/i2c-0/0-0050
lrwxrwxrwx 1 root root 0 Apr 10 13:48 0-0052 -> 
../../../devices/pci0000:00/0000:00:1f.3/i2c-0/0-0052
lrwxrwxrwx 1 root root 0 Apr 10 13:48 0-0054 -> 
../../../devices/pci0000:00/0000:00:1f.3/i2c-0/0-0054
lrwxrwxrwx 1 root root 0 Apr  7 17:12 i2c-0 -> 
../../../devices/pci0000:00/0000:00:1f.3/i2c-0

Re: Decode dimms on dual socket machines

[Jean Delvare]

Hallo Michael,

On Tue, 12 Apr 2011 12:10:18 +0200, Michael Fuckner wrote:
> I have a Dual-Socket Machine with Supermicro Motherboard (X8DTH-iF) and 
> I'd like to use decode-dimms, but it just decodes the modules on one 
> socket (dmidecode -t 17 and ipmitool sdr show all 6 modules on both 
> sockets).
> 
> 
> Does anybody have a clue how to access those other 3 modules?

Odds are that your SMBus is multiplexed and the other memory sockets
are connected to a different bus segment. Unfortunately there are
various ways to achieve multiplexing, this can't be detected
automatically and it is rarely documented in the board manual. So you
will have to ask the vendor for technical details on this point.
Multiplexing can be done through GPIO pins (either on the south bridge
or the Super-I/O chip) or with a chip on the SMBus itself.

Note that you will need a recent enough kernel (>= 2.6.36) to have
support for SMBus multiplexing.

> this is what I can see right now:
> test24:/media/i2c-tools # ./tools/i2cdetect 0
> WARNING! This program can confuse your I2C bus, cause data loss and worse!
> I will probe file /dev/i2c-0.
> I will probe address range 0x03-0x77.
> Continue? [Y/n]
>       0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
> 00:          -- -- -- -- -- 08 -- -- -- -- -- -- --
> 10: -- -- -- -- -- -- -- -- UU -- UU -- UU -- -- --
> 20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 2e 2f
> 30: 30 -- 32 -- 34 -- -- -- -- -- -- -- -- -- 3e --
> 40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
> 50: UU -- UU -- UU -- -- -- -- -- -- -- -- -- -- --
> 60: -- 61 62 -- -- -- -- -- -- 69 -- -- -- -- -- --
> 70: 70 -- -- -- -- -- -- --

FWIW, address 0x70 is used by the PCA954x multiplexers, so maybe this
is what you have (and a driver is available).

At 0x2f is probably a W83793 or W83795 monitoring chip, supported by
the w83793 and w83795 drivers, respectively. Only use these dedicated
drivers if you do not intend to use the IPMI features of the board.

> 18,1a,1c: jc42 temperature sensors
> 50,52,54: eeproms

-- 
Jean Delvare
http://khali.linux-fr.org/wishlist.html

Re: Decode dimms on dual socket machines

[Michael Fuckner]

On 04/12/2011 01:26 PM, Jean Delvare wrote:
Hi all,

> Note that you will need a recent enough kernel (>= 2.6.36) to have
> support for SMBus multiplexing.
currently I use OpenSUSE11.4 with 2.6.37.1-1.2-default


> FWIW, address 0x70 is used by the PCA954x multiplexers, so maybe this
> is what you have (and a driver is available).
>
> At 0x2f is probably a W83793 or W83795 monitoring chip, supported by
> the w83793 and w83795 drivers, respectively. Only use these dedicated
> drivers if you do not intend to use the IPMI features of the board.
>
>> 18,1a,1c: jc42 temperature sensors
>> 50,52,54: eeproms
>


Now I loaded the module

test24:/media/i2c-tools/tools # lsmod |grep pca
pca954x                 3288  0
i2c_mux                 2672  1 pca954x
test24:/media/i2c-tools/tools # ll /sys/bus/i2c/devices/
total 0
lrwxrwxrwx 1 root root 0 Apr 12 14:26 0-0050 -> 
../../../devices/pci0000:00/0000:00:1f.3/i2c-0/0-0050
lrwxrwxrwx 1 root root 0 Apr 12 14:26 0-0052 -> 
../../../devices/pci0000:00/0000:00:1f.3/i2c-0/0-0052
lrwxrwxrwx 1 root root 0 Apr 12 14:26 0-0054 -> 
../../../devices/pci0000:00/0000:00:1f.3/i2c-0/0-0054
lrwxrwxrwx 1 root root 0 Apr 12 13:54 i2c-0 -> 
../../../devices/pci0000:00/0000:00:1f.3/i2c-0

Seems like this is what I need to know, but I don't know how to 
interpret it. Is it multiplexed on the GPIO Pin 52 and 53 at the W83795ADG?

ftp://ftp.supermicro.com/utility/SuperoDoctorINI/AllMEMDIMM.ini

Regards,
  Michael!

Re: Decode dimms on dual socket machines

[Jean Delvare]

On Tue, 12 Apr 2011 14:52:21 +0200, Michael Fuckner wrote:
> On 04/12/2011 01:26 PM, Jean Delvare wrote:
> Hi all,
> 
> > Note that you will need a recent enough kernel (>= 2.6.36) to have
> > support for SMBus multiplexing.
> currently I use OpenSUSE11.4 with 2.6.37.1-1.2-default

This is OK then.

> > FWIW, address 0x70 is used by the PCA954x multiplexers, so maybe this
> > is what you have (and a driver is available).
> >
> > At 0x2f is probably a W83793 or W83795 monitoring chip, supported by
> > the w83793 and w83795 drivers, respectively. Only use these dedicated
> > drivers if you do not intend to use the IPMI features of the board.
> >
> >> 18,1a,1c: jc42 temperature sensors
> >> 50,52,54: eeproms
> >
> 
> 
> Now I loaded the module
> 
> test24:/media/i2c-tools/tools # lsmod |grep pca
> pca954x                 3288  0
> i2c_mux                 2672  1 pca954x
> test24:/media/i2c-tools/tools # ll /sys/bus/i2c/devices/
> total 0
> lrwxrwxrwx 1 root root 0 Apr 12 14:26 0-0050 -> 
> ../../../devices/pci0000:00/0000:00:1f.3/i2c-0/0-0050
> lrwxrwxrwx 1 root root 0 Apr 12 14:26 0-0052 -> 
> ../../../devices/pci0000:00/0000:00:1f.3/i2c-0/0-0052
> lrwxrwxrwx 1 root root 0 Apr 12 14:26 0-0054 -> 
> ../../../devices/pci0000:00/0000:00:1f.3/i2c-0/0-0054
> lrwxrwxrwx 1 root root 0 Apr 12 13:54 i2c-0 -> 
> ../../../devices/pci0000:00/0000:00:1f.3/i2c-0

Loading the pca954x driver doesn't do anything. It needs an
instantiated device with proper platform data before it will do
anything. This can only be done from within the kernel, using some sort
of platform initialization code.

> Seems like this is what I need to know, but I don't know how to 
> interpret it. Is it multiplexed on the GPIO Pin 52 and 53 at the W83795ADG?
> 
> ftp://ftp.supermicro.com/utility/SuperoDoctorINI/AllMEMDIMM.ini

Indeed, good finding. This shows that the vendor is using GPIOs to
switch between the SMBus segments. This rules out the chip at I2C
address 0x70. Unfortunately the document doesn't say of which chip the
GPIO 52 and 53 are used. It's not even clear if "52" means "pin 52" or
"GPIO group 5, pin 2". I'm certain it's not related to the W83795ADG
though, as this chip has only 48 pins and 4 GPIO pins max.

My first suspect would be the Intel ICH south bridge. The ICH10 has
GPIO pins numbered 52 and 53. I don't think there is any driver for
these GPIO pins though, so one will have to be written first. Only
then, board-specific glue code can be written to link between this new
driver, i2c-i801 and gpio-i2cmux.

-- 
Jean Delvare
http://khali.linux-fr.org/wishlist.html

Re: Decode dimms on dual socket machines

[Michael Lawnick]

Jean Delvare said the following:
>> Now I loaded the module
>> 
>> test24:/media/i2c-tools/tools # lsmod |grep pca
>> pca954x                 3288  0
>> i2c_mux                 2672  1 pca954x
>> test24:/media/i2c-tools/tools # ll /sys/bus/i2c/devices/
>> total 0
>> lrwxrwxrwx 1 root root 0 Apr 12 14:26 0-0050 -> 
>> ../../../devices/pci0000:00/0000:00:1f.3/i2c-0/0-0050
>> lrwxrwxrwx 1 root root 0 Apr 12 14:26 0-0052 -> 
>> ../../../devices/pci0000:00/0000:00:1f.3/i2c-0/0-0052
>> lrwxrwxrwx 1 root root 0 Apr 12 14:26 0-0054 -> 
>> ../../../devices/pci0000:00/0000:00:1f.3/i2c-0/0-0054
>> lrwxrwxrwx 1 root root 0 Apr 12 13:54 i2c-0 -> 
>> ../../../devices/pci0000:00/0000:00:1f.3/i2c-0
> 
> Loading the pca954x driver doesn't do anything. It needs an
> instantiated device with proper platform data before it will do
> anything. This can only be done from within the kernel, using some sort
> of platform initialization code.

I would expect a
#> echo "<deviceName> 0x70" > /sys/bus/i2c/devices/i2c-0/new_device
to span new buses, their numbers taken by random, even without platform
data (not tried by myself yet, but will have to do it soon).
The challenge will be to detect which of the 8 supported devices (
pca_9540, pca_9542, pca_9543, pca_9544, pca_9545, pca_9546, pca_9547,
pca_9548) is actually to be used.

JM2C
-- 
KR
Michael

Re: Decode dimms on dual socket machines

[Jean Delvare]

On Wed, 13 Apr 2011 08:13:54 +0200, Michael Lawnick wrote:
> Jean Delvare said the following:
> >> Now I loaded the module
> >> 
> >> test24:/media/i2c-tools/tools # lsmod |grep pca
> >> pca954x                 3288  0
> >> i2c_mux                 2672  1 pca954x
> >> test24:/media/i2c-tools/tools # ll /sys/bus/i2c/devices/
> >> total 0
> >> lrwxrwxrwx 1 root root 0 Apr 12 14:26 0-0050 -> 
> >> ../../../devices/pci0000:00/0000:00:1f.3/i2c-0/0-0050
> >> lrwxrwxrwx 1 root root 0 Apr 12 14:26 0-0052 -> 
> >> ../../../devices/pci0000:00/0000:00:1f.3/i2c-0/0-0052
> >> lrwxrwxrwx 1 root root 0 Apr 12 14:26 0-0054 -> 
> >> ../../../devices/pci0000:00/0000:00:1f.3/i2c-0/0-0054
> >> lrwxrwxrwx 1 root root 0 Apr 12 13:54 i2c-0 -> 
> >> ../../../devices/pci0000:00/0000:00:1f.3/i2c-0
> > 
> > Loading the pca954x driver doesn't do anything. It needs an
> > instantiated device with proper platform data before it will do
> > anything. This can only be done from within the kernel, using some sort
> > of platform initialization code.
> 
> I would expect a
> #> echo "<deviceName> 0x70" > /sys/bus/i2c/devices/i2c-0/new_device
> to span new buses, their numbers taken by random, even without platform
> data (not tried by myself yet, but will have to do it soon).

Hmm, you're right, my memory that we has made platform data mandatory
was wrong, sorry.

> The challenge will be to detect which of the 8 supported devices (
> pca_9540, pca_9542, pca_9543, pca_9544, pca_9545, pca_9546, pca_9547,
> pca_9548) is actually to be used.

In Michael's case it is irrelevant anyway, as the information he found
meanwhile clearly indicates that bus multiplexing is achieved by GPIOs
(most certainly on the ICH10) and not an PCA954x chip. The chip at 0x70
is probably the Intel 5500 IOH.

-- 
Jean Delvare

Re: Decode dimms on dual socket machines

[Michael Fuckner]

On 04/13/2011 10:39 AM, Jean Delvare wrote:

>> The challenge will be to detect which of the 8 supported devices (
>> pca_9540, pca_9542, pca_9543, pca_9544, pca_9545, pca_9546, pca_9547,
>> pca_9548) is actually to be used.
>
> In Michael's case it is irrelevant anyway, as the information he found
> meanwhile clearly indicates that bus multiplexing is achieved by GPIOs
> (most certainly on the ICH10) and not an PCA954x chip. The chip at 0x70
> is probably the Intel 5500 IOH.


Supermicro just gave me the following information:

To access all SPDs, you have to pull low GPIO49 (IO_0x528 bit 17) first. 
Then pull low GPIO52 (IO_0x528 bit 20) to access first CPU 
memory(0x50~0x55); pull high GPIO52 to access 2nd CPU memory(0x50~0x55).


I can't find anything about GPIO49/52,53 in Datasheet for 5500/5520 chipset
http://www.intel.com/assets/pdf/datasheet/321328.pdf

So it is probably connected to ICH10
http://www.intel.com/Assets/PDF/datasheet/319973.pdf

but I don't know how to use this information

Regards,
  Michael!

Re: Decode dimms on dual socket machines

[Jean Delvare]

Hallo Michael,

On Wed, 13 Apr 2011 12:06:59 +0200, Michael Fuckner wrote:
> Supermicro just gave me the following information:
> 
> To access all SPDs, you have to pull low GPIO49 (IO_0x528 bit 17) first. 
> Then pull low GPIO52 (IO_0x528 bit 20) to access first CPU 
> memory(0x50~0x55); pull high GPIO52 to access 2nd CPU memory(0x50~0x55).

I am surprised because AllMEMDIMM.ini suggests that GPIO 53 and 52 are
used, not 47 and 52.

> I can't find anything about GPIO49/52,53 in Datasheet for 5500/5520 chipset
> http://www.intel.com/assets/pdf/datasheet/321328.pdf
> 
> So it is probably connected to ICH10
> http://www.intel.com/Assets/PDF/datasheet/319973.pdf
> 
> but I don't know how to use this information

My own workstation board happens to have a similar (but slightly
different) GPIO-based SMBus multiplexing scheme. I wanted to look into
it but could never find the time. Your request gave me an opportunity
to look into it again. As a first step, I've improved isadump and
isaset to support 32-bit I/O access:
http://marc.info/?l=lm-sensors&m=130264185604706&w=2
http://marc.info/?l=lm-sensors&m=130264189004750&w=2

With these two patches applied, you should be able to control the
multiplexer from user-space. First of all, check the base I/O address
of the GPIO registers, because the values provided by Supermicro are
off by 16 compared to what I have:

# setpci -s 00:1f.0 48.l

It returns 00000501 for me, for I/O base 0x500. If it is the same for
you, you can dump the GPIO registers with:

# isadump -f -L 0x500 64

Preliminary note: wrong setup of GPIO could have disastrous effects on
your hardware. If at any point you are unsure of what you're doing, you
better just stop and ask.

The operations described by Supermicro would then be achieved with the
following commands:

: Pull GPIO49 low
# isaset -f -L 0x538 0x00000000 0x00020000
: Pull GPIO52 low: 
# isaset -f -L 0x538 0x00000000 0x00100000
: Pull GPIO52 up: 
# isaset -f -L 0x538 0x00100000 0x00100000

Note the 0x538 when your Supermicro contact said 0x528. 0x528 would be
correct only if the I/O base is 0x4f0 instead of 0x500... but I'd be
surprised if it is the case.

Of course this is all a hack and the kernel drivers won't notice the
multiplexer change happening in their backs. So if you happen to have
the same number of memory modules for both CPUs, plugged in symmetrical
sockets, "sensors" (and "decode-dimms") will nicely report the values
from the currently selected bus segment, but you can't see all modules
at once. For this, proper multiplexing support in the kernel will be
needed, which is a lot more work. Patience...

-- 
Jean Delvare
http://khali.linux-fr.org/wishlist.html

Re: Decode dimms on dual socket machines

[Michael Lawnick]

Jean Delvare said the following:
> On Wed, 13 Apr 2011 08:13:54 +0200, Michael Lawnick wrote:
>> I would expect a
>> #> echo "<deviceName> 0x70" > /sys/bus/i2c/devices/i2c-0/new_device
>> to span new buses, their numbers taken by random, even without platform
>> data (not tried by myself yet, but will have to do it soon).
> 
> Hmm, you're right, my memory that we has made platform data mandatory
> was wrong, sorry.
> 
>> The challenge will be to detect which of the 8 supported devices (
>> pca_9540, pca_9542, pca_9543, pca_9544, pca_9545, pca_9546, pca_9547,
>> pca_9548) is actually to be used.
> 
> In Michael's case it is irrelevant anyway, as the information he found
> meanwhile clearly indicates that bus multiplexing is achieved by GPIOs
> (most certainly on the ICH10) and not an PCA954x chip. The chip at 0x70
> is probably the Intel 5500 IOH.
> 
Well, if he implements a multiplexer code for it, he is done :-)

-- 
KR
Michael

Re: Decode dimms on dual socket machines

[Jean Delvare]

Hi Michael,

On Wed, 13 Apr 2011 12:06:59 +0200, Michael Fuckner wrote:
> On 04/13/2011 10:39 AM, Jean Delvare wrote:
> 
> >> The challenge will be to detect which of the 8 supported devices (
> >> pca_9540, pca_9542, pca_9543, pca_9544, pca_9545, pca_9546, pca_9547,
> >> pca_9548) is actually to be used.
> >
> > In Michael's case it is irrelevant anyway, as the information he found
> > meanwhile clearly indicates that bus multiplexing is achieved by GPIOs
> > (most certainly on the ICH10) and not an PCA954x chip. The chip at 0x70
> > is probably the Intel 5500 IOH.
> 
> 
> Supermicro just gave me the following information:
> 
> To access all SPDs, you have to pull low GPIO49 (IO_0x528 bit 17) first. 
> Then pull low GPIO52 (IO_0x528 bit 20) to access first CPU 
> memory(0x50~0x55); pull high GPIO52 to access 2nd CPU memory(0x50~0x55).
> 
> 
> I can't find anything about GPIO49/52,53 in Datasheet for 5500/5520 chipset
> http://www.intel.com/assets/pdf/datasheet/321328.pdf
> 
> So it is probably connected to ICH10
> http://www.intel.com/Assets/PDF/datasheet/319973.pdf
> 
> but I don't know how to use this information

I've written a kernel driver to gain access to the GPIO pins of the
ICH10 (and previous ICH chips too.) It's available here:
http://khali.linux-fr.org/devel/misc/i801_gpio/

Generic instructions for standalone drivers are here:
http://khali.linux-fr.org/devel/misc/INSTALL

I would appreciate if you could give it a try on your system and report
everything in the kernel log after loading the driver. By default it
won't change the GPIO values so it should be totally safe.

Thanks,
-- 
Jean Delvare
http://khali.linux-fr.org/wishlist.html