2.6.11.10 failure on Hypersparc sparc32 [was: Re: sparc32 2.6 work]

2.6.11.10 failure on Hypersparc sparc32 [was: Re: sparc32 2.6 work]

[Jurij Smakov]

Hi again,

I have now tried the pristine 2.6.11.10 kernel and it fails on mounting 
initrd too on my Hypersparc-powered SS10. The last messages (same as with 
Debian kernels) I see are:

NET: Registered protocol family 1
NET: Registered protocol family 17
RAMDISK: cramfs filesystem found at block 0
RAMDISK: Loading 2556KiB [1 disk] into ram disk... done.
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
  <0>Press L1-A to return to the boot prom

It would be nice if you (or someone else) could confirm (or not :-) that, 
since it would eliminate possible initrd/toolchain problems.

http://www.wooyd.org/misc/sparc32.config

Thanks and best regards,

Jurij Smakov                                        jurij@wooyd.org
Key: http://www.wooyd.org/pgpkey/                   KeyID: C99E03CC
A

Re: 2.6.11.10 failure on Hypersparc sparc32 [was: Re: sparc32 2.6

[Tom 'spot' Callaway]

On Thu, 2005-05-19 at 01:20 -0400, Jurij Smakov wrote:
> Hi again,
> 
> I have now tried the pristine 2.6.11.10 kernel and it fails on mounting 
> initrd too on my Hypersparc-powered SS10. The last messages (same as with 
> Debian kernels) I see are:

I'd suggest that you try the 2.6.12-rc kernels instead. Sparc32 is
anything but stable, and lots of fixes aren't going into that "stable"
patchset (but are going into the rc patches).

I'm not even going to spend any cycles with the 2.6.11.* patchese
because I know they're missing at least 10 or so of my fixes.

~spot
-- 
Tom "spot" Callaway: Red Hat Sales Engineer || GPG Fingerprint: 93054260
Fedora Extras Steering Committee Member (RPM Standards and Practices)
Aurora Linux Project Leader: http://auroralinux.org
Lemurs, llamas, and sparcs, oh my!

Re: 2.6.11.10 failure on Hypersparc sparc32 [was: Re: sparc32 2.6

[Bob Breuer]

Jurij Smakov wrote:
> Hi again,
> 
> I have now tried the pristine 2.6.11.10 kernel and it fails on mounting 
> initrd too on my Hypersparc-powered SS10. The last messages (same as 
> with Debian kernels) I see are:
> 
> NET: Registered protocol family 1
> NET: Registered protocol family 17
> RAMDISK: cramfs filesystem found at block 0
             ^^^^
I don't think my initrds have been of the cramfs type.

> RAMDISK: Loading 2556KiB [1 disk] into ram disk... done.
> Kernel panic - not syncing: VFS: Unable to mount root fs on 
> unknown-block(0,0)
>  <0>Press L1-A to return to the boot prom
> 
> It would be nice if you (or someone else) could confirm (or not :-) 
> that, since it would eliminate possible initrd/toolchain problems.
> 
> http://www.wooyd.org/misc/sparc32.config

With that config, I can't boot the resulting image because it's too 
large for the silo that I'm using.

[root@sparky linux-2.6.11.10]# ls -l arch/sparc/boot/image
-rwxr-xr-x  1 root root 2921764 May 19 04:34 arch/sparc/boot/image
Built with: gcc version 3.3.4 20040623 (Red Hat 3.3.4-2)

Here's the similar output from the latest Aurora kernel:

Linux version 2.6.11-1.1166sp1 (root@arthur.devel.redhat.com) (gcc 
version 3.4.2 20041017 (Red Hat 3.4.2-6.fc3)) #1 Fri Mar 4 21:55:08 EST 2005
...
NET: Registered protocol family 1
NET: Registered protocol family 17
Freeing unused kernel memory: 156k freed
Red Hat nash version 4.1.18 starting
Mounted /proc filesystem
...

No explicit mention of a ramdisk, but nash is the shell used with the 
initrd.

Some more details about that kernel and it's initrd:

[root@sparky boot]# ls -l *1166*
-rw-r--r--  1 root root   25816 Mar  4 21:07 config-2.6.11-1.1166sp1
-rw-r--r--  1 root root  753917 Mar  7 10:09 initrd-2.6.11-1.1166sp1.img
-rw-r--r--  1 root root  453452 Mar  4 21:07 System.map-2.6.11-1.1166sp1
-rwxr-xr-x  1 root root 2471360 Mar  4 21:31 vmlinuz-2.6.11-1.1166sp1
[root@sparky boot]# file vmlinuz-2.6.11-1.1166sp1 

vmlinuz-2.6.11-1.1166sp1: ELF 32-bit MSB executable, SPARC, version 1
(SYSV), statically linked, stripped
[root@sparky boot]# file initrd-2.6.11-1.1166sp1.img
initrd-2.6.11-1.1166sp1.img: gzip compressed data, from Unix, max 
compression
[root@sparky boot]# zcat initrd-2.6.11-1.1166sp1.img >/tmp/initrd
[root@sparky boot]# file /tmp/initrd
/tmp/initrd: ASCII cpio archive (SVR4 with no CRC)

Looks like my initrd is a cpio archive instead of cramfs.

And here's the versions of my mkinitrd and silo:
   mkinitrd-4.1.18-2
   silo-1.4.8-1

Bob

Re: 2.6.11.10 failure on Hypersparc sparc32 [was: Re: sparc32 2.6

[Jurij Smakov]

On Thu, 19 May 2005, Tom 'spot' Callaway wrote:

> I'd suggest that you try the 2.6.12-rc kernels instead. Sparc32 is
> anything but stable, and lots of fixes aren't going into that "stable"
> patchset (but are going into the rc patches).
>
> I'm not even going to spend any cycles with the 2.6.11.* patchese
> because I know they're missing at least 10 or so of my fixes.
>
> ~spot

Thanks for the tip Tom, I'll try .12-rc. However, from what I remember, 
the patches that went there where mostly framebuffer and cosmetic fixes, 
hardly anything which could affect such things as initrd mounting.

Best regards,

Jurij Smakov                                        jurij@wooyd.org
Key: http://www.wooyd.org/pgpkey/                   KeyID: C99E03CC

Re: 2.6.11.10 failure on Hypersparc sparc32 [was: Re: sparc32 2.6

[Jurij Smakov]

On Thu, 19 May 2005, Bob Breuer wrote:

> With that config, I can't boot the resulting image because it's too large for 
> the silo that I'm using.

Right, sorry about that. Debian's sparc kernels are stripped using the 
command

strip -R .comment -R .note -K sun4u_init -K _end -K _start image

after building, that reduces their size to acceptable by SILO.

> Some more details about that kernel and it's initrd:
>
> [root@sparky boot]# ls -l *1166*
> -rw-r--r--  1 root root   25816 Mar  4 21:07 config-2.6.11-1.1166sp1
> -rw-r--r--  1 root root  753917 Mar  7 10:09 initrd-2.6.11-1.1166sp1.img
> -rw-r--r--  1 root root  453452 Mar  4 21:07 System.map-2.6.11-1.1166sp1
> -rwxr-xr-x  1 root root 2471360 Mar  4 21:31 vmlinuz-2.6.11-1.1166sp1
> [root@sparky boot]# file vmlinuz-2.6.11-1.1166sp1 
> vmlinuz-2.6.11-1.1166sp1: ELF 32-bit MSB executable, SPARC, version 1
> (SYSV), statically linked, stripped
> [root@sparky boot]# file initrd-2.6.11-1.1166sp1.img
> initrd-2.6.11-1.1166sp1.img: gzip compressed data, from Unix, max compression
> [root@sparky boot]# zcat initrd-2.6.11-1.1166sp1.img >/tmp/initrd
> [root@sparky boot]# file /tmp/initrd
> /tmp/initrd: ASCII cpio archive (SVR4 with no CRC)
>
> Looks like my initrd is a cpio archive instead of cramfs.

Interesting. I guess that it actually uses initramfs capability to load 
cpio archives into ramdisk. We are planning this transition at some point, 
but at the moment cramfs/ext2 initrds is what is used.

Thanks and best regards,

Jurij Smakov                                        jurij@wooyd.org
Key: http://www.wooyd.org/pgpkey/                   KeyID: C99E03CC

Re: 2.6.11.10 failure on Hypersparc sparc32 [was: Re: sparc32 2.6

[Tom 'spot' Callaway]

On Thu, 2005-05-19 at 12:41 -0400, Jurij Smakov wrote:

> Interesting. I guess that it actually uses initramfs capability to load 
> cpio archives into ramdisk. We are planning this transition at some point, 
> but at the moment cramfs/ext2 initrds is what is used.

I'm not aware that cramfs initrds have ever worked properly with 2.6...
I could be wrong. I thought the direction was initramfs with 2.6.
Perhaps this is a good time to make that transition? :)

~spot
-- 
Tom "spot" Callaway: Red Hat Sales Engineer || GPG Fingerprint: 93054260
Fedora Extras Steering Committee Member (RPM Standards and Practices)
Aurora Linux Project Leader: http://auroralinux.org
Lemurs, llamas, and sparcs, oh my!

Re: 2.6.11.10 failure on Hypersparc sparc32 [was: Re: sparc32 2.6

[Bob Breuer]

Jurij Smakov wrote:
> 
> strip -R .comment -R .note -K sun4u_init -K _end -K _start image

Yes, that worked.  I'll have to remember that next time.

> Interesting. I guess that it actually uses initramfs capability to load 
> cpio archives into ramdisk. We are planning this transition at some 
> point, but at the moment cramfs/ext2 initrds is what is used.

I just noticed the initramfs messages earlier in the boot:

Mount-cache hash table entries: 512 (order: 0, 4096 bytes)
checking if image is initramfs... it is
Freeing initrd memory: 741k freed

Does a cramfs/ext2 initrd get freed at that point also?

Even though my build loads the initrd, it still has 2 problems.
1) it complains about udiv:
scsi_mod: no version for "udiv" found: kernel tainted.

2) blows up when probing the scsi bus:
Attached scsi disk sda at scsi0, channel 0, id 1, lun 0
esp0: Aborting command
esp0: dumping state
esp0: dma -- cond_reg<a6400310> addr<f0083e41>
esp0: SW [sreg<11> sstep<04> ireg<18>]
esp0: HW reread [sreg<01> sstep<c4> ireg<00>]
esp0: current command [tgt<01> lun<01> pphase<CLUELESS> cphase<DATAIN>]
esp0: disconnected
...
   and more scsi errors, eventually leading to a kernel panic.

I've seen this scsi problem before and worked around it by un-selecting 
SCSI_MULTI_LUN when scsi was a module.

Bob

Re: 2.6.11.10 failure on Hypersparc sparc32 [was: Re: sparc32 2.6

[Jurij Smakov]

Hi,

I have tried to dig a little bit more on this issue, with some success. 
It appears that initrd does not mount because something is wrong with the
__copy_1page function, invoked by memcpy() for copying page-sized chunks.
It is used in fs/cramfs/inode.c to read in the superblock of the cramfs 
filesystem and, surprisingly, in this process first 16 bytes (containing 
the magic cramfs number, among other things), get lost somehow. After I've 
replaced a call to __copy_1page with a call to __memcpy in 
include/asm-sparc/string.h, initrd mounted, however boot still fails. The 
next problem is that /linuxrc script from initrd is not executed as it 
should. This script is supposed to save real-root-device from /proc and 
set the real-root-device to ramdisk, where initrd is located (it all 
happens in do_mounts_initrd.c). However, the execve call to /linuxrc 
quietly fails, and the script is never executed. After that kernel tries 
to immediately mount /dev/sda1 (which is the real root) and fails, because 
the scsi modules have not been loaded. So it now looks like this:

RAMDISK: cramfs filesystem found at block 0
RAMDISK: Loading 2580KiB [1 disk] into ram disk... done.
VFS: Mounted root (cramfs filesystem) readonly.
VFS: Cannot open root device "sda1" or unknown-block(0,0)
Please append a correct "root=" boot option
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

I don't know how I should proceed with debugging this.

Thanks and best regards,

Jurij Smakov                                        jurij@wooyd.org
Key: http://www.wooyd.org/pgpkey/                   KeyID: C99E03CC

Re: 2.6.11.10 failure on Hypersparc sparc32 [was: Re: sparc32 2.6 work]

[William Lee Irwin III]

On Sat, Jun 04, 2005 at 12:21:22AM -0400, Jurij Smakov wrote:
> I have tried to dig a little bit more on this issue, with some success. 
> It appears that initrd does not mount because something is wrong with the
> __copy_1page function, invoked by memcpy() for copying page-sized chunks.
> It is used in fs/cramfs/inode.c to read in the superblock of the cramfs 
> filesystem and, surprisingly, in this process first 16 bytes (containing 
> the magic cramfs number, among other things), get lost somehow. After I've 
> replaced a call to __copy_1page with a call to __memcpy in 
> include/asm-sparc/string.h, initrd mounted, however boot still fails. The 
> next problem is that /linuxrc script from initrd is not executed as it 
> should. This script is supposed to save real-root-device from /proc and 
> set the real-root-device to ramdisk, where initrd is located (it all 
> happens in do_mounts_initrd.c). However, the execve call to /linuxrc 
> quietly fails, and the script is never executed. After that kernel tries 
> to immediately mount /dev/sda1 (which is the real root) and fails, because 
> the scsi modules have not been loaded. So it now looks like this:
[...]

This is a tremendous amount of progress. I'll look at __copy_1page,
though others should do likewise as well.


-- wli

Re: 2.6.11.10 failure on Hypersparc sparc32 [was: Re: sparc32 2.6

[Bob Breuer]

William Lee Irwin III wrote:
> On Sat, Jun 04, 2005 at 12:21:22AM -0400, Jurij Smakov wrote:
> 
>>I have tried to dig a little bit more on this issue, with some success. 
>>It appears that initrd does not mount because something is wrong with the
>>__copy_1page function, invoked by memcpy() for copying page-sized chunks.
>>It is used in fs/cramfs/inode.c to read in the superblock of the cramfs 
>>filesystem and, surprisingly, in this process first 16 bytes (containing 
>>the magic cramfs number, among other things), get lost somehow. After I've 
>>replaced a call to __copy_1page with a call to __memcpy in 
>>include/asm-sparc/string.h, initrd mounted, however boot still fails. The 
>>next problem is that /linuxrc script from initrd is not executed as it 
>>should. This script is supposed to save real-root-device from /proc and 
>>set the real-root-device to ramdisk, where initrd is located (it all 
>>happens in do_mounts_initrd.c). However, the execve call to /linuxrc 
>>quietly fails, and the script is never executed. After that kernel tries 
>>to immediately mount /dev/sda1 (which is the real root) and fails, because 
>>the scsi modules have not been loaded. So it now looks like this:
> 
> [...]
> 
> This is a tremendous amount of progress. I'll look at __copy_1page,
> though others should do likewise as well.
> 

Look for hypersparc_copy_1page in arch/sparc/mm/hypersparc.S where it 
uses ASI_M_BCOPY.  I think the src and dest need to be 32 byte aligned 
in this case.

The core problem here is that cramfs uses statically allocated page 
sized buffers:
   nm -S arch/sparc/boot/image |grep read_buffers
   f022bab0 00008000 b read_buffers

Looks like cramfs also has the possibility of using a bad alignment with 
bzero_1page.

As for execve failing, can you print out the value of errno when it fails?

Bob

Re: 2.6.11.10 failure on Hypersparc sparc32 [was: Re: sparc32 2.6

[Jurij Smakov]

On Sat, 4 Jun 2005, Bob Breuer wrote:

> As for execve failing, can you print out the value of errno when it fails?

Hi Bob,

execve call does not return, so it presumably succeeds. After this piece 
in init/do_mounts_initrd.c:handle_initrd()

         pid = kernel_thread(do_linuxrc, "/linuxrc", SIGCHLD);
         if (pid > 0) {
                 while (pid != sys_wait4(-1, &i, 0, NULL))
                         yield();
         }

i has the value of 11, which according to my research indicates that it 
was terminated by SIGSEGV. If you have any ideas on how to debug it 
further, please let me know.

Best regards,

Jurij Smakov                                        jurij@wooyd.org
Key: http://www.wooyd.org/pgpkey/                   KeyID: C99E03CC