Access to older 64-bit sparcs for developers

Access to older 64-bit sparcs for developers

[Meelis Roos]

I see a lot of new development activity on sparclinux, that is great.

I also see different views on supporting older hardware on current 
kernels, and testing on old ultrasparcs seens to be one of the issues.

I have most of my sparclinux testbed in DMZ-s and I can make them 
available to any developers who want it, along with root access and 
built in management (if any).

Site 1: Ultra 1 143 MHz and Blade 100.

Third one is slowly rotating between E3500/E250/E450 because of power 
constraints so it is not stable at the moment.

Site 2: Ultra 2 (2x400), E220R (2 CPUs), T1-105, T1-200, V120, V210, 
V240, V440, V245, T1000, T2000, T5120.

I can add Netra X1 and V100 with some effort (relocating to the DMZ).

Planning to set up and add V445, T5140, E280R, Blade 1500.

In the queue of getting fixed (or rebuilt from parts of multiple 
machines): U5, U10, Blade 150, E3000, E3800, V480, V880, Netra T2000, 
Netra T5220 and Netra 240. These are all future stuff but if there is 
any recommendation on order of these, I might reshuffle them.

Remote managenet access is also possible for the machines that have a 
RSC/LOM/ALOM. For the machines that are without hardware management 
interface, restart is possible in site 2, site 1 is slow with respect to 
hands-on.

Both sites are behind IPv4 NAT and have native IPv6, so for IPv4 access, 
SSH through the gateway is needed and that requires key based access.

Please refer to ther prtconf repo for sample configs of the models, to 
see what CPUs and OF stuff there are (most models have a fixed CPU 
family but not all).

-- 
Meelis Roos (mroos@linux.ee)

Re: Access to older 64-bit sparcs for developers

[David Miller]

From: Meelis Roos <mroos@linux.ee>
Date: Tue, 27 Jun 2017 16:21:16 +0300 (EEST)

> I see a lot of new development activity on sparclinux, that is great.
> 
> I also see different views on supporting older hardware on current 
> kernels, and testing on old ultrasparcs seens to be one of the issues.
> 
> I have most of my sparclinux testbed in DMZ-s and I can make them 
> available to any developers who want it, along with root access and 
> built in management (if any).

Thank you for offering this, I hope some developers take advantage of
this opportunity.

Re: Access to older 64-bit sparcs for developers

[Julian Calaby]

Hi,

On Wed, Jun 28, 2017 at 4:42 AM, David Miller <davem@davemloft.net> wrote:
> From: Meelis Roos <mroos@linux.ee>
> Date: Tue, 27 Jun 2017 16:21:16 +0300 (EEST)
>
>> I see a lot of new development activity on sparclinux, that is great.
>>
>> I also see different views on supporting older hardware on current
>> kernels, and testing on old ultrasparcs seens to be one of the issues.
>>
>> I have most of my sparclinux testbed in DMZ-s and I can make them
>> available to any developers who want it, along with root access and
>> built in management (if any).
>
> Thank you for offering this, I hope some developers take advantage of
> this opportunity.

I'm hoping that Oracle comes back and says "we have that already, we
just don't fire up the old hardware that much".

Thanks,

-- 
Julian Calaby

Email: julian.calaby@gmail.com
Profile: http://www.google.com/profiles/julian.calaby/

Re: Access to older 64-bit sparcs for developers

[John Paul Adrian Glaubitz]

On Tue, Jun 27, 2017 at 04:21:16PM +0300, Meelis Roos wrote:
> I have most of my sparclinux testbed in DMZ-s and I can make them 
> available to any developers who want it, along with root access and 
> built in management (if any).

FWIW, we have more modern hardware available to external developers in
Debian. There is one SPARC-T5 running Debian unstable (sparc64) inside
an LDOM and a Sun Fire T2000 that are available.

I have already provided access to QEMU upstream, glibc upstream, Free
Pascal upstream and others that I forgot. The FPC people, for example,
already made great progress on adding sparc64 support to their
compiler ;).

So, if anyone needs faster sparc64 hardware running a current version
of Linux for testing, please let me know.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz@debian.org
`. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: Access to older 64-bit sparcs for developers

[chase rayfield]

"and testing on old ultrasparcs seems to be one of the issues."

Testing on old hardware is important... because there are differences
and stuff breaks. I still haven't figured out how to get an X server
going on my Ultra45 for instance... perhaps fbdev but that is
decidedly bleh. It's happily running Gentoo and my Homepage etc in the
meantime...

I can provide access to just about any 32bit hardware except
TurboSparc (working on that ;-) ), an Ultra 1E, 10, U45 and T2000, for
anyone wanting to dork around with that as well.... on a 100/10 link.
I don't have a fancy setup to power cycle them at the moment though I
can easily wire up serial consoles. I could stick a Radeon RX560 in
the U45 if someone wants to try getting mesa + X11 acceleration
working on Sparc64 on that specific hardware at least.

Chase

On Wed, Jun 28, 2017 at 4:56 AM, John Paul Adrian Glaubitz
<glaubitz@physik.fu-berlin.de> wrote:
> On Tue, Jun 27, 2017 at 04:21:16PM +0300, Meelis Roos wrote:
>> I have most of my sparclinux testbed in DMZ-s and I can make them
>> available to any developers who want it, along with root access and
>> built in management (if any).
>
> FWIW, we have more modern hardware available to external developers in
> Debian. There is one SPARC-T5 running Debian unstable (sparc64) inside
> an LDOM and a Sun Fire T2000 that are available.
>
> I have already provided access to QEMU upstream, glibc upstream, Free
> Pascal upstream and others that I forgot. The FPC people, for example,
> already made great progress on adding sparc64 support to their
> compiler ;).
>
> So, if anyone needs faster sparc64 hardware running a current version
> of Linux for testing, please let me know.
>
> Adrian
>
> --
>  .''`.  John Paul Adrian Glaubitz
> : :' :  Debian Developer - glaubitz@debian.org
> `. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
>   `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913
> --
> To unsubscribe from this list: send the line "unsubscribe sparclinux" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Access to older 64-bit sparcs for developers

[John Paul Adrian Glaubitz]

On 06/28/2017 06:32 PM, chase rayfield wrote:
> Testing on old hardware is important... because there are differences
> and stuff breaks. I still haven't figured out how to get an X server
> going on my Ultra45 for instance... perhaps fbdev but that is
> decidedly bleh. It's happily running Gentoo and my Homepage etc in the
> meantime...

There is currently no native display driver for these old framebuffers
as far as I know but I am happy to be corrected.

> I can provide access to just about any 32bit hardware except
> TurboSparc (working on that ;-) ), an Ultra 1E, 10, U45 and T2000, for
> anyone wanting to dork around with that as well.... on a 100/10 link.

UltraSparcs are 64 bits, not 32 bits. 32 bit SPARC is currently out of
scope as most people focus on UltraSparcs. No idea what the current
state of affairs is though. Have you tried a recent kernel on the
TurboSparc?

> I don't have a fancy setup to power cycle them at the moment though I
> can easily wire up serial consoles. I could stick a Radeon RX560 in
> the U45 if someone wants to try getting mesa + X11 acceleration
> working on Sparc64 on that specific hardware at least.

Having access to a serial console is always essential as networking
might not always work. Having remote power cycle control is even
better, although the T2000 does that through ALOM.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz@debian.org
`. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: Access to older 64-bit sparcs for developers

[David Miller]

From: John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de>
Date: Wed, 28 Jun 2017 19:03:04 +0200

> On 06/28/2017 06:32 PM, chase rayfield wrote:
>> Testing on old hardware is important... because there are differences
>> and stuff breaks. I still haven't figured out how to get an X server
>> going on my Ultra45 for instance... perhaps fbdev but that is
>> decidedly bleh. It's happily running Gentoo and my Homepage etc in the
>> meantime...
> 
> There is currently no native display driver for these old framebuffers
> as far as I know but I am happy to be corrected.

My Ultra45 has an ATI Radeon in it, works fine and I am playing
quake3 on it all the time.

Re: Access to older 64-bit sparcs for developers

[chase rayfield]

David, is that an XVR-100 or something newer? Mine doesn't work with
my Gentoo installation currently beyond the console... if so what
Distro/Release are you on? Maybe I can match to that and get it
working.

Also, as far as "scope" what someone does at work is obvious subject
to different "scope" than everyone working with and on Linux at large
heh... having an open minded view of what is going on kernel wise you
will undoubtedly go a long way to preventing you from being
blindsided.

Chase

On Wed, Jun 28, 2017 at 3:42 PM, David Miller <davem@davemloft.net> wrote:
> From: John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de>
> Date: Wed, 28 Jun 2017 19:03:04 +0200
>
>> On 06/28/2017 06:32 PM, chase rayfield wrote:
>>> Testing on old hardware is important... because there are differences
>>> and stuff breaks. I still haven't figured out how to get an X server
>>> going on my Ultra45 for instance... perhaps fbdev but that is
>>> decidedly bleh. It's happily running Gentoo and my Homepage etc in the
>>> meantime...
>>
>> There is currently no native display driver for these old framebuffers
>> as far as I know but I am happy to be corrected.
>
> My Ultra45 has an ATI Radeon in it, works fine and I am playing
> quake3 on it all the time.

Re: Access to older 64-bit sparcs for developers

[John Paul Adrian Glaubitz]

On 06/28/2017 10:32 PM, chase rayfield wrote:
> Also, as far as "scope" what someone does at work is obvious subject
> to different "scope" than everyone working with and on Linux at large
> heh... having an open minded view of what is going on kernel wise you
> will undoubtedly go a long way to preventing you from being
> blindsided.
Well, if you look at the majority of patches these days that are sent
to this list they mostly deal with modern sparc64 hardware and many
of them written by developers who are paid for this work. So, I think
there is definitely more a focus on modern hardware which is not
surprising. You don't see many patches for old x86 hardware in the kernel
either.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz@debian.org
`. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: Access to older 64-bit sparcs for developers

[David Miller]

From: chase rayfield <cusbrar1@gmail.com>
Date: Wed, 28 Jun 2017 16:32:06 -0400

> David, is that an XVR-100 or something newer? Mine doesn't work with
> my Gentoo installation currently beyond the console... if so what
> Distro/Release are you on? Maybe I can match to that and get it
> working.

It's a built-from-source X server on an old debian install from
about 5 years ago.

Re: Access to older 64-bit sparcs for developers

[Mark Cave-Ayland]

On 28/06/17 21:37, David Miller wrote:

> From: chase rayfield <cusbrar1@gmail.com>
> Date: Wed, 28 Jun 2017 16:32:06 -0400
> 
>> David, is that an XVR-100 or something newer? Mine doesn't work with
>> my Gentoo installation currently beyond the console... if so what
>> Distro/Release are you on? Maybe I can match to that and get it
>> working.
> 
> It's a built-from-source X server on an old debian install from
> about 5 years ago.

Is that using drm at all? As per my post a while back, I'm still
chipping away at trying to get the bochs_drm framebuffer fired up in
QEMU's sun4u.

So far I've managed to figure out that I need to add Simba support to
QEMU which is taking longer than expected as various OSs make
assumptions about the interrupt-swizzling for Ultra 5 on-board devices.

However outside of the interrupt issues, I now have something that
seemingly boots a kernel built from source and initialises the IOMMU
correctly. Now all I need to do now is figure out why the DRM memory
mapping routines never map the framebuffer to a virtual address and
always fallback to returning the physical address...


ATB,

Mark.

Re: Access to older 64-bit sparcs for developers

[David Miller]

From: Mark Cave-Ayland <mark.cave-ayland@ilande.co.uk>
Date: Thu, 29 Jun 2017 08:18:40 +0100

> Is that using drm at all?

Using DRM fully.

Re: Access to older 64-bit sparcs for developers

[Mark Cave-Ayland]

On 29/06/17 16:23, David Miller wrote:

> From: Mark Cave-Ayland <mark.cave-ayland@ilande.co.uk>
> Date: Thu, 29 Jun 2017 08:18:40 +0100
> 
>> Is that using drm at all?
> 
> Using DRM fully.

So I guess it must be something that bochs_drm is specifically doing?
With a bit more poking today I managed to get the stacktrace from QEMU
trying to initialise the bochs_drm device (please excuse the extra
debugging printks):


[   14.442315] boch addr: 1ff21000000  fbmap: 000001ff21000000  mmio:
000001ff22000000
[   14.443209] [drm] Found bochs VGA, ID 0xb0c5.
[   14.443808] [drm] Framebuffer size 16384 kB @ 0x1ff21000000, mmio @
0x1ff22000000.
[   14.455617] [TTM] Zone  kernel: Available graphics memory: 252816 kiB
[   14.457582] [TTM] Initializing pool allocator
[   14.594986] MCA drm_fb_helper_sys_imageblit
[   14.595148] MCA: sys_imageblit to 000001ff21000000
[   14.595938] Unable to handle kernel paging request at virtual address
000001ff21000000
[   14.596008] tsk->{mm,active_mm}->context = 0000000000000000
[   14.596070] tsk->{mm,active_mm}->pgd = fffff80000402000
[   14.596134]               \|/ ____ \|/
[   14.596134]               "@'/ .. \`@"
[   14.596134]               /_| \__/ |_\
[   14.596134]                  \__U_/
[   14.596197] swapper/0(1): Oops [#1]
[   14.597167] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.11.0-rc7+ #42
[   14.597397] task: fffff8001c097800 task.stack: fffff8001c09c000
[   14.597490] TSTATE: 0000000080001607 TPC: 00000000006f3afc TNPC:
00000000006f3b08 Y: 00000000    Not tainted
[   14.598443] TPC: <sys_imageblit+0x1dc/0x438>
[   14.598686] g0: 0000000000000080 g1: fffff8001c38c000 g2:
0000000000000000 g3: 0000000000000007
[   14.598753] g4: fffff8001c097800 g5: fffff8001ecea000 g6:
fffff8001c09c000 g7: 000000000090dfc0
[   14.598818] o0: 0000000000000026 o1: 0000000000000001 o2:
000000000000000f o3: 000001ff21000000
[   14.598883] o4: 0000000000000007 o5: 0000000000000008 sp:
fffff8001c09e0a1 ret_pc: 0000000000000001
[   14.599337] RPC: <0x1>
[   14.599504] l0: 0000000000000020 l1: fffff8001f807728 l2:
0000000000000080 l3: 0000000000000001
[   14.599570] l4: 0000000000b93800 l5: 0000000000000080 l6:
0000000000000030 l7: 0000000000000000
[   14.599635] i0: fffff8001c346800 i1: fffff8001c38c000 i2:
0000000000000000 i3: 0000000000000000
[   14.599701] i4: 0000000000000001 i5: 000001ff21000000 i6:
fffff8001c09e151 i7: 000000000073806c
[   14.599794] I7: <drm_fb_helper_sys_imageblit+0x14/0x34>
[   14.599973] Call Trace:
[   14.600245]  [000000000073806c] drm_fb_helper_sys_imageblit+0x14/0x34
[   14.600366]  [00000000006e6b74] soft_cursor+0x174/0x19c
[   14.600408]  [00000000006e661c] bit_cursor+0x45c/0x490
[   14.600449]  [00000000006e3124] fbcon_cursor+0x16c/0x17c
[   14.600493]  [0000000000710e38] hide_cursor+0x2c/0xa8
[   14.600532]  [0000000000711fd4] redraw_screen+0xc4/0x208
[   14.600573]  [00000000006e2200] fbcon_prepare_logo+0x288/0x358
[   14.600613]  [00000000006e26a4] fbcon_init+0x3d4/0x448
[   14.600653]  [00000000007112bc] visual_init+0xa4/0x100
[   14.600694]  [0000000000712a78] do_bind_con_driver+0x1c8/0x300
[   14.600734]  [0000000000712f24] do_take_over_console+0x170/0x198
[   14.600775]  [00000000006e279c] do_fbcon_takeover+0x84/0xe8
[   14.600826]  [00000000004747dc] notifier_call_chain+0x38/0x74
[   14.600871]  [0000000000474a5c] __blocking_notifier_call_chain+0x28/0x44
[   14.600920]  [00000000006ec0ec] register_framebuffer+0x2b8/0x2ec
[   14.600961]  [0000000000739888] drm_fb_helper_initial_config+0x2d0/0x36c


Ultimately what it comes down to is that bochs_hw_init() maps the
framebuffer like this:

  bochs->fb_map = ioremap(addr, size);
  if (bochs->fb_map = NULL) {
      DRM_ERROR("Cannot map framebuffer\n");
      return -ENOMEM;
  }

And in arch/sparc/include/asm/io_64.h:

  /* On sparc64 we have the whole physical IO address space accessible
   * using physically addressed loads and stores, so this does nothing.
   */
  static inline void __iomem *ioremap(unsigned long offset, unsigned
  long size)
  {
      return (void __iomem *)offset;
  }

This physical address bubbles all the way up to sys_imageblit() which
tries to write to the pointer returned from ioremap() which of course
faults.

Having a look at the drm documentation at
https://01.org/linuxgraphics/gfx-docs/drm/driver-api/device-io.html the
recommendation is to use ioremap() although there is also mention of a
pci_iomap() function.

Unfortunately pci_iomap() also falls back to ioremap() even for PCI
memory BARs so I'm a little unsure as to exactly where the problem lies
- any pointers?


ATB,

Mark.

Re: Access to older 64-bit sparcs for developers

[David Miller]

From: Mark Cave-Ayland <mark.cave-ayland@ilande.co.uk>
Date: Thu, 29 Jun 2017 20:32:56 +0100

> On 29/06/17 16:23, David Miller wrote:
> 
>> From: Mark Cave-Ayland <mark.cave-ayland@ilande.co.uk>
>> Date: Thu, 29 Jun 2017 08:18:40 +0100
>> 
>>> Is that using drm at all?
>> 
>> Using DRM fully.
> 
> So I guess it must be something that bochs_drm is specifically doing?

Who knows, it means that DRM hasn't been tested and tracked on sparc64
since I got the radeon working several years ago.

When I'm in front of that computer in a few weeks I can try to play
with things again.

Re: Access to older 64-bit sparcs for developers

[Mark Cave-Ayland]

On 29/06/17 21:10, David Miller wrote:

> From: Mark Cave-Ayland <mark.cave-ayland@ilande.co.uk>
> Date: Thu, 29 Jun 2017 20:32:56 +0100
> 
>> On 29/06/17 16:23, David Miller wrote:
>>
>>> From: Mark Cave-Ayland <mark.cave-ayland@ilande.co.uk>
>>> Date: Thu, 29 Jun 2017 08:18:40 +0100
>>>
>>>> Is that using drm at all?
>>>
>>> Using DRM fully.
>>
>> So I guess it must be something that bochs_drm is specifically doing?
> 
> Who knows, it means that DRM hasn't been tested and tracked on sparc64
> since I got the radeon working several years ago.
> 
> When I'm in front of that computer in a few weeks I can try to play
> with things again.

Okay. Peeking at the various other PCI drivers I see a similar pattern
with ioremap() there too, i.e.

    addr = pci_resource_start(pdev, 0);
    size = pci_resource_len(pdev, 0);
    bochs->fb_map = ioremap(addr, size);

Looking at a couple of other driver examples I see in several cases that
accesses to memory returned by ioremap() are done with
readb/readw/readl/readq accessors which seems correct for SPARC64 in
that the accesses would take place with the ASI_PHYS_BYPASS_EC_E_L ASI.

This definitely isn't the case with the blit routines in
drivers/video/fbdev/core/sysimgblt.c as they reference the address
directly via a pointer so could this be the bug? This would seem to
agree with the documentation at
https://01.org/linuxgraphics/gfx-docs/drm/driver-api/device-io.html.

Finally for reference, what was your last known good kernel for radeon?


ATB,

Mark.

Re: Access to older 64-bit sparcs for developers

[David Miller]

From: Mark Cave-Ayland <mark.cave-ayland@ilande.co.uk>
Date: Thu, 29 Jun 2017 23:09:54 +0100

> This definitely isn't the case with the blit routines in
> drivers/video/fbdev/core/sysimgblt.c as they reference the address
> directly via a pointer so could this be the bug?

That would be a bug.

Re: Access to older 64-bit sparcs for developers

[Mark Cave-Ayland]

On 29/06/17 23:09, Mark Cave-Ayland wrote:

> Okay. Peeking at the various other PCI drivers I see a similar pattern
> with ioremap() there too, i.e.
> 
>     addr = pci_resource_start(pdev, 0);
>     size = pci_resource_len(pdev, 0);
>     bochs->fb_map = ioremap(addr, size);
> 
> Looking at a couple of other driver examples I see in several cases that
> accesses to memory returned by ioremap() are done with
> readb/readw/readl/readq accessors which seems correct for SPARC64 in
> that the accesses would take place with the ASI_PHYS_BYPASS_EC_E_L ASI.
> 
> This definitely isn't the case with the blit routines in
> drivers/video/fbdev/core/sysimgblt.c as they reference the address
> directly via a pointer so could this be the bug? This would seem to
> agree with the documentation at
> https://01.org/linuxgraphics/gfx-docs/drm/driver-api/device-io.html.

So after a lot of digging I've finally found and have a potential fix
for the issue. The good news is that it's restricted to the bochs_drm
module.

The clue eventually came from working through the git history and coming
across this commit 68648ed1f58d98b8e8d994022e5e25331fbfe42a with the
following message:


fbdev: add drawing functions for framebuffers in system RAM

The generic drawing functions (cfbimgblt, cfbcopyarea, cfbfillrect)
assume that the framebuffer is in IO memory.  However, we have 3 drivers
(hecubafb, arcfb, and vfb) where the framebuffer is allocated from
system RAM (via vmalloc). Using _raw_read/write and family for these
drivers (as used in the cfb* functions) is illegal, especially in other
platforms.


Switching across to use the cfb family of helper functions in bochs_drm
ensures that the correct macros for IO memory access are used and is
enough to allow Linux to boot with a framebuffer here under
qemu-system-sparc64 with the bochs_drm module enabled (as per the
current debian 9 images):

diff --git a/drivers/gpu/drm/bochs/bochs_fbdev.c
b/drivers/gpu/drm/bochs/bochs_fbdev.c


index 932a769..d3a75c2 100644



--- a/drivers/gpu/drm/bochs/bochs_fbdev.c



+++ b/drivers/gpu/drm/bochs/bochs_fbdev.c



@@ -23,9 +23,9 @@ static int bochsfb_mmap(struct fb_info *info,



 static struct fb_ops bochsfb_ops = {
        .owner = THIS_MODULE,
        DRM_FB_HELPER_DEFAULT_OPS,
-       .fb_fillrect = drm_fb_helper_sys_fillrect,
-       .fb_copyarea = drm_fb_helper_sys_copyarea,
-       .fb_imageblit = drm_fb_helper_sys_imageblit,
+       .fb_fillrect = drm_fb_helper_cfb_fillrect,
+       .fb_copyarea = drm_fb_helper_cfb_copyarea,
+       .fb_imageblit = drm_fb_helper_cfb_imageblit,
        .fb_mmap = bochsfb_mmap,
 };

I'm not sure yet if the choice of functions will need to be
platform-specific, but as I'm running out of time before the weekend
I'll have to follow this up next week.


ATB,

Mark.