Re: Xen ARM - Exposing a PL011 to the guest

[Bhupinder Thakur <bhupinder.thakur@linaro.org>]

[-- Attachment #1: Type: text/plain, Size: 2985 bytes --]

Hi Stefano,

On 9 February 2017 at 05:40, Stefano Stabellini <sstabellini@kernel.org> wrote:
> On Wed, 8 Feb 2017, Bhupinder Thakur wrote:
>> Hi Julien,
>>
>> On 3 February 2017 at 19:38, Julien Grall <julien.grall@arm.com> wrote:
>> > So if I understand correctly, you don't receive anymore output. Correct?
>> > Have you tried to see whether the pl011 driver is receiving interrupt or
>> > even Linux calling it.
>>
>> The pl011 driver is not receiving the interrupts (I put a
>> HYPERVISOR_console_io() call inside the interrupt handler and I do not
>> see the print on the xen console) . When the console is in this state,
>> if I keep typing the characters on the console then Xen keeps raising
>> the interrupts (for each character typed) but all those interrupts are
>> marked as inflight. So it seems that interrupt is not getting cleared
>> in GIC. Which GIC function can I look at to further debug what is the
>> issue?
>>
>> Earlier, when I used to manually write characters to /dev/ttyAMA0 then
>> I used to receive the interrupts all fine.
>
> DomU doesn't get emulated PL011 interrupts from Xen? Make sure you don't
> mark the interrupt as a real hardware interrupt by mistake
> (GICH_V2_LR_HW). At the same time, check that the virtual PL011
> interrupt has been added correctly to the LRs
> (xen/arch/arm/gic-v2.c:gicv2_update_lr).

I verified that the PL011 interrupt is getting marked pending by Xen
but guest does not receive it. I am not sure if the local interrupt
delivery is disabled on the guest because of which it may not be
receiving the interrupts. Is there a way to verify that interrupts are
indeed disabled at that point. Since the interrupt delivery works fine
when I manually read/write characters to /dev/ttyAMA0, I think there
is no issue from the Xen side in terms of raising an interrupt.

One observation is that pl011_request_port () calls request_mem_region
() for the MMIO space allocated to pl011. The function returns -EBUSY
indicating that the memory region is already in use. But when I do cat
/proc/iomem, I do not see any overlaps. I am using 0x22000000 as the
start of the pl011 MMIO space. I see the same error while booting the
dom0 kernel also.

I have attached the guest boot logs when console=ttyAMA0. I have added
debug prints marked with [DEBUG]. I added debug prints in amba-pl011.c
and serial_core.c. The output from the console seems to suddenly stop
once the pl011 port is started. I have not added getty for ttyAMA0 in
the /etc/inittab file. I believe auto-serial-console should be able to
attach a getty session to ttyAMA0 automatically (i did try with adding
a getty command in the inittab file but that did not solve the issue).

Towards the end of the log, we can see that every time a character is
typed, Xen raises an interrupt and the characters keep getting
collected in the IN RING buffer (in_prod keeps incrementing). Later
interrupts remain in inflight as the earlier interrupt is not cleared.

Regards,
Bhupinder

[-- Attachment #2: pl011.log --]
[-- Type: text/x-log, Size: 14134 bytes --]


(none):/root # cat domU.cfg
kernel = "/root/Image"
name = "guest"
memory = 512
vcpus = 1
extra = "console=ttyAMA0 root=/dev/xvda ro"
disk = [ 'phy:/dev/loop0,xvda,w' ]

(none):/root # xl create domU.cfg
Parsing config from domU.cfg

(XEN) allocated xen_consumer = 1 for port = 1, remote dom=0
(XEN) searching virq in < 32, 33 >
(XEN) HVM_PARAM_VPL011_VIRQ param get val=32
(XEN) HVM_PARAM_VPL011_CONSOLE_PFN = 0x39003
(XEN) HVM_PARAM_VPL011_CONSOLE_PFN param set val=0x39003
seq 1204 queued, 'add' 'xen-backend'

Validate module index
Check if link configuration needs reloading.
seq 1204 forked new worker [1556]
seq 1204 running
IMPORT builtin 'hwdb' /usr/lib/udev/rules.d/50-udev-default.rules:15
IMPORT builtin 'hwdb' returned non-zero
RUN 'kmod load $env{MODALIAS}' /usr/lib/udev/rules.d/80-drivers.rules:5
Execute 'load' 'xen-backend:vbd'
passed device to netlink monitor 0xaaaab38dad90
seq 1204 processed
cleanup idle workers
Unload module index
Unloaded link configuration context.
worker [1556] exited
seq 1205 queued, 'add' 'xen-backend'
seq 1205 forked new worker [1614]
seq 1205 running
IMPORT builtin 'hwdb' /usr/lib/udev/rules.d/50-udev-default.rules:15
IMPORT builtin 'hwdb' returned non-zero
RUN 'kmod load $env{MODALIAS}' /usr/lib/udev/rules.d/80-drivers.rules:5
Execute 'load' 'xen-backend:console'
passed device to netlink monitor 0xaaaab38dad90
seq 1205 processed
cleanup idle workers
Unload module index
Unloaded link configuration context.
worker [1614] exited


(none):/root # xl console guest

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 4.9.0-rc3-00243-g0c183d9-dirty (btuser@blr-ubuntu-linaro) (gcc version 4.9.3 20141031 (prerelease) (Linaro GCC 2014.11) ) #94 SMP PREEMPT Mon Feb 13 12:12:48 IST 2017
[    0.000000] Boot CPU: AArch64 Processor [410fd000]
[    0.000000] Xen 4.9 support found
[    0.000000] efi: Getting EFI parameters from FDT:
[    0.000000] efi: UEFI not found.
[    0.000000] cma: Reserved 16 MiB at 0x000000005f000000
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv0.2 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: Trusted OS migration not required
[    0.000000] percpu: Embedded 21 pages/cpu @ffff80001efdc000 s48000 r8192 d29824 u86016
[    0.000000] Detected PIPT I-cache on CPU0
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 129024
[    0.000000] Kernel command line: console=ttyAMA0 root=/dev/xvda ro
[    0.000000] PID hash table entries: 2048 (order: 2, 16384 bytes)
[    0.000000] Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)
[    0.000000] Inode-cache hash table entries: 32768 (order: 6, 262144 bytes)
[    0.000000] Memory: 476280K/524288K available (8380K kernel code, 876K rwdata, 3676K rodata, 1024K init, 283K bss, 31624K reserved, 16384K cma-reserved)
[    0.000000] Virtual kernel memory layout:
[    0.000000]     modules : 0xffff000000000000 - 0xffff000008000000   (   128 MB)
[    0.000000]     vmalloc : 0xffff000008000000 - 0xffff7dffbfff0000   (129022 GB)
[    0.000000]       .text : 0xffff000008080000 - 0xffff0000088b0000   (  8384 KB)
[    0.000000]     .rodata : 0xffff0000088b0000 - 0xffff000008c50000   (  3712 KB)
[    0.000000]       .init : 0xffff000008c50000 - 0xffff000008d50000   (  1024 KB)
[    0.000000]       .data : 0xffff000008d50000 - 0xffff000008e2b200   (   877 KB)
[    0.000000]        .bss : 0xffff000008e2b200 - 0xffff000008e71ec0   (   284 KB)
[    0.000000]     fixed   : 0xffff7dfffe7fd000 - 0xffff7dfffec00000   (  4108 KB)
[    0.000000]     PCI I/O : 0xffff7dfffee00000 - 0xffff7dffffe00000   (    16 MB)
[    0.000000]     vmemmap : 0xffff7e0000000000 - 0xffff800000000000   (  2048 GB maximum)
[    0.000000]               0xffff7e0000000000 - 0xffff7e0000800000   (     8 MB actual)
[    0.000000]     memory  : 0xffff800000000000 - 0xffff800020000000   (   512 MB)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] Preemptible hierarchical RCU implementation.
[    0.000000] Build-time adjustment of leaf fanout to 64.
[    0.000000] RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=1.
[    0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=1
[    0.000000] NR_IRQS:64 nr_irqs:64 0
[    0.000000] arm_arch_timer: Architected cp15 timer(s) running at 100.00MHz (virt).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x171024e7e0, max_idle_ns: 440795205315 ns
[    0.000024] sched_clock: 56 bits at 100MHz, resolution 10ns, wraps every 4398046511100ns
[    0.000660] Console: colour dummy device 80x25
[    0.000720] Calibrating delay loop (skipped), value calculated using timer frequency.. 200.00 BogoMIPS (lpj=400000)
[    0.000771] pid_max: default: 32768 minimum: 301
[    0.001138] Security Framework initialized
[    0.001338] Mount-cache hash table entries: 1024 (order: 1, 8192 bytes)
[    0.001389] Mountpoint-cache hash table entries: 1024 (order: 1, 8192 bytes)
[    0.004333] ASID allocator initialised with 65536 entries
[    0.005545] xen:grant_table: Grant tables using version 1 layout
[    0.005653] Grant table initialized
[    0.005731] xen:events: Using FIFO-based ABI
[    0.005813] Xen: initializing cpu0
[    0.038093] EFI services will not be available.
[    0.052923] Brought up 1 CPUs
[    0.052955] SMP: Total of 1 processors activated.
[    0.052994] CPU features: detected feature: Privileged Access Never
[    0.053040] CPU features: detected feature: User Access Override
[    0.053094] CPU features: detected feature: 32-bit EL0 Support
[    0.053252] CPU: All CPU(s) started at EL1
[    0.053298] alternatives: patching kernel code
[    0.057685] devtmpfs: initialized
[    0.065647] DMI not present or invalid.
[    0.066549] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.070126] pinctrl core: initialized pinctrl subsystem
[    0.075662] NET: Registered protocol family 16
[    0.090773] vdso: 2 pages (1 code @ ffff0000088b7000, 1 data @ ffff000008d54000)
[    0.090827] hw-breakpoint: found 16 breakpoint and 16 watchpoint registers.
[    0.096374] DMA: preallocated 256 KiB pool for atomic allocations
[    0.101159] Serial: AMBA PL011 UART driver
[    0.107941] [DEBUG] DT baudrate=115200
[    0.108053] [DEBUG] pl011_setup_port: mmiobase = 0x22000000
[    0.108082] [DEBUG] registering pl011 port...
[    0.108288] [DEBUG] pl011_request_port called with return -16
[    0.108321] [DEBUG] pl011_config_port return -16
[    0.108376] 22000000.sbsa-pl011: ttyAMA0 at MMIO 0x22000000 (irq = 5, base_baud = 0) is a SBSA
[    0.951889] console [ttyAMA0] enabled
[    1.079520] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    1.126153] ACPI: Interpreter disabled.
[    1.157902] xen:balloon: Initialising balloon driver
[    1.186169] xen_balloon: Initialising balloon driver
[    1.221471] vgaarb: loaded
[    1.253097] SCSI subsystem initialized
[    1.261315] usbcore: registered new interface driver usbfs
[    1.292045] usbcore: registered new interface driver hub
[    1.294535] usbcore: registered new device driver usb
[    1.299146] pps_core: LinuxPPS API ver. 1 registered
[    1.301146] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    1.305592] PTP clock support registered
[    1.309078] dmi: Firmware registration failed.
[    1.312554] Advanced Linux Sound Architecture Driver Initialized.
[    1.321390] clocksource: Switched to clocksource arch_sys_counter
[    1.351568] VFS: Disk quotas dquot_6.6.0
[    1.354063] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    1.357800] pnp: PnP ACPI: disabled
[    1.414296] NET: Registered protocol family 2
[    1.419180] TCP established hash table entries: 4096 (order: 3, 32768 bytes)
[    1.423418] TCP bind hash table entries: 4096 (order: 4, 65536 bytes)
[    1.427725] TCP: Hash tables configured (established 4096 bind 4096)
[    1.431440] UDP hash table entries: 256 (order: 1, 8192 bytes)
[    1.434356] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
[    1.437637] NET: Registered protocol family 1
[    1.448956] RPC: Registered named UNIX socket transport module.
[    1.479221] RPC: Registered udp transport module.
[    1.482021] RPC: Registered tcp transport module.
[    1.483433] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    1.487911] kvm [1]: HYP mode not available
[    1.492819] futex hash table entries: 256 (order: 3, 32768 bytes)
[    1.554422] audit: initializing netlink subsys (disabled)
[    1.556524] audit: type=2000 audit(0.708:1): initialized
[    1.565774] workingset: timestamp_bits=46 max_order=17 bucket_order=0
[    1.671737] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    1.680715] NFS: Registering the id_resolver key type
[    1.684352] Key type id_resolver registered
[    1.687312] Key type id_legacy registered
[    1.690059] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    1.693517] 9p: Installing v9fs 9p2000 file system support
[    1.711701] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247)
[    1.715750] io scheduler noop registered
[    1.719041] io scheduler cfq registered (default)
[    1.725609] libphy: mdio_driver_register: phy-bcm-ns2-pci
[    1.760436] xen:xen_evtchn: Event-channel device installed
[    1.791998] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
[    1.804472] SuperH (H)SCI(F) driver initialized
[    1.808884] msm_serial: driver initialized
[    1.812963] Unable to detect cache hierarchy from DT for CPU 0
[    1.870954] loop: module loaded
[    1.873610] Invalid max_queues (4), will use default max: 1.
[    1.885531] hisi_sas: driver version v1.6
[    1.947514] libphy: Fixed MDIO Bus: probed
[42080.693216] xen-blkback: backend/vbd/3/51712: using 1 queues, protocol 1 (arm-abi) persistent grants
[    2.004909] tun: Universal TUN/TAP device driver, 1.6
[    2.062463] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
[    2.122473] e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k
[    2.153762] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[    2.157244] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k
[    2.161758] igb: Copyright (c) 2007-2014 Intel Corporation.
[    2.165164] igbvf: Intel(R) Gigabit Virtual Function Network Driver - version 2.4.0-k
[    2.169878] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.
[    2.173102] sky2: driver version 1.30
[    2.176530] QLogic/NetXen Network Driver v4.0.82
[    2.182603] xen_netfront: Initialising Xen virtual ethernet driver
[    2.193794] VFIO - User Level meta-driver version: 0.3
[    2.206433] blkfront: xvda: flush diskcache: enabled; persistent grants: enabled; indirect descriptors: enabled;
[    2.234777] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    2.234789] ehci-pci: EHCI PCI platform driver
[    2.235235] ehci-platform: EHCI generic platform driver
[    2.235603] ehci-exynos: EHCI EXYNOS driver
[    2.235953] ehci-msm: Qualcomm On-Chip EHCI Host Controller
[    2.237978] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    2.238005] ohci-pci: OHCI PCI platform driver
[    2.238323] ohci-platform: OHCI generic platform driver
[    2.238667] ohci-exynos: OHCI EXYNOS driver
[    2.240473] usbcore: registered new interface driver usb-storage
[    2.247037] mousedev: PS/2 mouse device common for all mice
[    2.254747] i2c /dev entries driver
[    2.264399] xen_wdt: Xen WatchDog Timer Driver v0.01
[    2.267832] xen_wdt: initialized (timeout=60s, nowayout=0)
[    2.269948] sdhci: Secure Digital Host Controller Interface driver
[    2.269957] sdhci: Copyright(c) Pierre Ossman
[    2.271099] Synopsys Designware Multimedia Card Interface Driver
[    2.273296] sdhci-pltfm: SDHCI platform and OF driver helper
[    2.276918] ledtrig-cpu: registered to indicate activity on CPUs
[    2.284412] usbcore: registered new interface driver usbhid
[    2.284420] usbhid: USB HID core driver
[    2.296057] NET: Registered protocol family 17
[    2.296431] 9pnet: Installing 9P2000 support
[    2.297058] Key type dns_resolver registered
[    2.305379] registered taskstats version 1
[    2.409868] hctosys: unable to open rtc device (rtc0)
[    2.412447] ALSA device list:
[    2.414132]   No soundcards found.
[    2.415352] [DEBUG] uart_port_startup entry with port type = 32
[    2.418303] [DEBUG] sbsa_uart_startup called...
[    2.419736] [DEBUG] pl011_allocate_irq irq=5...
[    2.421320] [DEBUG] sbsa_uart_startup done...
[    2.568210] [DEBUG] uart_port_startup exit
[    2.570649] EXT4-fs (xvda): mounting ext3 file system using the ext4 subsystem
[    2.615662] EXT4-fs (xvda): mounted filesystem with ordered data mode. Opts: (null)
[    2.644906] VFS: Mounted root (ext3 filesystem) readonly on device 202:0.
[    2.653947] devtmpfs: mounted
[    2.685517] Freeing unused kernel memory: 1024K (ffff800000c50000 - ffff800000d50000)
[    2.750920] [DEBUG] uart_wait_until_sent entry ...
[    2.780537] [DEBUG] pl011_tx_empty called with return 1
[    2.781983] [DEBUG] uart_wait_until_sent exit ...
[    2.783592] [DEBUG] uart_wait_until_sent entry ...
[    2.786313] [DEBUG] pl011_tx_empty called with return 1
[    2.787676] [DEBUG] uart_wait_until_sent exit ...
(XEN) [DEBUG]SENding guest pl011 intr 32 with intr=0x30, flag=0x80, out_cons=12494, out_prod=12494, in_cons=0, in_prod=1, status=0...
(XEN) [DEBUG]vgic_vcpu_inject_irq: raised
(XEN) [DEBUG]gic_raise_guest_irq: pending
(XEN) [DEBUG]irq 32 marked lr pending
(XEN) [DEBUG]setting lr 1 for irq 32
(XEN) [DEBUG]gicv2_update_lr: 1, 1
(XEN) [DEBUG]SENding guest pl011 intr 32 with intr=0x30, flag=0x80, out_cons=12494, out_prod=12494, in_cons=0, in_prod=2, status=0...
(XEN) [DEBUG]vgic_vcpu_inject_irq: inflight

[-- Attachment #3: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel