Re: [PATCH for-5.1 V4 3/4] hw/mips: Add Loongson-3 machine support (with KVM)

[Jiaxun Yang <jiaxun.yang@flygoat.com>]

在 2020/6/11 15:49, Huacai Chen 写道:
> Hi, Jiaxun,
> 
> On Thu, Jun 11, 2020 at 1:59 PM Jiaxun Yang <jiaxun.yang@flygoat.com> wrote:
>>
>>
>>
>> 在 2020/6/2 10:39, Huacai Chen 写道:
>>> Add Loongson-3 based machine support, it use i8259 as the interrupt
>>> controler and use GPEX as the pci controller. Currently it can only
>>> work with KVM, but we will add TCG support in future.
>>>
>>> We already have a full functional Linux kernel (based on Linux-5.4.x LTS
>>> but not upstream yet) here:
>>>
>>> https://github.com/chenhuacai/linux
>>>
>>> How to use QEMU/Loongson-3?
>>> 1, Download kernel source from the above URL;
>>> 2, Build a kernel with arch/mips/configs/loongson3_{def,hpc}config;
>>> 3, Boot the a Loongson-3A4000 host with this kernel;
>>> 4, Build QEMU-5.0.0 with this patchset;
>>> 5, modprobe kvm;
>>> 6, Use QEMU with TCG (available in future):
>>>          qemu-system-mips64el -M loongson3,accel=tcg -cpu Loongson-3A1000 -kernel <path_to_kernel> -append ...
>>>      Use QEMU with KVM (available at present):
>>>          qemu-system-mips64el -M loongson3,accel=kvm -cpu Loongson-3A4000 -kernel <path_to_kernel> -append ...
>>>
>>>      The "-cpu" parameter can be omitted here and QEMU will use the correct type for TCG/KVM automatically.
>>>
>>> Signed-off-by: Huacai Chen <chenhc@lemote.com>
>>> Co-developed-by: Jiaxun Yang <jiaxun.yang@flygoat.com>
>>> Reviewed-by: Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
>>> ---
>>>    default-configs/mips64el-softmmu.mak |   1 +
>>>    hw/mips/Kconfig                      |  10 +
>>>    hw/mips/Makefile.objs                |   1 +
>>>    hw/mips/loongson3.c                  | 901 +++++++++++++++++++++++++++++++++++
>>>    4 files changed, 913 insertions(+)
>>>    create mode 100644 hw/mips/loongson3.c
>>
>> Hi there,
>>
>> I was working on TCG support based on this machine, and noticed some
>> minor issue here.
>>
>> Huacai, would you mind me to include your machine support in my TCG
>> series? As currently KVM support is blocked kernel.
>>
>>

[...]

>>> +
>>> +static int get_host_cpu_freq(void)
>>> +{
>>
>> "model name" have not been accppted by mainline kernel.
>> Probably asking kernel frequency via a part of QEMU IOCTL is a better
>> option?
> Is there exsiting better method? I will use if possible.

CPUCFG instruction introduced by 3A4000 have CCfreq domain to describe 
the frequency.
Or we can add it to kernel's data structure?

> 
>>
>>> +    int fd = 0, freq = 0;
>>> +    char buf[1024], *buf_p;
>>> +
>>> +    fd = open("/proc/cpuinfo", O_RDONLY);
>>> +    if (fd == -1) {
>>> +        fprintf(stderr, "Failed to open /proc/cpuinfo!\n");
>>> +        return 0;
>>> +    }
>>> +
>>> +    if (read(fd, buf, 1024) < 0) {
>>> +        close(fd);
>>> +        fprintf(stderr, "Failed to read /proc/cpuinfo!\n");
>>> +        return 0;
>>> +    }
>>> +    close(fd);
>>> +
>>> +    buf_p = strstr(buf, "model name");
>>> +    while (*buf_p != '@') {
>>> +        buf_p++;
>>> +    }
>>> +
>>> +    buf_p += 2;
>>> +    memcpy(buf, buf_p, 12);
>>> +    buf_p = buf;
>>> +    while ((*buf_p >= '0') && (*buf_p <= '9')) {
>>> +        buf_p++;
>>> +    }
>>> +    *buf_p = '\0';
>>> +
>>> +    freq = atoi(buf);
>>> +
>>> +    return freq * 1000 * 1000;
>>> +}

[...]

>>> +
>>> +    setenv("memsize", memenv, 1);
>>> +    setenv("highmemsize", highmemenv, 1);
>>
>> These setenv looks pointless.
> memsize and highmemsize is standard MIPS kernel parameters.

By setenv you're not adding it to kernel parameter but adding it to 
*host's* environment variable.

Also memory size have already been passed by loongson's boot_param 
structure so no need to supply it in env again.

> 
>>
>>> +
>>> +    ret = ((ret + 32) & ~31);
>>> +
>>> +    boot_params_buf = (void *)(params_buf + ret);
>>> +    boot_params_p = boot_params_buf + align(sizeof(struct boot_params));
>>> +
>>> +    init_boot_param(boot_params_buf);
>>> +
>>> +    rom_add_blob_fixed("params", params_buf, params_size,
>>> +                       BOOTPARAM_PHYADDR);
>>> +    loaderparams.a0 = 2;
>>> +    loaderparams.a1 = 0xffffffff80000000ULL + BOOTPARAM_PHYADDR;
>>> +    loaderparams.a2 = 0xffffffff80000000ULL + BOOTPARAM_PHYADDR + ret;
>>> +
>>> +    return 0;
>>> +}
>>> +
>>> +static int64_t load_kernel(CPUMIPSState *env)
>>> +{
>>> +    long kernel_size;
>>> +    ram_addr_t initrd_offset;
>>> +    int64_t kernel_entry, kernel_low, kernel_high, initrd_size;
>>> +
>>> +    kernel_size = load_elf(loaderparams.kernel_filename, NULL,
>>> +                           cpu_mips_kseg0_to_phys, NULL,
>>> +                           (uint64_t *)&kernel_entry,
>>> +                           (uint64_t *)&kernel_low, (uint64_t *)&kernel_high,
>>> +                           NULL, 0, EM_MIPS, 1, 0);
>>> +    if (kernel_size < 0) {
>>> +        error_report("could not load kernel '%s': %s",
>>> +                     loaderparams.kernel_filename,
>>> +                     load_elf_strerror(kernel_size));
>>> +        exit(1);
>>> +    }
>>> +
>>> +    /* load initrd */
>>> +    initrd_size = 0;
>>> +    initrd_offset = 0;
>>> +    if (loaderparams.initrd_filename) {
>>> +        initrd_size = get_image_size(loaderparams.initrd_filename);
>>> +        if (initrd_size > 0) {
>>> +            initrd_offset = (kernel_high + ~INITRD_PAGE_MASK) &
>>> +                            INITRD_PAGE_MASK;
>>> +            initrd_offset = MAX(initrd_offset, INITRD_OFFSET);
>>> +
>>> +            if (initrd_offset + initrd_size > ram_size) {
>>> +                error_report("memory too small for initial ram disk '%s'",
>>> +                             loaderparams.initrd_filename);
>>> +                exit(1);
>>> +            }
>>> +
>>> +            initrd_size = load_image_targphys(loaderparams.initrd_filename,
>>> +                                              initrd_offset,
>>> +                                              ram_size - initrd_offset);
>>> +        }
>>> +
>>> +        if (initrd_size == (target_ulong) -1) {
>>> +            error_report("could not load initial ram disk '%s'",
>>> +                         loaderparams.initrd_filename);
>>> +            exit(1);
>>> +        }
>>> +    }
>>> +
>>> +    /* Setup prom parameters. */
>>> +    set_prom_bootparam(initrd_offset, initrd_size);
>>> +
>>> +    return kernel_entry;
>>> +}
>>> +
>>> +static void main_cpu_reset(void *opaque)
>>> +{
>>> +    MIPSCPU *cpu = opaque;
>>> +    CPUMIPSState *env = &cpu->env;
>>> +
>>> +    cpu_reset(CPU(cpu));
>>> +
>>> +    /* Loongson-3 reset stuff */
>>> +    if (loaderparams.kernel_filename) {
>>> +        if (cpu == MIPS_CPU(first_cpu)) {
>>> +            env->active_tc.gpr[4] = loaderparams.a0;
>>> +            env->active_tc.gpr[5] = loaderparams.a1;
>>> +            env->active_tc.gpr[6] = loaderparams.a2;
>>> +            env->active_tc.PC = loaderparams.kernel_entry;
>>> +        }
>>> +        env->CP0_Status &= ~((1 << CP0St_BEV) | (1 << CP0St_ERL));
>>> +    }
>>> +}
>>> +
>>> +static void loongson3_isa_init(qemu_irq intc)
>>> +{
>>> +    qemu_irq *i8259;
>>> +    ISABus *isa_bus;
>>> +
>>> +    isa_bus = isa_bus_new(NULL, get_system_memory(), get_system_io(), &error_abort);
>>> +
>>> +    /* Interrupt controller */
>>> +    /* The 8259 -> IP3  */
>>> +    i8259 = i8259_init(isa_bus, intc);
>>> +    isa_bus_irqs(isa_bus, i8259);
>>> +    /* init other devices */
>>> +    isa_create_simple(isa_bus, "i8042");
>>> +    mc146818_rtc_init(isa_bus, 2000, NULL);
>>> +}
>>> +
>>> +static inline void loongson3_pcie_init(MachineState *machine, DeviceState *pic)
>>> +{
>>> +    int i;
>>> +    qemu_irq irq;
>>> +    PCIBus *pci_bus;
>>> +    DeviceState *dev;
>>> +    MemoryRegion *pio_alias;
>>> +    MemoryRegion *mmio_alias, *mmio_reg;
>>> +    MemoryRegion *ecam_alias, *ecam_reg;
>>> +
>>> +    dev = qdev_create(NULL, TYPE_GPEX_HOST);
>>> +
>>> +    qdev_init_nofail(dev);
>>> +    pci_bus = PCI_HOST_BRIDGE(dev)->bus;
>>> +
>>> +    ecam_alias = g_new0(MemoryRegion, 1);
>>> +    ecam_reg = sysbus_mmio_get_region(SYS_BUS_DEVICE(dev), 0);
>>> +    memory_region_init_alias(ecam_alias, OBJECT(dev), "pcie-ecam",
>>> +                             ecam_reg, 0, VIRT_PCI_ECAM_SIZE);
>>> +    memory_region_add_subregion(get_system_memory(), VIRT_PCI_ECAM_BASE, ecam_alias);
>>> +
>>> +    mmio_alias = g_new0(MemoryRegion, 1);
>>> +    mmio_reg = sysbus_mmio_get_region(SYS_BUS_DEVICE(dev), 1);
>>> +    memory_region_init_alias(mmio_alias, OBJECT(dev), "pcie-mmio",
>>> +                             mmio_reg, VIRT_PCI_MEM_BASE, VIRT_PCI_MEM_SIZE);
>>> +    memory_region_add_subregion(get_system_memory(), VIRT_PCI_MEM_BASE, mmio_alias);
>>> +
>>> +    pio_alias = g_new0(MemoryRegion, 1);
>>> +    memory_region_init_alias(pio_alias, OBJECT(dev), "pcie-pio",
>>> +                             get_system_io(), 0, VIRT_PCI_IO_SIZE);
>>> +    memory_region_add_subregion(get_system_memory(), VIRT_PCI_IO_BASE, pio_alias);
>>> +    sysbus_mmio_map(SYS_BUS_DEVICE(dev), 2, VIRT_PCI_IO_BASE);
>>> +
>>> +    for (i = 0; i < GPEX_NUM_IRQS; i++) {
>>> +        irq = qdev_get_gpio_in(pic, PCIE_IRQ_BASE + i);
>>> +        sysbus_connect_irq(SYS_BUS_DEVICE(dev), i, irq);
>>> +        gpex_set_irq_num(GPEX_HOST(dev), i, PCIE_IRQ_BASE + i);
>>> +    }
>>> +
>>> +    pci_vga_init(pci_bus);
>>> +
>>> +    for (i = 0; i < nb_nics; i++) {
>>> +        NICInfo *nd = &nd_table[i];
>>> +
>>> +        if (!nd->model) {
>>> +            nd->model = g_strdup("virtio");
>>> +        }
>>> +
>>> +        pci_nic_init_nofail(nd, pci_bus, nd->model, NULL);
>>> +    }
>>> +}
>>> +
>>> +static void mips_loongson3_init(MachineState *machine)
>>> +{
>>> +    int i;
>>> +    long bios_size;
>>> +    MIPSCPU *cpu;
>>> +    CPUMIPSState *env;
>>> +    char *filename;
>>> +    const char *kernel_cmdline = machine->kernel_cmdline;
>>> +    const char *kernel_filename = machine->kernel_filename;
>>> +    const char *initrd_filename = machine->initrd_filename;
>>> +    ram_addr_t ram_size = machine->ram_size;
>>> +    MemoryRegion *address_space_mem = get_system_memory();
>>> +    MemoryRegion *ram = g_new(MemoryRegion, 1);
>>> +    MemoryRegion *bios = g_new(MemoryRegion, 1);
>>> +    MemoryRegion *iomem = g_new(MemoryRegion, 1);
>>> +
>>> +    if (!kvm_enabled()) {
>>> +        if (!machine->cpu_type) {
>>> +            machine->cpu_type = MIPS_CPU_TYPE_NAME("Loongson-3A1000");
>>> +        }
>>> +        if (!strstr(machine->cpu_type, "Loongson-3A1000")) {
>>> +            error_report("Loongson-3/TCG need cpu type Loongson-3A1000");
>>> +            exit(1);
>>> +        }
>>> +    } else {
>>> +        if (!machine->cpu_type) {
>>> +            machine->cpu_type = MIPS_CPU_TYPE_NAME("Loongson-3A4000");
>>> +        }
>>> +        if (!strstr(machine->cpu_type, "Loongson-3A4000")) {
>>> +            error_report("Loongson-3/KVM need cpu type Loongson-3A4000");
>>> +            exit(1);
>>> +        }
>>> +    }
>>> +
>>> +    if (ram_size < 256 * 0x100000) {
>>> +        error_report("Loongson-3 need at least 256MB memory");
>>> +        exit(1);
>>> +    }
>>> +
>>> +    for (i = 0; i < machine->smp.cpus; i++) {
>>> +        /* init CPUs */
>>> +        cpu = MIPS_CPU(cpu_create(machine->cpu_type));
>>> +
>>> +        /* Init internal devices */
>>> +        cpu_mips_irq_init_cpu(cpu);
>>> +        cpu_mips_clock_init(cpu);
>>> +        qemu_register_reset(main_cpu_reset, cpu);
>>> +    }
>>> +    env = &MIPS_CPU(first_cpu)->env;
>>> +
>>> +    /* Allocate RAM/BIOS, 0x00000000~0x10000000 is alias of 0x80000000~0x90000000 */
>>> +    memory_region_init_rom(bios, NULL, "loongson3.bios",
>>> +                           BIOS_SIZE, &error_fatal);
>>> +    memory_region_init_alias(ram, NULL, "loongson3.lowram",
>>> +                           machine->ram, 0, 256 * 0x100000);
>>> +    memory_region_init_io(iomem, NULL, &loongson3_pm_ops,
>>> +                           NULL, "loongson3_pm", PM_MMIO_SIZE);
>>> +
>>> +    memory_region_add_subregion(address_space_mem, 0x00000000LL, ram);
>>> +    memory_region_add_subregion(address_space_mem, 0x1fc00000LL, bios);
>>> +    memory_region_add_subregion(address_space_mem, 0x80000000LL, machine->ram);
>>> +    memory_region_add_subregion(address_space_mem, PM_MMIO_ADDR, iomem);
>>> +
>>> +    /*
>>> +     * We do not support flash operation, just loading pmon.bin as raw BIOS.
>>> +     * Please use -L to set the BIOS path and -bios to set bios name.
>>> +     */
>>> +
>>> +    if (kernel_filename) {
>>> +        loaderparams.ram_size = ram_size;
>>> +        loaderparams.kernel_filename = kernel_filename;
>>> +        loaderparams.kernel_cmdline = kernel_cmdline;
>>> +        loaderparams.initrd_filename = initrd_filename;
>>> +        loaderparams.kernel_entry = load_kernel(env);
>>> +        rom_add_blob_fixed("bios",
>>> +                         bios_boot_code, sizeof(bios_boot_code), 0x1fc00000LL);
>>> +    } else {
>>> +        if (bios_name == NULL) {
>>> +                bios_name = LOONGSON3_BIOSNAME;
>>> +        }
>>> +        filename = qemu_find_file(QEMU_FILE_TYPE_BIOS, bios_name);
>>> +        if (filename) {
>>> +            bios_size = load_image_targphys(filename, 0x1fc00000LL,
>>> +                                            BIOS_SIZE);
>>> +            g_free(filename);
>>> +        } else {
>>> +            bios_size = -1;
>>> +        }
>>> +
>>> +        if ((bios_size < 0 || bios_size > BIOS_SIZE) &&
>>> +            !kernel_filename && !qtest_enabled()) {
>>> +            error_report("Could not load MIPS bios '%s'", bios_name);
>>> +            exit(1);
>>> +        }
>>> +
>>> +        fw_conf_init(ram_size);
>>> +        rom_add_blob_fixed("fw_conf",
>>> +                         &fw_config, sizeof(fw_config), FW_CONF_ADDR);
>>> +    }
>>> +
>>> +    msi_nonbroken = true;
>>
>> As this machine is not reflecting any actual Loongson-3 system, I would
>> say "loongson3-virt" can be a better name.
> I think "loongson3" can be a generic name, we don't need to show
> "virt" explicitly.
> 
>>
>>> +    loongson3_isa_init(env->irq[3]);
>>> +    loongson3_pcie_init(machine, isa_pic);
>>> +
>>> +    if (serial_hd(0)) {
>>> +        serial_mm_init(address_space_mem, 0x1fe001e0, 0, env->irq[2],
>>> +                           115200, serial_hd(0), DEVICE_NATIVE_ENDIAN);
>>> +    }
>>> +}
>>> +
>>> +static void mips_loongson3_machine_init(MachineClass *mc)
>>> +{
>>> +    mc->desc = "Generic Loongson-3 Platform";
>>> +    mc->init = mips_loongson3_init;
>>> +    mc->block_default_type = IF_IDE;
>>> +    mc->max_cpus = LOONGSON_MAX_VCPUS;
>>> +    mc->default_ram_id = "loongson3.highram";
>>> +    mc->default_ram_size = 1200 * MiB;
>>
>> 1200MiB looks wired... Why not 1024?
> Oh, it is just because our Fedora28 needs more than 1024MB to work
> fine, maybe 1280 is better?

Ahh if that's the reason then it looks fine for me.

> 
>>
>>> +    mc->kvm_type = mips_kvm_type;
>>> +    mc->minimum_page_bits = 14;
>>> +}
>>> +
>>> +DEFINE_MACHINE("loongson3", mips_loongson3_machine_init)
>>>
>>
>> As this machine is not reflecting any actual Loongson-3 system, I would
>> say "loongson3-virt" can be a better name
>>
>> Furthermore, the design of machine can be updated over time. So probably
>> we can add a version suffix to it just like what arm-virt did.
>>
>> Thanks!
>>
>> --
>> - Jiaxun
>>

-- 
- Jiaxun