[PATCH 0/6] Modify some registers operations and move decode_cpucfg() to loongson_regs.h

[PATCH 0/6] Modify some registers operations and move decode_cpucfg() to loongson_regs.h

[Tiezhu Yang]

Tiezhu Yang (6):
  MIPS: Loongson64: Do not write the read only field LPA of CP0_CONFIG3
  MIPS: Loongson64: Set the field ELPA of CP0_PAGEGRAIN only once
  MIPS: Loongson64: Set IPI_Enable register per core by itself
  MIPS: Loongson64: Add Mail_Send support for 3A4000+ CPU
  MIPS: Loongson64: Make sure the PC address is correct when 3A4000+ CPU
    hotplug
  MIPS: Loongson64: Move decode_cpucfg() to loongson_regs.h

 .../asm/mach-loongson64/kernel-entry-init.h        |  8 ---
 .../include/asm/mach-loongson64/loongson_regs.h    | 34 ++++++++++
 arch/mips/kernel/cpu-probe.c                       | 31 +--------
 arch/mips/loongson64/numa.c                        | 20 +-----
 arch/mips/loongson64/smp.c                         | 78 +++++++++++++++++-----
 5 files changed, 97 insertions(+), 74 deletions(-)

-- 
2.1.0

[PATCH 1/6] MIPS: Loongson64: Do not write the read only field LPA of CP0_CONFIG3

[Tiezhu Yang]

The field LPA of CP0_CONFIG3 register is read only for Loongson64, so the
write operations are meaningless, remove them.

Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
---
 arch/mips/include/asm/mach-loongson64/kernel-entry-init.h | 8 --------
 arch/mips/loongson64/numa.c                               | 3 ---
 2 files changed, 11 deletions(-)

diff --git a/arch/mips/include/asm/mach-loongson64/kernel-entry-init.h b/arch/mips/include/asm/mach-loongson64/kernel-entry-init.h
index 87a5bfb..e4d77f4 100644
--- a/arch/mips/include/asm/mach-loongson64/kernel-entry-init.h
+++ b/arch/mips/include/asm/mach-loongson64/kernel-entry-init.h
@@ -19,10 +19,6 @@
 	.macro	kernel_entry_setup
 	.set	push
 	.set	mips64
-	/* Set LPA on LOONGSON3 config3 */
-	mfc0	t0, CP0_CONFIG3
-	or	t0, (0x1 << 7)
-	mtc0	t0, CP0_CONFIG3
 	/* Set ELPA on LOONGSON3 pagegrain */
 	mfc0	t0, CP0_PAGEGRAIN
 	or	t0, (0x1 << 29)
@@ -54,10 +50,6 @@
 	.macro	smp_slave_setup
 	.set	push
 	.set	mips64
-	/* Set LPA on LOONGSON3 config3 */
-	mfc0	t0, CP0_CONFIG3
-	or	t0, (0x1 << 7)
-	mtc0	t0, CP0_CONFIG3
 	/* Set ELPA on LOONGSON3 pagegrain */
 	mfc0	t0, CP0_PAGEGRAIN
 	or	t0, (0x1 << 29)
diff --git a/arch/mips/loongson64/numa.c b/arch/mips/loongson64/numa.c
index cf9459f..c7e3cced 100644
--- a/arch/mips/loongson64/numa.c
+++ b/arch/mips/loongson64/numa.c
@@ -40,9 +40,6 @@ static void enable_lpa(void)
 	unsigned long value;
 
 	value = __read_32bit_c0_register($16, 3);
-	value |= 0x00000080;
-	__write_32bit_c0_register($16, 3, value);
-	value = __read_32bit_c0_register($16, 3);
 	pr_info("CP0_Config3: CP0 16.3 (0x%lx)\n", value);
 
 	value = __read_32bit_c0_register($5, 1);
-- 
2.1.0

[PATCH 2/6] MIPS: Loongson64: Set the field ELPA of CP0_PAGEGRAIN only once

[Tiezhu Yang]

The field ELPA of CP0_PAGEGRAIN register is set at the beginning
of the kernel entry point in kernel-entry-init.h, no need to set
it again in numa.c, we can remove enable_lpa() and only print the
related information.

Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
---
 arch/mips/loongson64/numa.c | 17 ++---------------
 1 file changed, 2 insertions(+), 15 deletions(-)

diff --git a/arch/mips/loongson64/numa.c b/arch/mips/loongson64/numa.c
index c7e3cced..509b360 100644
--- a/arch/mips/loongson64/numa.c
+++ b/arch/mips/loongson64/numa.c
@@ -35,20 +35,6 @@ EXPORT_SYMBOL(__node_data);
 cpumask_t __node_cpumask[MAX_NUMNODES];
 EXPORT_SYMBOL(__node_cpumask);
 
-static void enable_lpa(void)
-{
-	unsigned long value;
-
-	value = __read_32bit_c0_register($16, 3);
-	pr_info("CP0_Config3: CP0 16.3 (0x%lx)\n", value);
-
-	value = __read_32bit_c0_register($5, 1);
-	value |= 0x20000000;
-	__write_32bit_c0_register($5, 1, value);
-	value = __read_32bit_c0_register($5, 1);
-	pr_info("CP0_PageGrain: CP0 5.1 (0x%lx)\n", value);
-}
-
 static void cpu_node_probe(void)
 {
 	int i;
@@ -240,7 +226,8 @@ EXPORT_SYMBOL(pcibus_to_node);
 
 void __init prom_init_numa_memory(void)
 {
-	enable_lpa();
+	pr_info("CP0_Config3: CP0 16.3 (0x%x)\n", read_c0_config3());
+	pr_info("CP0_PageGrain: CP0 5.1 (0x%x)\n", read_c0_pagegrain());
 	prom_meminit();
 }
 EXPORT_SYMBOL(prom_init_numa_memory);
-- 
2.1.0

[PATCH 3/6] MIPS: Loongson64: Set IPI_Enable register per core by itself

[Tiezhu Yang]

In the current code, for example, core 1 sets Core[0, 1, 2, 3]_IPI_Enalbe
register and core 2, 3 do the same thing on the 1-way Loongson64 platform,
this is not necessary. Set IPI_Enable register per core by itself to avoid
duplicate operations and make the logic more clear.

Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
---
 arch/mips/loongson64/smp.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/arch/mips/loongson64/smp.c b/arch/mips/loongson64/smp.c
index e744e1b..7d58853 100644
--- a/arch/mips/loongson64/smp.c
+++ b/arch/mips/loongson64/smp.c
@@ -348,8 +348,7 @@ static void loongson3_init_secondary(void)
 	/* Set interrupt mask, but don't enable */
 	change_c0_status(ST0_IM, imask);
 
-	for (i = 0; i < num_possible_cpus(); i++)
-		loongson3_ipi_write32(0xffffffff, ipi_en0_regs[cpu_logical_map(i)]);
+	loongson3_ipi_write32(0xffffffff, ipi_en0_regs[cpu_logical_map(cpu)]);
 
 	per_cpu(cpu_state, cpu) = CPU_ONLINE;
 	cpu_set_core(&cpu_data[cpu],
@@ -420,6 +419,7 @@ static void __init loongson3_smp_setup(void)
 	ipi_status0_regs_init();
 	ipi_en0_regs_init();
 	ipi_mailbox_buf_init();
+	loongson3_ipi_write32(0xffffffff, ipi_en0_regs[cpu_logical_map(0)]);
 	cpu_set_core(&cpu_data[0],
 		     cpu_logical_map(0) % loongson_sysconf.cores_per_package);
 	cpu_data[0].package = cpu_logical_map(0) / loongson_sysconf.cores_per_package;
-- 
2.1.0

[PATCH 4/6] MIPS: Loongson64: Add Mail_Send support for 3A4000+ CPU

[Tiezhu Yang]

Loongson 3A4000+ CPU has per-core Mail_Send register to send mail,
there is no need to maintain register address of each core and node,
just simply specify cpu number.

Signed-off-by: Lu Zeng <zenglu@loongson.cn>
Signed-off-by: Jianmin Lv <lvjianmin@loongson.cn>
Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
---
 .../include/asm/mach-loongson64/loongson_regs.h    | 10 +++
 arch/mips/loongson64/smp.c                         | 75 +++++++++++++++++-----
 2 files changed, 68 insertions(+), 17 deletions(-)

diff --git a/arch/mips/include/asm/mach-loongson64/loongson_regs.h b/arch/mips/include/asm/mach-loongson64/loongson_regs.h
index 83dbb9f..1659935 100644
--- a/arch/mips/include/asm/mach-loongson64/loongson_regs.h
+++ b/arch/mips/include/asm/mach-loongson64/loongson_regs.h
@@ -227,6 +227,16 @@ static inline void csr_writeq(u64 val, u32 reg)
 #define CSR_IPI_SEND_CPU_SHIFT	16
 #define CSR_IPI_SEND_BLOCK	BIT(31)
 
+#define LOONGSON_CSR_MAIL_BUF0		0x1020
+#define LOONGSON_CSR_MAIL_SEND		0x1048
+#define CSR_MAIL_SEND_BLOCK		BIT_ULL(31)
+#define CSR_MAIL_SEND_BOX_LOW(box)	(box << 1)
+#define CSR_MAIL_SEND_BOX_HIGH(box)	((box << 1) + 1)
+#define CSR_MAIL_SEND_BOX_SHIFT		2
+#define CSR_MAIL_SEND_CPU_SHIFT		16
+#define CSR_MAIL_SEND_BUF_SHIFT		32
+#define CSR_MAIL_SEND_H32_MASK		0xFFFFFFFF00000000ULL
+
 static inline u64 drdtime(void)
 {
 	int rID = 0;
diff --git a/arch/mips/loongson64/smp.c b/arch/mips/loongson64/smp.c
index 7d58853..33597d2 100644
--- a/arch/mips/loongson64/smp.c
+++ b/arch/mips/loongson64/smp.c
@@ -54,6 +54,26 @@ static uint32_t core0_c0count[NR_CPUS];
 u32 (*ipi_read_clear)(int cpu);
 void (*ipi_write_action)(int cpu, u32 action);
 
+/* send mail via Mail_Send register for 3A4000+ CPU */
+static void csr_mail_send(uint64_t data, int cpu, int mailbox)
+{
+	uint64_t val;
+
+	/* send high 32 bits */
+	val = CSR_MAIL_SEND_BLOCK;
+	val |= (CSR_MAIL_SEND_BOX_HIGH(mailbox) << CSR_MAIL_SEND_BOX_SHIFT);
+	val |= (cpu << CSR_MAIL_SEND_CPU_SHIFT);
+	val |= (data & CSR_MAIL_SEND_H32_MASK);
+	csr_writeq(val, LOONGSON_CSR_MAIL_SEND);
+
+	/* send low 32 bits */
+	val = CSR_MAIL_SEND_BLOCK;
+	val |= (CSR_MAIL_SEND_BOX_LOW(mailbox) << CSR_MAIL_SEND_BOX_SHIFT);
+	val |= (cpu << CSR_MAIL_SEND_CPU_SHIFT);
+	val |= (data << CSR_MAIL_SEND_BUF_SHIFT);
+	csr_writeq(val, LOONGSON_CSR_MAIL_SEND);
+};
+
 static u32 csr_ipi_read_clear(int cpu)
 {
 	u32 action;
@@ -348,7 +368,10 @@ static void loongson3_init_secondary(void)
 	/* Set interrupt mask, but don't enable */
 	change_c0_status(ST0_IM, imask);
 
-	loongson3_ipi_write32(0xffffffff, ipi_en0_regs[cpu_logical_map(cpu)]);
+	if (cpu_has_csr() && csr_readl(LOONGSON_CSR_FEATURES) & LOONGSON_CSRF_IPI)
+		csr_writel(0xffffffff, LOONGSON_CSR_IPI_EN);
+	else
+		loongson3_ipi_write32(0xffffffff, ipi_en0_regs[cpu_logical_map(cpu)]);
 
 	per_cpu(cpu_state, cpu) = CPU_ONLINE;
 	cpu_set_core(&cpu_data[cpu],
@@ -380,8 +403,12 @@ static void loongson3_smp_finish(void)
 
 	write_c0_compare(read_c0_count() + mips_hpt_frequency/HZ);
 	local_irq_enable();
-	loongson3_ipi_write64(0,
-			ipi_mailbox_buf[cpu_logical_map(cpu)] + 0x0);
+
+	if (cpu_has_csr() && csr_readl(LOONGSON_CSR_FEATURES) & LOONGSON_CSRF_IPI)
+		csr_writel(0, LOONGSON_CSR_MAIL_BUF0);
+	else
+		loongson3_ipi_write64(0, ipi_mailbox_buf[cpu_logical_map(cpu)] + 0x0);
+
 	pr_info("CPU#%d finished, CP0_ST=%x\n",
 			smp_processor_id(), read_c0_status());
 }
@@ -414,12 +441,18 @@ static void __init loongson3_smp_setup(void)
 	}
 
 	csr_ipi_probe();
-	ipi_set0_regs_init();
-	ipi_clear0_regs_init();
-	ipi_status0_regs_init();
-	ipi_en0_regs_init();
-	ipi_mailbox_buf_init();
-	loongson3_ipi_write32(0xffffffff, ipi_en0_regs[cpu_logical_map(0)]);
+
+	if (cpu_has_csr() && csr_readl(LOONGSON_CSR_FEATURES) & LOONGSON_CSRF_IPI) {
+		csr_writel(0xffffffff, LOONGSON_CSR_IPI_EN);
+	} else {
+		ipi_set0_regs_init();
+		ipi_clear0_regs_init();
+		ipi_status0_regs_init();
+		ipi_en0_regs_init();
+		ipi_mailbox_buf_init();
+		loongson3_ipi_write32(0xffffffff, ipi_en0_regs[cpu_logical_map(0)]);
+	}
+
 	cpu_set_core(&cpu_data[0],
 		     cpu_logical_map(0) % loongson_sysconf.cores_per_package);
 	cpu_data[0].package = cpu_logical_map(0) / loongson_sysconf.cores_per_package;
@@ -452,14 +485,22 @@ static int loongson3_boot_secondary(int cpu, struct task_struct *idle)
 	pr_debug("CPU#%d, func_pc=%lx, sp=%lx, gp=%lx\n",
 			cpu, startargs[0], startargs[1], startargs[2]);
 
-	loongson3_ipi_write64(startargs[3],
-			ipi_mailbox_buf[cpu_logical_map(cpu)] + 0x18);
-	loongson3_ipi_write64(startargs[2],
-			ipi_mailbox_buf[cpu_logical_map(cpu)] + 0x10);
-	loongson3_ipi_write64(startargs[1],
-			ipi_mailbox_buf[cpu_logical_map(cpu)] + 0x8);
-	loongson3_ipi_write64(startargs[0],
-			ipi_mailbox_buf[cpu_logical_map(cpu)] + 0x0);
+	if (cpu_has_csr() && csr_readl(LOONGSON_CSR_FEATURES) & LOONGSON_CSRF_IPI) {
+		csr_mail_send(startargs[3], cpu_logical_map(cpu), 3);
+		csr_mail_send(startargs[2], cpu_logical_map(cpu), 2);
+		csr_mail_send(startargs[1], cpu_logical_map(cpu), 1);
+		csr_mail_send(startargs[0], cpu_logical_map(cpu), 0);
+	} else {
+		loongson3_ipi_write64(startargs[3],
+				ipi_mailbox_buf[cpu_logical_map(cpu)] + 0x18);
+		loongson3_ipi_write64(startargs[2],
+				ipi_mailbox_buf[cpu_logical_map(cpu)] + 0x10);
+		loongson3_ipi_write64(startargs[1],
+				ipi_mailbox_buf[cpu_logical_map(cpu)] + 0x8);
+		loongson3_ipi_write64(startargs[0],
+				ipi_mailbox_buf[cpu_logical_map(cpu)] + 0x0);
+	}
+
 	return 0;
 }
 
-- 
2.1.0

[PATCH 5/6] MIPS: Loongson64: Make sure the PC address is correct when 3A4000+ CPU hotplug

[Tiezhu Yang]

In loongson3_type3_play_dead(), in order to make sure the PC address is
correct, use lw to read the low 32 bits first, if the result is not zero,
then use ld to read the whole 64 bits, otherwise there maybe exists atomic
problem due to write high 32 bits first and then low 32 bits, like this:

high 32 bits (write done)
                                  -- only read high 32-bits which is wrong
low 32 bits (not yet write done)

This problem is especially for Loongson 3A4000+ CPU due to using Mail_Send
register which can only send 32 bits data one time. Although it is hard to
reproduce, we can do something at the software level to avoid the risks for
3A4000+ CPU, this change has no influence on the other Loongson CPUs.

Signed-off-by: Lu Zeng <zenglu@loongson.cn>
Signed-off-by: Jun Yi <yijun@loongson.cn>
Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
---
 arch/mips/loongson64/smp.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/arch/mips/loongson64/smp.c b/arch/mips/loongson64/smp.c
index 33597d2..e486bbf 100644
--- a/arch/mips/loongson64/smp.c
+++ b/arch/mips/loongson64/smp.c
@@ -731,9 +731,10 @@ static void loongson3_type3_play_dead(int *state_addr)
 		"1: li    %[count], 0x100             \n" /* wait for init loop */
 		"2: bnez  %[count], 2b                \n" /* limit mailbox access */
 		"   addiu %[count], -1                \n"
-		"   ld    %[initfunc], 0x20(%[base])  \n" /* get PC via mailbox */
+		"   lw    %[initfunc], 0x20(%[base])  \n" /* get PC (low 32 bits) via mailbox */
 		"   beqz  %[initfunc], 1b             \n"
 		"   nop                               \n"
+		"   ld    %[initfunc], 0x20(%[base])  \n" /* get PC (whole 64 bits) via mailbox */
 		"   ld    $sp, 0x28(%[base])          \n" /* get SP via mailbox */
 		"   ld    $gp, 0x30(%[base])          \n" /* get GP via mailbox */
 		"   ld    $a1, 0x38(%[base])          \n"
-- 
2.1.0

[PATCH 6/6] MIPS: Loongson64: Move decode_cpucfg() to loongson_regs.h

[Tiezhu Yang]

Since decode_cpucfg() is only used for Loongson64, just move
it to loongson_regs.h to avoid the pollution of common code
with #ifdef CONFIG_CPU_LOONGSON64.

Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
---
 .../include/asm/mach-loongson64/loongson_regs.h    | 24 +++++++++++++++++
 arch/mips/kernel/cpu-probe.c                       | 31 +---------------------
 2 files changed, 25 insertions(+), 30 deletions(-)

diff --git a/arch/mips/include/asm/mach-loongson64/loongson_regs.h b/arch/mips/include/asm/mach-loongson64/loongson_regs.h
index 1659935..2d469d6 100644
--- a/arch/mips/include/asm/mach-loongson64/loongson_regs.h
+++ b/arch/mips/include/asm/mach-loongson64/loongson_regs.h
@@ -129,6 +129,30 @@ static inline u32 read_cpucfg(u32 reg)
 #define LOONGSON_CFG7_GCCAEQRP	BIT(0)
 #define LOONGSON_CFG7_UCAWINP	BIT(1)
 
+static inline void decode_cpucfg(struct cpuinfo_mips *c)
+{
+	u32 cfg1 = read_cpucfg(LOONGSON_CFG1);
+	u32 cfg2 = read_cpucfg(LOONGSON_CFG2);
+	u32 cfg3 = read_cpucfg(LOONGSON_CFG3);
+
+	if (cfg1 & LOONGSON_CFG1_MMI)
+		c->ases |= MIPS_ASE_LOONGSON_MMI;
+
+	if (cfg2 & LOONGSON_CFG2_LEXT1)
+		c->ases |= MIPS_ASE_LOONGSON_EXT;
+
+	if (cfg2 & LOONGSON_CFG2_LEXT2)
+		c->ases |= MIPS_ASE_LOONGSON_EXT2;
+
+	if (cfg2 & LOONGSON_CFG2_LSPW) {
+		c->options |= MIPS_CPU_LDPTE;
+		c->guest.options |= MIPS_CPU_LDPTE;
+	}
+
+	if (cfg3 & LOONGSON_CFG3_LCAMP)
+		c->ases |= MIPS_ASE_LOONGSON_CAM;
+}
+
 static inline bool cpu_has_csr(void)
 {
 	if (cpu_has_cfg())
diff --git a/arch/mips/kernel/cpu-probe.c b/arch/mips/kernel/cpu-probe.c
index e685369..1fa2c8b 100644
--- a/arch/mips/kernel/cpu-probe.c
+++ b/arch/mips/kernel/cpu-probe.c
@@ -31,6 +31,7 @@
 #include "fpu-probe.h"
 
 #include <asm/mach-loongson64/cpucfg-emul.h>
+#include <asm/mach-loongson64/loongson_regs.h>
 
 /* Hardware capabilities */
 unsigned int elf_hwcap __read_mostly;
@@ -1692,33 +1693,6 @@ static inline void cpu_probe_cavium(struct cpuinfo_mips *c, unsigned int cpu)
 	}
 }
 
-#ifdef CONFIG_CPU_LOONGSON64
-#include <loongson_regs.h>
-
-static inline void decode_cpucfg(struct cpuinfo_mips *c)
-{
-	u32 cfg1 = read_cpucfg(LOONGSON_CFG1);
-	u32 cfg2 = read_cpucfg(LOONGSON_CFG2);
-	u32 cfg3 = read_cpucfg(LOONGSON_CFG3);
-
-	if (cfg1 & LOONGSON_CFG1_MMI)
-		c->ases |= MIPS_ASE_LOONGSON_MMI;
-
-	if (cfg2 & LOONGSON_CFG2_LEXT1)
-		c->ases |= MIPS_ASE_LOONGSON_EXT;
-
-	if (cfg2 & LOONGSON_CFG2_LEXT2)
-		c->ases |= MIPS_ASE_LOONGSON_EXT2;
-
-	if (cfg2 & LOONGSON_CFG2_LSPW) {
-		c->options |= MIPS_CPU_LDPTE;
-		c->guest.options |= MIPS_CPU_LDPTE;
-	}
-
-	if (cfg3 & LOONGSON_CFG3_LCAMP)
-		c->ases |= MIPS_ASE_LOONGSON_CAM;
-}
-
 static inline void cpu_probe_loongson(struct cpuinfo_mips *c, unsigned int cpu)
 {
 	decode_configs(c);
@@ -1787,9 +1761,6 @@ static inline void cpu_probe_loongson(struct cpuinfo_mips *c, unsigned int cpu)
 		break;
 	}
 }
-#else
-static inline void cpu_probe_loongson(struct cpuinfo_mips *c, unsigned int cpu) { }
-#endif
 
 static inline void cpu_probe_ingenic(struct cpuinfo_mips *c, unsigned int cpu)
 {
-- 
2.1.0

Re: [PATCH 1/6] MIPS: Loongson64: Do not write the read only field LPA of CP0_CONFIG3

[Jiaxun Yang]

在 2020/10/29 16:02, Tiezhu Yang 写道:
> The field LPA of CP0_CONFIG3 register is read only for Loongson64, so the
> write operations are meaningless, remove them.
>
> Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
> ---
>   arch/mips/include/asm/mach-loongson64/kernel-entry-init.h | 8 --------
>   arch/mips/loongson64/numa.c                               | 3 ---
>   2 files changed, 11 deletions(-)
Hi Tiezhu,

AFAIK it's not read only on 3A1000.
Have you check with hardware guys?

Thanks.

- Jiaxun

Re: [PATCH 4/6] MIPS: Loongson64: Add Mail_Send support for 3A4000+ CPU

[Jiaxun Yang]

在 2020/10/29 16:02, Tiezhu Yang 写道:
> Loongson 3A4000+ CPU has per-core Mail_Send register to send mail,
> there is no need to maintain register address of each core and node,
> just simply specify cpu number.
>
> Signed-off-by: Lu Zeng <zenglu@loongson.cn>
> Signed-off-by: Jianmin Lv <lvjianmin@loongson.cn>
> Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
> ---
>   .../include/asm/mach-loongson64/loongson_regs.h    | 10 +++
>   arch/mips/loongson64/smp.c                         | 75 +++++++++++++++++-----
>   2 files changed, 68 insertions(+), 17 deletions(-)
>
> diff --git a/arch/mips/include/asm/mach-loongson64/loongson_regs.h b/arch/mips/include/asm/mach-loongson64/loongson_regs.h
> index 83dbb9f..1659935 100644
> --- a/arch/mips/include/asm/mach-loongson64/loongson_regs.h
> +++ b/arch/mips/include/asm/mach-loongson64/loongson_regs.h
> @@ -227,6 +227,16 @@ static inline void csr_writeq(u64 val, u32 reg)
>   #define CSR_IPI_SEND_CPU_SHIFT	16
>   #define CSR_IPI_SEND_BLOCK	BIT(31)
>   
> +#define LOONGSON_CSR_MAIL_BUF0		0x1020
> +#define LOONGSON_CSR_MAIL_SEND		0x1048
> +#define CSR_MAIL_SEND_BLOCK		BIT_ULL(31)
> +#define CSR_MAIL_SEND_BOX_LOW(box)	(box << 1)
> +#define CSR_MAIL_SEND_BOX_HIGH(box)	((box << 1) + 1)
> +#define CSR_MAIL_SEND_BOX_SHIFT		2
> +#define CSR_MAIL_SEND_CPU_SHIFT		16
> +#define CSR_MAIL_SEND_BUF_SHIFT		32
> +#define CSR_MAIL_SEND_H32_MASK		0xFFFFFFFF00000000ULL
> +
>   static inline u64 drdtime(void)
>   {
>   	int rID = 0;
> diff --git a/arch/mips/loongson64/smp.c b/arch/mips/loongson64/smp.c
> index 7d58853..33597d2 100644
> --- a/arch/mips/loongson64/smp.c
> +++ b/arch/mips/loongson64/smp.c
> @@ -54,6 +54,26 @@ static uint32_t core0_c0count[NR_CPUS];
>   u32 (*ipi_read_clear)(int cpu);
>   void (*ipi_write_action)(int cpu, u32 action);
>   
> +/* send mail via Mail_Send register for 3A4000+ CPU */
> +static void csr_mail_send(uint64_t data, int cpu, int mailbox)
> +{
> +	uint64_t val;
> +
> +	/* send high 32 bits */
> +	val = CSR_MAIL_SEND_BLOCK;
> +	val |= (CSR_MAIL_SEND_BOX_HIGH(mailbox) << CSR_MAIL_SEND_BOX_SHIFT);
> +	val |= (cpu << CSR_MAIL_SEND_CPU_SHIFT);
> +	val |= (data & CSR_MAIL_SEND_H32_MASK);
> +	csr_writeq(val, LOONGSON_CSR_MAIL_SEND);
> +
> +	/* send low 32 bits */
> +	val = CSR_MAIL_SEND_BLOCK;
> +	val |= (CSR_MAIL_SEND_BOX_LOW(mailbox) << CSR_MAIL_SEND_BOX_SHIFT);
> +	val |= (cpu << CSR_MAIL_SEND_CPU_SHIFT);
> +	val |= (data << CSR_MAIL_SEND_BUF_SHIFT);
> +	csr_writeq(val, LOONGSON_CSR_MAIL_SEND);
> +};
> +
>   static u32 csr_ipi_read_clear(int cpu)
>   {
>   	u32 action;
> @@ -348,7 +368,10 @@ static void loongson3_init_secondary(void)
>   	/* Set interrupt mask, but don't enable */
>   	change_c0_status(ST0_IM, imask);
>   
> -	loongson3_ipi_write32(0xffffffff, ipi_en0_regs[cpu_logical_map(cpu)]);
> +	if (cpu_has_csr() && csr_readl(LOONGSON_CSR_FEATURES) & LOONGSON_CSRF_IPI)
> +		csr_writel(0xffffffff, LOONGSON_CSR_IPI_EN);

Hi Tiezhu,

Feature cheking is duplicated. could you please abstract them to 
callbacks and set
them in csr_ipi probe?

e.g.
ipi_write_enable


> +	else
> +		loongson3_ipi_write32(0xffffffff, ipi_en0_regs[cpu_logical_map(cpu)]);
>   
>   	per_cpu(cpu_state, cpu) = CPU_ONLINE;
>   	cpu_set_core(&cpu_data[cpu],
> @@ -380,8 +403,12 @@ static void loongson3_smp_finish(void)
>   
>   	write_c0_compare(read_c0_count() + mips_hpt_frequency/HZ);
>   	local_irq_enable();
> -	loongson3_ipi_write64(0,
> -			ipi_mailbox_buf[cpu_logical_map(cpu)] + 0x0);
> +
> +	if (cpu_has_csr() && csr_readl(LOONGSON_CSR_FEATURES) & LOONGSON_CSRF_IPI)
> +		csr_writel(0, LOONGSON_CSR_MAIL_BUF0);
> +	else
> +		loongson3_ipi_write64(0, ipi_mailbox_buf[cpu_logical_map(cpu)] + 0x0);
> +

ditto here,
ipi_write_buf

>   	pr_info("CPU#%d finished, CP0_ST=%x\n",
>   			smp_processor_id(), read_c0_status());
>   }
> @@ -414,12 +441,18 @@ static void __init loongson3_smp_setup(void)
>   	}
>   
>   	csr_ipi_probe();
> -	ipi_set0_regs_init();
> -	ipi_clear0_regs_init();
> -	ipi_status0_regs_init();
> -	ipi_en0_regs_init();
> -	ipi_mailbox_buf_init();
> -	loongson3_ipi_write32(0xffffffff, ipi_en0_regs[cpu_logical_map(0)]);
> +
> +	if (cpu_has_csr() && csr_readl(LOONGSON_CSR_FEATURES) & LOONGSON_CSRF_IPI) {
> +		csr_writel(0xffffffff, LOONGSON_CSR_IPI_EN);
> +	} else {
> +		ipi_set0_regs_init();
> +		ipi_clear0_regs_init();
> +		ipi_status0_regs_init();
> +		ipi_en0_regs_init();
> +		ipi_mailbox_buf_init();
> +		loongson3_ipi_write32(0xffffffff, ipi_en0_regs[cpu_logical_map(0)]);
> +	}
> +
>   	cpu_set_core(&cpu_data[0],
>   		     cpu_logical_map(0) % loongson_sysconf.cores_per_package);
>   	cpu_data[0].package = cpu_logical_map(0) / loongson_sysconf.cores_per_package;
> @@ -452,14 +485,22 @@ static int loongson3_boot_secondary(int cpu, struct task_struct *idle)
>   	pr_debug("CPU#%d, func_pc=%lx, sp=%lx, gp=%lx\n",
>   			cpu, startargs[0], startargs[1], startargs[2]);
>   
> -	loongson3_ipi_write64(startargs[3],
> -			ipi_mailbox_buf[cpu_logical_map(cpu)] + 0x18);
> -	loongson3_ipi_write64(startargs[2],
> -			ipi_mailbox_buf[cpu_logical_map(cpu)] + 0x10);
> -	loongson3_ipi_write64(startargs[1],
> -			ipi_mailbox_buf[cpu_logical_map(cpu)] + 0x8);
> -	loongson3_ipi_write64(startargs[0],
> -			ipi_mailbox_buf[cpu_logical_map(cpu)] + 0x0);
> +	if (cpu_has_csr() && csr_readl(LOONGSON_CSR_FEATURES) & LOONGSON_CSRF_IPI) {
> +		csr_mail_send(startargs[3], cpu_logical_map(cpu), 3);
> +		csr_mail_send(startargs[2], cpu_logical_map(cpu), 2);
> +		csr_mail_send(startargs[1], cpu_logical_map(cpu), 1);
> +		csr_mail_send(startargs[0], cpu_logical_map(cpu), 0);
ditto

Thanks.

- Jiaxun

> +	} else {
> +		loongson3_ipi_write64(startargs[3],
> +				ipi_mailbox_buf[cpu_logical_map(cpu)] + 0x18);
> +		loongson3_ipi_write64(startargs[2],
> +				ipi_mailbox_buf[cpu_logical_map(cpu)] + 0x10);
> +		loongson3_ipi_write64(startargs[1],
> +				ipi_mailbox_buf[cpu_logical_map(cpu)] + 0x8);
> +		loongson3_ipi_write64(startargs[0],
> +				ipi_mailbox_buf[cpu_logical_map(cpu)] + 0x0);
> +	}
> +
>   	return 0;
>   }
>   

Re: [PATCH 1/6] MIPS: Loongson64: Do not write the read only field LPA of CP0_CONFIG3

[Tiezhu Yang]

On 10/30/2020 12:00 PM, Jiaxun Yang wrote:
>
>
> 在 2020/10/29 16:02, Tiezhu Yang 写道:
>> The field LPA of CP0_CONFIG3 register is read only for Loongson64, so 
>> the
>> write operations are meaningless, remove them.
>>
>> Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
>> ---
>>   arch/mips/include/asm/mach-loongson64/kernel-entry-init.h | 8 --------
>>   arch/mips/loongson64/numa.c                               | 3 ---
>>   2 files changed, 11 deletions(-)
> Hi Tiezhu,
>
> AFAIK it's not read only on 3A1000.
> Have you check with hardware guys?

Hi Jiaxun,

Yes, I checked it before send this patch,  the field LPA of CP0_CONFIG3
on 3A1000 is also read only,  we can see the related description in the
chapter 3.22 of Loongson_3A1000_cpu_user_2.

And also we can see the following description in the chapter
"9.45 Configuration Register 3 (CP0 Register 16, Select 3)" of
MD00091-2B-MIPS64PRA-AFP-05.04:
"The Config3 register encodes additional capabilities.
All fields in the Config3 register are read-only."

Thanks,
Tiezhu

>
> Thanks.
>
> - Jiaxun

Re: [PATCH 4/6] MIPS: Loongson64: Add Mail_Send support for 3A4000+ CPU

[Tiezhu Yang]

On 10/30/2020 12:06 PM, Jiaxun Yang wrote:
>
>
> 在 2020/10/29 16:02, Tiezhu Yang 写道:
>> Loongson 3A4000+ CPU has per-core Mail_Send register to send mail,
>> there is no need to maintain register address of each core and node,
>> just simply specify cpu number.
>>
>> Signed-off-by: Lu Zeng <zenglu@loongson.cn>
>> Signed-off-by: Jianmin Lv <lvjianmin@loongson.cn>
>> Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
>> ---
>>   .../include/asm/mach-loongson64/loongson_regs.h    | 10 +++
>>   arch/mips/loongson64/smp.c                         | 75 
>> +++++++++++++++++-----
>>   2 files changed, 68 insertions(+), 17 deletions(-)
>>
>> diff --git a/arch/mips/include/asm/mach-loongson64/loongson_regs.h 
>> b/arch/mips/include/asm/mach-loongson64/loongson_regs.h
>> index 83dbb9f..1659935 100644
>> --- a/arch/mips/include/asm/mach-loongson64/loongson_regs.h
>> +++ b/arch/mips/include/asm/mach-loongson64/loongson_regs.h
>> @@ -227,6 +227,16 @@ static inline void csr_writeq(u64 val, u32 reg)
>>   #define CSR_IPI_SEND_CPU_SHIFT    16
>>   #define CSR_IPI_SEND_BLOCK    BIT(31)
>>   +#define LOONGSON_CSR_MAIL_BUF0        0x1020
>> +#define LOONGSON_CSR_MAIL_SEND        0x1048
>> +#define CSR_MAIL_SEND_BLOCK        BIT_ULL(31)
>> +#define CSR_MAIL_SEND_BOX_LOW(box)    (box << 1)
>> +#define CSR_MAIL_SEND_BOX_HIGH(box)    ((box << 1) + 1)
>> +#define CSR_MAIL_SEND_BOX_SHIFT        2
>> +#define CSR_MAIL_SEND_CPU_SHIFT        16
>> +#define CSR_MAIL_SEND_BUF_SHIFT        32
>> +#define CSR_MAIL_SEND_H32_MASK        0xFFFFFFFF00000000ULL
>> +
>>   static inline u64 drdtime(void)
>>   {
>>       int rID = 0;
>> diff --git a/arch/mips/loongson64/smp.c b/arch/mips/loongson64/smp.c
>> index 7d58853..33597d2 100644
>> --- a/arch/mips/loongson64/smp.c
>> +++ b/arch/mips/loongson64/smp.c
>> @@ -54,6 +54,26 @@ static uint32_t core0_c0count[NR_CPUS];
>>   u32 (*ipi_read_clear)(int cpu);
>>   void (*ipi_write_action)(int cpu, u32 action);
>>   +/* send mail via Mail_Send register for 3A4000+ CPU */
>> +static void csr_mail_send(uint64_t data, int cpu, int mailbox)
>> +{
>> +    uint64_t val;
>> +
>> +    /* send high 32 bits */
>> +    val = CSR_MAIL_SEND_BLOCK;
>> +    val |= (CSR_MAIL_SEND_BOX_HIGH(mailbox) << 
>> CSR_MAIL_SEND_BOX_SHIFT);
>> +    val |= (cpu << CSR_MAIL_SEND_CPU_SHIFT);
>> +    val |= (data & CSR_MAIL_SEND_H32_MASK);
>> +    csr_writeq(val, LOONGSON_CSR_MAIL_SEND);
>> +
>> +    /* send low 32 bits */
>> +    val = CSR_MAIL_SEND_BLOCK;
>> +    val |= (CSR_MAIL_SEND_BOX_LOW(mailbox) << CSR_MAIL_SEND_BOX_SHIFT);
>> +    val |= (cpu << CSR_MAIL_SEND_CPU_SHIFT);
>> +    val |= (data << CSR_MAIL_SEND_BUF_SHIFT);
>> +    csr_writeq(val, LOONGSON_CSR_MAIL_SEND);
>> +};
>> +
>>   static u32 csr_ipi_read_clear(int cpu)
>>   {
>>       u32 action;
>> @@ -348,7 +368,10 @@ static void loongson3_init_secondary(void)
>>       /* Set interrupt mask, but don't enable */
>>       change_c0_status(ST0_IM, imask);
>>   -    loongson3_ipi_write32(0xffffffff, 
>> ipi_en0_regs[cpu_logical_map(cpu)]);
>> +    if (cpu_has_csr() && csr_readl(LOONGSON_CSR_FEATURES) & 
>> LOONGSON_CSRF_IPI)
>> +        csr_writel(0xffffffff, LOONGSON_CSR_IPI_EN);
>
> Hi Tiezhu,
>
> Feature cheking is duplicated. could you please abstract them to 
> callbacks and set
> them in csr_ipi probe?

Hi Jiaxun,

Thanks for your suggestion. I will do it and then send v2.

Thanks,
Tiezhu

>
> e.g.
> ipi_write_enable
>
>
>> +    else
>> +        loongson3_ipi_write32(0xffffffff, 
>> ipi_en0_regs[cpu_logical_map(cpu)]);
>>         per_cpu(cpu_state, cpu) = CPU_ONLINE;
>>       cpu_set_core(&cpu_data[cpu],
>> @@ -380,8 +403,12 @@ static void loongson3_smp_finish(void)
>>         write_c0_compare(read_c0_count() + mips_hpt_frequency/HZ);
>>       local_irq_enable();
>> -    loongson3_ipi_write64(0,
>> -            ipi_mailbox_buf[cpu_logical_map(cpu)] + 0x0);
>> +
>> +    if (cpu_has_csr() && csr_readl(LOONGSON_CSR_FEATURES) & 
>> LOONGSON_CSRF_IPI)
>> +        csr_writel(0, LOONGSON_CSR_MAIL_BUF0);
>> +    else
>> +        loongson3_ipi_write64(0, 
>> ipi_mailbox_buf[cpu_logical_map(cpu)] + 0x0);
>> +
>
> ditto here,
> ipi_write_buf
>
>>       pr_info("CPU#%d finished, CP0_ST=%x\n",
>>               smp_processor_id(), read_c0_status());
>>   }
>> @@ -414,12 +441,18 @@ static void __init loongson3_smp_setup(void)
>>       }
>>         csr_ipi_probe();
>> -    ipi_set0_regs_init();
>> -    ipi_clear0_regs_init();
>> -    ipi_status0_regs_init();
>> -    ipi_en0_regs_init();
>> -    ipi_mailbox_buf_init();
>> -    loongson3_ipi_write32(0xffffffff, 
>> ipi_en0_regs[cpu_logical_map(0)]);
>> +
>> +    if (cpu_has_csr() && csr_readl(LOONGSON_CSR_FEATURES) & 
>> LOONGSON_CSRF_IPI) {
>> +        csr_writel(0xffffffff, LOONGSON_CSR_IPI_EN);
>> +    } else {
>> +        ipi_set0_regs_init();
>> +        ipi_clear0_regs_init();
>> +        ipi_status0_regs_init();
>> +        ipi_en0_regs_init();
>> +        ipi_mailbox_buf_init();
>> +        loongson3_ipi_write32(0xffffffff, 
>> ipi_en0_regs[cpu_logical_map(0)]);
>> +    }
>> +
>>       cpu_set_core(&cpu_data[0],
>>                cpu_logical_map(0) % loongson_sysconf.cores_per_package);
>>       cpu_data[0].package = cpu_logical_map(0) / 
>> loongson_sysconf.cores_per_package;
>> @@ -452,14 +485,22 @@ static int loongson3_boot_secondary(int cpu, 
>> struct task_struct *idle)
>>       pr_debug("CPU#%d, func_pc=%lx, sp=%lx, gp=%lx\n",
>>               cpu, startargs[0], startargs[1], startargs[2]);
>>   -    loongson3_ipi_write64(startargs[3],
>> -            ipi_mailbox_buf[cpu_logical_map(cpu)] + 0x18);
>> -    loongson3_ipi_write64(startargs[2],
>> -            ipi_mailbox_buf[cpu_logical_map(cpu)] + 0x10);
>> -    loongson3_ipi_write64(startargs[1],
>> -            ipi_mailbox_buf[cpu_logical_map(cpu)] + 0x8);
>> -    loongson3_ipi_write64(startargs[0],
>> -            ipi_mailbox_buf[cpu_logical_map(cpu)] + 0x0);
>> +    if (cpu_has_csr() && csr_readl(LOONGSON_CSR_FEATURES) & 
>> LOONGSON_CSRF_IPI) {
>> +        csr_mail_send(startargs[3], cpu_logical_map(cpu), 3);
>> +        csr_mail_send(startargs[2], cpu_logical_map(cpu), 2);
>> +        csr_mail_send(startargs[1], cpu_logical_map(cpu), 1);
>> +        csr_mail_send(startargs[0], cpu_logical_map(cpu), 0);
> ditto
>
> Thanks.
>
> - Jiaxun
>
>> +    } else {
>> +        loongson3_ipi_write64(startargs[3],
>> +                ipi_mailbox_buf[cpu_logical_map(cpu)] + 0x18);
>> +        loongson3_ipi_write64(startargs[2],
>> +                ipi_mailbox_buf[cpu_logical_map(cpu)] + 0x10);
>> +        loongson3_ipi_write64(startargs[1],
>> +                ipi_mailbox_buf[cpu_logical_map(cpu)] + 0x8);
>> +        loongson3_ipi_write64(startargs[0],
>> +                ipi_mailbox_buf[cpu_logical_map(cpu)] + 0x0);
>> +    }
>> +
>>       return 0;
>>   }

Re: [PATCH 1/6] MIPS: Loongson64: Do not write the read only field LPA of CP0_CONFIG3

[Jiaxun Yang]

在 2020/10/30 14:22, Tiezhu Yang 写道:
> On 10/30/2020 12:00 PM, Jiaxun Yang wrote:
>>
>>
>> 在 2020/10/29 16:02, Tiezhu Yang 写道:
>>> The field LPA of CP0_CONFIG3 register is read only for Loongson64, 
>>> so the
>>> write operations are meaningless, remove them.
>>>
>>> Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
>>> ---
>>>   arch/mips/include/asm/mach-loongson64/kernel-entry-init.h | 8 
>>> --------
>>>   arch/mips/loongson64/numa.c                               | 3 ---
>>>   2 files changed, 11 deletions(-)
>> Hi Tiezhu,
>>
>> AFAIK it's not read only on 3A1000.
>> Have you check with hardware guys?
>
> Hi Jiaxun,
>
> Yes, I checked it before send this patch,  the field LPA of CP0_CONFIG3
> on 3A1000 is also read only,  we can see the related description in the
> chapter 3.22 of Loongson_3A1000_cpu_user_2.

On page 36, it said:
"能否写ELPA位依赖于Config3寄存器的LPA域 。当Config3的LPA位为0时，
PageGrain的ELPA位被置位。"

Don't know if it matters.

@Huacai, Any comments? Do you know why it exists here?

Thanks

- Jiaxun

Re: [PATCH 1/6] MIPS: Loongson64: Do not write the read only field LPA of CP0_CONFIG3

[Jiaxun Yang]

在 2020/10/30 20:13, Jiaxun Yang 写道:
>
>
> 在 2020/10/30 14:22, Tiezhu Yang 写道:
>> On 10/30/2020 12:00 PM, Jiaxun Yang wrote:
>>>
>>>
>>> 在 2020/10/29 16:02, Tiezhu Yang 写道:
>>>> The field LPA of CP0_CONFIG3 register is read only for Loongson64, 
>>>> so the
>>>> write operations are meaningless, remove them.
>>>>
>>>> Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
>>>> ---
>>>>   arch/mips/include/asm/mach-loongson64/kernel-entry-init.h | 8 
>>>> --------
>>>>   arch/mips/loongson64/numa.c | 3 ---
>>>>   2 files changed, 11 deletions(-)
>>> Hi Tiezhu,
>>>
>>> AFAIK it's not read only on 3A1000.
>>> Have you check with hardware guys?
>>
>> Hi Jiaxun,
>>
>> Yes, I checked it before send this patch,  the field LPA of CP0_CONFIG3
>> on 3A1000 is also read only,  we can see the related description in the
>> chapter 3.22 of Loongson_3A1000_cpu_user_2.
>
> On page 36, it said:
> "能否写ELPA位依赖于Config3寄存器的LPA域 。当Config3的LPA位为0时，
> PageGrain的ELPA位被置位。"
>
> Don't know if it matters.
>
> @Huacai, Any comments? Do you know why it exists here?

Confirmed by another rev of the manual that it's only a tip for implementer.
I guess my assumption was wrong.
LPA should be read-only though.....

Thanks
- Jiaxun

>
> Thanks
>
> - Jiaxun