* [PATCH 00/22] tcg: exit_tb tidy, goto_tb reorg
@ 2022-12-06 4:16 Richard Henderson
2022-12-06 4:16 ` [PATCH 01/22] tcg: Split out tcg_out_exit_tb Richard Henderson
` (21 more replies)
0 siblings, 22 replies; 35+ messages in thread
From: Richard Henderson @ 2022-12-06 4:16 UTC (permalink / raw)
To: qemu-devel; +Cc: r
Small patch for exit_tb. Large reorg for goto_tb, primarily aimed at
fixing a race condition in which a host thread gets suspended in the
middle of executing a two insn sequence, and the sequence is updated.
The updated second insn does not match the previous first insn, so
when the thread restarts the entire sequence will not branch to either
the old or the new destination.
The middle-end is adjusted to support both direct and indirect jumps
simultaneously, instead of depending on TCG_TARGET_HAS_direct_jump.
This allows the backend to decide whether to use direct or indirect
based on the branch displacement. Which allows us to only update a
single instruction, which means there's no multi-insn sequence that
can be interrupted.
This patch set fixes all except loongarch64, which will follow
separately along with other cleanup.
r~
Richard Henderson (22):
tcg: Split out tcg_out_exit_tb
tcg/i386: Remove unused goto_tb code for indirect jump
tcg/ppc: Remove unused goto_tb code for indirect jump
tcg/sparc64: Remove unused goto_tb code for indirect jump
tcg: Replace asserts on tcg_jmp_insn_offset
tcg: Introduce set_jmp_insn_offset
tcg: Introduce get_jmp_target_addr
tcg: Split out tcg_out_goto_tb
tcg: Rename TB_JMP_RESET_OFFSET_INVALID to TB_JMP_OFFSET_INVALID
tcg: Add gen_tb to TCGContext
tcg: Add TranslationBlock.jmp_insn_offset
tcg: Change tb_target_set_jmp_target arguments
tcg: Move tb_target_set_jmp_target declaration to tcg.h
tcg: Always define tb_target_set_jmp_target
tcg: Remove TCG_TARGET_HAS_direct_jump
tcg/aarch64: Reorg goto_tb implementation
tcg/ppc: Reorg goto_tb implementation
tcg/sparc64: Remove USE_REG_TB
tcg/sparc64: Reorg goto_tb implementation
tcg/arm: Implement direct branch for goto_tb
tcg/riscv: Introduce OPC_NOP
tcg/riscv: Implement direct branch for goto_tb
include/exec/exec-all.h | 5 +-
include/tcg/tcg.h | 14 +--
tcg/aarch64/tcg-target.h | 6 +-
tcg/arm/tcg-target.h | 5 -
tcg/i386/tcg-target.h | 9 --
tcg/loongarch64/tcg-target.h | 3 -
tcg/mips/tcg-target.h | 5 -
tcg/ppc/tcg-target.h | 7 +-
tcg/riscv/tcg-target.h | 5 +-
tcg/s390x/tcg-target.h | 11 --
tcg/sparc64/tcg-target.h | 4 -
tcg/tci/tcg-target.h | 4 -
accel/tcg/cpu-exec.c | 16 ++-
accel/tcg/translate-all.c | 10 +-
tcg/tcg-op.c | 14 +--
tcg/tcg.c | 42 +++++--
tcg/aarch64/tcg-target.c.inc | 104 ++++++++--------
tcg/arm/tcg-target.c.inc | 89 +++++++++-----
tcg/i386/tcg-target.c.inc | 68 ++++++-----
tcg/loongarch64/tcg-target.c.inc | 66 +++++-----
tcg/mips/tcg-target.c.inc | 59 +++++----
tcg/ppc/tcg-target.c.inc | 193 ++++++++++-------------------
tcg/riscv/tcg-target.c.inc | 65 ++++++----
tcg/s390x/tcg-target.c.inc | 106 +++++++++-------
tcg/sparc64/tcg-target.c.inc | 200 ++++++++++++-------------------
tcg/tci/tcg-target.c.inc | 31 +++--
26 files changed, 544 insertions(+), 597 deletions(-)
--
2.34.1
^ permalink raw reply [flat|nested] 35+ messages in thread
* [PATCH 01/22] tcg: Split out tcg_out_exit_tb
2022-12-06 4:16 [PATCH 00/22] tcg: exit_tb tidy, goto_tb reorg Richard Henderson
@ 2022-12-06 4:16 ` Richard Henderson
2022-12-06 7:24 ` Philippe Mathieu-Daudé
2022-12-06 4:16 ` [PATCH 02/22] tcg/i386: Remove unused goto_tb code for indirect jump Richard Henderson
` (20 subsequent siblings)
21 siblings, 1 reply; 35+ messages in thread
From: Richard Henderson @ 2022-12-06 4:16 UTC (permalink / raw)
To: qemu-devel; +Cc: r
The INDEX_op_exit_tb opcode needs no register allocation.
Split out a dedicated helper function for it.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
tcg/tcg.c | 4 ++++
tcg/aarch64/tcg-target.c.inc | 22 ++++++++++--------
tcg/arm/tcg-target.c.inc | 11 +++++----
tcg/i386/tcg-target.c.inc | 21 +++++++++--------
tcg/loongarch64/tcg-target.c.inc | 22 ++++++++++--------
tcg/mips/tcg-target.c.inc | 33 +++++++++++++--------------
tcg/ppc/tcg-target.c.inc | 11 +++++----
tcg/riscv/tcg-target.c.inc | 22 ++++++++++--------
tcg/s390x/tcg-target.c.inc | 23 ++++++++++---------
tcg/sparc64/tcg-target.c.inc | 39 +++++++++++++++++---------------
tcg/tci/tcg-target.c.inc | 10 ++++----
11 files changed, 121 insertions(+), 97 deletions(-)
diff --git a/tcg/tcg.c b/tcg/tcg.c
index 436fcf6ebd..b53961baf7 100644
--- a/tcg/tcg.c
+++ b/tcg/tcg.c
@@ -107,6 +107,7 @@ static void tcg_out_ld(TCGContext *s, TCGType type, TCGReg ret, TCGReg arg1,
static bool tcg_out_mov(TCGContext *s, TCGType type, TCGReg ret, TCGReg arg);
static void tcg_out_movi(TCGContext *s, TCGType type,
TCGReg ret, tcg_target_long arg);
+static void tcg_out_exit_tb(TCGContext *s, uintptr_t arg);
static void tcg_out_op(TCGContext *s, TCGOpcode opc,
const TCGArg args[TCG_MAX_OP_ARGS],
const int const_args[TCG_MAX_OP_ARGS]);
@@ -4302,6 +4303,9 @@ int tcg_gen_code(TCGContext *s, TranslationBlock *tb, target_ulong pc_start)
case INDEX_op_call:
tcg_reg_alloc_call(s, op);
break;
+ case INDEX_op_exit_tb:
+ tcg_out_exit_tb(s, op->args[0]);
+ break;
case INDEX_op_dup2_vec:
if (tcg_reg_alloc_dup2(s, op)) {
break;
diff --git a/tcg/aarch64/tcg-target.c.inc b/tcg/aarch64/tcg-target.c.inc
index 344b63e20f..16c5e33b69 100644
--- a/tcg/aarch64/tcg-target.c.inc
+++ b/tcg/aarch64/tcg-target.c.inc
@@ -1886,6 +1886,17 @@ static void tcg_out_qemu_st(TCGContext *s, TCGReg data_reg, TCGReg addr_reg,
static const tcg_insn_unit *tb_ret_addr;
+static void tcg_out_exit_tb(TCGContext *s, uintptr_t a0)
+{
+ /* Reuse the zeroing that exists for goto_ptr. */
+ if (a0 == 0) {
+ tcg_out_goto_long(s, tcg_code_gen_epilogue);
+ } else {
+ tcg_out_movi(s, TCG_TYPE_I64, TCG_REG_X0, a0);
+ tcg_out_goto_long(s, tb_ret_addr);
+ }
+}
+
static void tcg_out_op(TCGContext *s, TCGOpcode opc,
const TCGArg args[TCG_MAX_OP_ARGS],
const int const_args[TCG_MAX_OP_ARGS])
@@ -1905,16 +1916,6 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
#define REG0(I) (const_args[I] ? TCG_REG_XZR : (TCGReg)args[I])
switch (opc) {
- case INDEX_op_exit_tb:
- /* Reuse the zeroing that exists for goto_ptr. */
- if (a0 == 0) {
- tcg_out_goto_long(s, tcg_code_gen_epilogue);
- } else {
- tcg_out_movi(s, TCG_TYPE_I64, TCG_REG_X0, a0);
- tcg_out_goto_long(s, tb_ret_addr);
- }
- break;
-
case INDEX_op_goto_tb:
tcg_debug_assert(s->tb_jmp_insn_offset != NULL);
/*
@@ -2304,6 +2305,7 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
case INDEX_op_mov_i32: /* Always emitted via tcg_out_mov. */
case INDEX_op_mov_i64:
case INDEX_op_call: /* Always emitted via tcg_out_call. */
+ case INDEX_op_exit_tb: /* Always emitted via tcg_out_exit_tb. */
default:
g_assert_not_reached();
}
diff --git a/tcg/arm/tcg-target.c.inc b/tcg/arm/tcg-target.c.inc
index 2c6c353eea..e5c2eae5a5 100644
--- a/tcg/arm/tcg-target.c.inc
+++ b/tcg/arm/tcg-target.c.inc
@@ -1927,6 +1927,12 @@ static void tcg_out_qemu_st(TCGContext *s, const TCGArg *args, bool is64)
static void tcg_out_epilogue(TCGContext *s);
+static void tcg_out_exit_tb(TCGContext *s, uintptr_t arg)
+{
+ tcg_out_movi(s, TCG_TYPE_PTR, TCG_REG_R0, arg);
+ tcg_out_epilogue(s);
+}
+
static void tcg_out_op(TCGContext *s, TCGOpcode opc,
const TCGArg args[TCG_MAX_OP_ARGS],
const int const_args[TCG_MAX_OP_ARGS])
@@ -1935,10 +1941,6 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
int c;
switch (opc) {
- case INDEX_op_exit_tb:
- tcg_out_movi(s, TCG_TYPE_PTR, TCG_REG_R0, args[0]);
- tcg_out_epilogue(s);
- break;
case INDEX_op_goto_tb:
{
/* Indirect jump method */
@@ -2250,6 +2252,7 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
case INDEX_op_mov_i32: /* Always emitted via tcg_out_mov. */
case INDEX_op_call: /* Always emitted via tcg_out_call. */
+ case INDEX_op_exit_tb: /* Always emitted via tcg_out_exit_tb. */
default:
tcg_abort();
}
diff --git a/tcg/i386/tcg-target.c.inc b/tcg/i386/tcg-target.c.inc
index cb04e4b3ad..5c20bedd20 100644
--- a/tcg/i386/tcg-target.c.inc
+++ b/tcg/i386/tcg-target.c.inc
@@ -2346,6 +2346,17 @@ static void tcg_out_qemu_st(TCGContext *s, const TCGArg *args, bool is64)
#endif
}
+static void tcg_out_exit_tb(TCGContext *s, uintptr_t a0)
+{
+ /* Reuse the zeroing that exists for goto_ptr. */
+ if (a0 == 0) {
+ tcg_out_jmp(s, tcg_code_gen_epilogue);
+ } else {
+ tcg_out_movi(s, TCG_TYPE_PTR, TCG_REG_EAX, a0);
+ tcg_out_jmp(s, tb_ret_addr);
+ }
+}
+
static inline void tcg_out_op(TCGContext *s, TCGOpcode opc,
const TCGArg args[TCG_MAX_OP_ARGS],
const int const_args[TCG_MAX_OP_ARGS])
@@ -2370,15 +2381,6 @@ static inline void tcg_out_op(TCGContext *s, TCGOpcode opc,
const_a2 = const_args[2];
switch (opc) {
- case INDEX_op_exit_tb:
- /* Reuse the zeroing that exists for goto_ptr. */
- if (a0 == 0) {
- tcg_out_jmp(s, tcg_code_gen_epilogue);
- } else {
- tcg_out_movi(s, TCG_TYPE_PTR, TCG_REG_EAX, a0);
- tcg_out_jmp(s, tb_ret_addr);
- }
- break;
case INDEX_op_goto_tb:
if (s->tb_jmp_insn_offset) {
/* direct jump method */
@@ -2793,6 +2795,7 @@ static inline void tcg_out_op(TCGContext *s, TCGOpcode opc,
case INDEX_op_mov_i32: /* Always emitted via tcg_out_mov. */
case INDEX_op_mov_i64:
case INDEX_op_call: /* Always emitted via tcg_out_call. */
+ case INDEX_op_exit_tb: /* Always emitted via tcg_out_exit_tb. */
default:
tcg_abort();
}
diff --git a/tcg/loongarch64/tcg-target.c.inc b/tcg/loongarch64/tcg-target.c.inc
index d326e28740..41fc5ffa91 100644
--- a/tcg/loongarch64/tcg-target.c.inc
+++ b/tcg/loongarch64/tcg-target.c.inc
@@ -1067,6 +1067,17 @@ void tb_target_set_jmp_target(uintptr_t tc_ptr, uintptr_t jmp_rx,
static const tcg_insn_unit *tb_ret_addr;
+static void tcg_out_exit_tb(TCGContext *s, uintptr_t a0)
+{
+ /* Reuse the zeroing that exists for goto_ptr. */
+ if (a0 == 0) {
+ tcg_out_call_int(s, tcg_code_gen_epilogue, true);
+ } else {
+ tcg_out_movi(s, TCG_TYPE_PTR, TCG_REG_A0, a0);
+ tcg_out_call_int(s, tb_ret_addr, true);
+ }
+}
+
static void tcg_out_op(TCGContext *s, TCGOpcode opc,
const TCGArg args[TCG_MAX_OP_ARGS],
const int const_args[TCG_MAX_OP_ARGS])
@@ -1077,16 +1088,6 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
int c2 = const_args[2];
switch (opc) {
- case INDEX_op_exit_tb:
- /* Reuse the zeroing that exists for goto_ptr. */
- if (a0 == 0) {
- tcg_out_call_int(s, tcg_code_gen_epilogue, true);
- } else {
- tcg_out_movi(s, TCG_TYPE_PTR, TCG_REG_A0, a0);
- tcg_out_call_int(s, tb_ret_addr, true);
- }
- break;
-
case INDEX_op_goto_tb:
tcg_debug_assert(s->tb_jmp_insn_offset != NULL);
/*
@@ -1499,6 +1500,7 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
case INDEX_op_mov_i32: /* Always emitted via tcg_out_mov. */
case INDEX_op_mov_i64:
case INDEX_op_call: /* Always emitted via tcg_out_call. */
+ case INDEX_op_exit_tb: /* Always emitted via tcg_out_exit_tb. */
default:
g_assert_not_reached();
}
diff --git a/tcg/mips/tcg-target.c.inc b/tcg/mips/tcg-target.c.inc
index bd76f0c97f..819648b100 100644
--- a/tcg/mips/tcg-target.c.inc
+++ b/tcg/mips/tcg-target.c.inc
@@ -1950,6 +1950,21 @@ static void tcg_out_clz(TCGContext *s, MIPSInsn opcv2, MIPSInsn opcv6,
}
}
+static void tcg_out_exit_tb(TCGContext *s, uintptr_t a0)
+{
+ TCGReg b0 = TCG_REG_ZERO;
+
+ if (a0 & ~0xffff) {
+ tcg_out_movi(s, TCG_TYPE_PTR, TCG_REG_V0, a0 & ~0xffff);
+ b0 = TCG_REG_V0;
+ }
+ if (!tcg_out_opc_jmp(s, OPC_J, tb_ret_addr)) {
+ tcg_out_movi(s, TCG_TYPE_PTR, TCG_TMP0, (uintptr_t)tb_ret_addr);
+ tcg_out_opc_reg(s, OPC_JR, 0, TCG_TMP0, 0);
+ }
+ tcg_out_opc_imm(s, OPC_ORI, TCG_REG_V0, b0, a0 & 0xffff);
+}
+
static void tcg_out_op(TCGContext *s, TCGOpcode opc,
const TCGArg args[TCG_MAX_OP_ARGS],
const int const_args[TCG_MAX_OP_ARGS])
@@ -1969,23 +1984,6 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
c2 = const_args[2];
switch (opc) {
- case INDEX_op_exit_tb:
- {
- TCGReg b0 = TCG_REG_ZERO;
-
- a0 = (intptr_t)a0;
- if (a0 & ~0xffff) {
- tcg_out_movi(s, TCG_TYPE_PTR, TCG_REG_V0, a0 & ~0xffff);
- b0 = TCG_REG_V0;
- }
- if (!tcg_out_opc_jmp(s, OPC_J, tb_ret_addr)) {
- tcg_out_movi(s, TCG_TYPE_PTR, TCG_TMP0,
- (uintptr_t)tb_ret_addr);
- tcg_out_opc_reg(s, OPC_JR, 0, TCG_TMP0, 0);
- }
- tcg_out_opc_imm(s, OPC_ORI, TCG_REG_V0, b0, a0 & 0xffff);
- }
- break;
case INDEX_op_goto_tb:
/* indirect jump method */
tcg_debug_assert(s->tb_jmp_insn_offset == 0);
@@ -2402,6 +2400,7 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
case INDEX_op_mov_i32: /* Always emitted via tcg_out_mov. */
case INDEX_op_mov_i64:
case INDEX_op_call: /* Always emitted via tcg_out_call. */
+ case INDEX_op_exit_tb: /* Always emitted via tcg_out_exit_tb. */
default:
tcg_abort();
}
diff --git a/tcg/ppc/tcg-target.c.inc b/tcg/ppc/tcg-target.c.inc
index e3dba47697..186599dae0 100644
--- a/tcg/ppc/tcg-target.c.inc
+++ b/tcg/ppc/tcg-target.c.inc
@@ -2616,6 +2616,12 @@ static void tcg_target_qemu_prologue(TCGContext *s)
tcg_out32(s, BCLR | BO_ALWAYS);
}
+static void tcg_out_exit_tb(TCGContext *s, uintptr_t arg)
+{
+ tcg_out_movi(s, TCG_TYPE_PTR, TCG_REG_R3, arg);
+ tcg_out_b(s, 0, tcg_code_gen_epilogue);
+}
+
static void tcg_out_op(TCGContext *s, TCGOpcode opc,
const TCGArg args[TCG_MAX_OP_ARGS],
const int const_args[TCG_MAX_OP_ARGS])
@@ -2623,10 +2629,6 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
TCGArg a0, a1, a2;
switch (opc) {
- case INDEX_op_exit_tb:
- tcg_out_movi(s, TCG_TYPE_PTR, TCG_REG_R3, args[0]);
- tcg_out_b(s, 0, tcg_code_gen_epilogue);
- break;
case INDEX_op_goto_tb:
if (s->tb_jmp_insn_offset) {
/* Direct jump. */
@@ -3185,6 +3187,7 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
case INDEX_op_mov_i32: /* Always emitted via tcg_out_mov. */
case INDEX_op_mov_i64:
case INDEX_op_call: /* Always emitted via tcg_out_call. */
+ case INDEX_op_exit_tb: /* Always emitted via tcg_out_exit_tb. */
default:
tcg_abort();
}
diff --git a/tcg/riscv/tcg-target.c.inc b/tcg/riscv/tcg-target.c.inc
index 81a83e45b1..1381c835af 100644
--- a/tcg/riscv/tcg-target.c.inc
+++ b/tcg/riscv/tcg-target.c.inc
@@ -1264,6 +1264,17 @@ static void tcg_out_qemu_st(TCGContext *s, const TCGArg *args, bool is_64)
static const tcg_insn_unit *tb_ret_addr;
+static void tcg_out_exit_tb(TCGContext *s, uintptr_t a0)
+{
+ /* Reuse the zeroing that exists for goto_ptr. */
+ if (a0 == 0) {
+ tcg_out_call_int(s, tcg_code_gen_epilogue, true);
+ } else {
+ tcg_out_movi(s, TCG_TYPE_PTR, TCG_REG_A0, a0);
+ tcg_out_call_int(s, tb_ret_addr, true);
+ }
+}
+
static void tcg_out_op(TCGContext *s, TCGOpcode opc,
const TCGArg args[TCG_MAX_OP_ARGS],
const int const_args[TCG_MAX_OP_ARGS])
@@ -1274,16 +1285,6 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
int c2 = const_args[2];
switch (opc) {
- case INDEX_op_exit_tb:
- /* Reuse the zeroing that exists for goto_ptr. */
- if (a0 == 0) {
- tcg_out_call_int(s, tcg_code_gen_epilogue, true);
- } else {
- tcg_out_movi(s, TCG_TYPE_PTR, TCG_REG_A0, a0);
- tcg_out_call_int(s, tb_ret_addr, true);
- }
- break;
-
case INDEX_op_goto_tb:
assert(s->tb_jmp_insn_offset == 0);
/* indirect jump method */
@@ -1592,6 +1593,7 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
case INDEX_op_mov_i32: /* Always emitted via tcg_out_mov. */
case INDEX_op_mov_i64:
case INDEX_op_call: /* Always emitted via tcg_out_call. */
+ case INDEX_op_exit_tb: /* Always emitted via tcg_out_exit_tb. */
default:
g_assert_not_reached();
}
diff --git a/tcg/s390x/tcg-target.c.inc b/tcg/s390x/tcg-target.c.inc
index 33becd7694..9568452773 100644
--- a/tcg/s390x/tcg-target.c.inc
+++ b/tcg/s390x/tcg-target.c.inc
@@ -2070,6 +2070,17 @@ static void tcg_out_qemu_st(TCGContext* s, TCGReg data_reg, TCGReg addr_reg,
#endif
}
+static void tcg_out_exit_tb(TCGContext *s, uintptr_t a0)
+{
+ /* Reuse the zeroing that exists for goto_ptr. */
+ if (a0 == 0) {
+ tgen_gotoi(s, S390_CC_ALWAYS, tcg_code_gen_epilogue);
+ } else {
+ tcg_out_movi(s, TCG_TYPE_PTR, TCG_REG_R2, a0);
+ tgen_gotoi(s, S390_CC_ALWAYS, tb_ret_addr);
+ }
+}
+
# define OP_32_64(x) \
case glue(glue(INDEX_op_,x),_i32): \
case glue(glue(INDEX_op_,x),_i64)
@@ -2082,17 +2093,6 @@ static inline void tcg_out_op(TCGContext *s, TCGOpcode opc,
TCGArg a0, a1, a2;
switch (opc) {
- case INDEX_op_exit_tb:
- /* Reuse the zeroing that exists for goto_ptr. */
- a0 = args[0];
- if (a0 == 0) {
- tgen_gotoi(s, S390_CC_ALWAYS, tcg_code_gen_epilogue);
- } else {
- tcg_out_movi(s, TCG_TYPE_PTR, TCG_REG_R2, a0);
- tgen_gotoi(s, S390_CC_ALWAYS, tb_ret_addr);
- }
- break;
-
case INDEX_op_goto_tb:
a0 = args[0];
if (s->tb_jmp_insn_offset) {
@@ -2661,6 +2661,7 @@ static inline void tcg_out_op(TCGContext *s, TCGOpcode opc,
case INDEX_op_mov_i32: /* Always emitted via tcg_out_mov. */
case INDEX_op_mov_i64:
case INDEX_op_call: /* Always emitted via tcg_out_call. */
+ case INDEX_op_exit_tb: /* Always emitted via tcg_out_exit_tb. */
default:
tcg_abort();
}
diff --git a/tcg/sparc64/tcg-target.c.inc b/tcg/sparc64/tcg-target.c.inc
index cb9453efdd..1e3351a4e8 100644
--- a/tcg/sparc64/tcg-target.c.inc
+++ b/tcg/sparc64/tcg-target.c.inc
@@ -1427,6 +1427,26 @@ static void tcg_out_qemu_st(TCGContext *s, TCGReg data, TCGReg addr,
#endif /* CONFIG_SOFTMMU */
}
+static void tcg_out_exit_tb(TCGContext *s, uintptr_t a0)
+{
+ if (check_fit_ptr(a0, 13)) {
+ tcg_out_arithi(s, TCG_REG_G0, TCG_REG_I7, 8, RETURN);
+ tcg_out_movi_imm13(s, TCG_REG_O0, a0);
+ return;
+ } else if (USE_REG_TB) {
+ intptr_t tb_diff = tcg_tbrel_diff(s, (void *)a0);
+ if (check_fit_ptr(tb_diff, 13)) {
+ tcg_out_arithi(s, TCG_REG_G0, TCG_REG_I7, 8, RETURN);
+ /* Note that TCG_REG_TB has been unwound to O1. */
+ tcg_out_arithi(s, TCG_REG_O0, TCG_REG_O1, tb_diff, ARITH_ADD);
+ return;
+ }
+ }
+ tcg_out_movi(s, TCG_TYPE_PTR, TCG_REG_I0, a0 & ~0x3ff);
+ tcg_out_arithi(s, TCG_REG_G0, TCG_REG_I7, 8, RETURN);
+ tcg_out_arithi(s, TCG_REG_O0, TCG_REG_O0, a0 & 0x3ff, ARITH_OR);
+}
+
static void tcg_out_op(TCGContext *s, TCGOpcode opc,
const TCGArg args[TCG_MAX_OP_ARGS],
const int const_args[TCG_MAX_OP_ARGS])
@@ -1441,24 +1461,6 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
c2 = const_args[2];
switch (opc) {
- case INDEX_op_exit_tb:
- if (check_fit_ptr(a0, 13)) {
- tcg_out_arithi(s, TCG_REG_G0, TCG_REG_I7, 8, RETURN);
- tcg_out_movi_imm13(s, TCG_REG_O0, a0);
- break;
- } else if (USE_REG_TB) {
- intptr_t tb_diff = tcg_tbrel_diff(s, (void *)a0);
- if (check_fit_ptr(tb_diff, 13)) {
- tcg_out_arithi(s, TCG_REG_G0, TCG_REG_I7, 8, RETURN);
- /* Note that TCG_REG_TB has been unwound to O1. */
- tcg_out_arithi(s, TCG_REG_O0, TCG_REG_O1, tb_diff, ARITH_ADD);
- break;
- }
- }
- tcg_out_movi(s, TCG_TYPE_PTR, TCG_REG_I0, a0 & ~0x3ff);
- tcg_out_arithi(s, TCG_REG_G0, TCG_REG_I7, 8, RETURN);
- tcg_out_arithi(s, TCG_REG_O0, TCG_REG_O0, a0 & 0x3ff, ARITH_OR);
- break;
case INDEX_op_goto_tb:
if (s->tb_jmp_insn_offset) {
/* direct jump method */
@@ -1715,6 +1717,7 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
case INDEX_op_mov_i32: /* Always emitted via tcg_out_mov. */
case INDEX_op_mov_i64:
case INDEX_op_call: /* Always emitted via tcg_out_call. */
+ case INDEX_op_exit_tb: /* Always emitted via tcg_out_exit_tb. */
default:
tcg_abort();
}
diff --git a/tcg/tci/tcg-target.c.inc b/tcg/tci/tcg-target.c.inc
index f3d7441e06..34583a3499 100644
--- a/tcg/tci/tcg-target.c.inc
+++ b/tcg/tci/tcg-target.c.inc
@@ -593,6 +593,11 @@ static void tcg_out_call(TCGContext *s, const tcg_insn_unit *func,
# define CASE_64(x)
#endif
+static void tcg_out_exit_tb(TCGContext *s, uintptr_t arg)
+{
+ tcg_out_op_p(s, INDEX_op_exit_tb, (void *)arg);
+}
+
static void tcg_out_op(TCGContext *s, TCGOpcode opc,
const TCGArg args[TCG_MAX_OP_ARGS],
const int const_args[TCG_MAX_OP_ARGS])
@@ -600,10 +605,6 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
TCGOpcode exts;
switch (opc) {
- case INDEX_op_exit_tb:
- tcg_out_op_p(s, opc, (void *)args[0]);
- break;
-
case INDEX_op_goto_tb:
tcg_debug_assert(s->tb_jmp_insn_offset == 0);
/* indirect jump method. */
@@ -782,6 +783,7 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
case INDEX_op_mov_i32: /* Always emitted via tcg_out_mov. */
case INDEX_op_mov_i64:
case INDEX_op_call: /* Always emitted via tcg_out_call. */
+ case INDEX_op_exit_tb: /* Always emitted via tcg_out_exit_tb. */
default:
tcg_abort();
}
--
2.34.1
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH 02/22] tcg/i386: Remove unused goto_tb code for indirect jump
2022-12-06 4:16 [PATCH 00/22] tcg: exit_tb tidy, goto_tb reorg Richard Henderson
2022-12-06 4:16 ` [PATCH 01/22] tcg: Split out tcg_out_exit_tb Richard Henderson
@ 2022-12-06 4:16 ` Richard Henderson
2022-12-06 4:16 ` [PATCH 03/22] tcg/ppc: " Richard Henderson
` (19 subsequent siblings)
21 siblings, 0 replies; 35+ messages in thread
From: Richard Henderson @ 2022-12-06 4:16 UTC (permalink / raw)
To: qemu-devel; +Cc: r
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
tcg/i386/tcg-target.c.inc | 14 +++++---------
1 file changed, 5 insertions(+), 9 deletions(-)
diff --git a/tcg/i386/tcg-target.c.inc b/tcg/i386/tcg-target.c.inc
index 5c20bedd20..f3a40fc428 100644
--- a/tcg/i386/tcg-target.c.inc
+++ b/tcg/i386/tcg-target.c.inc
@@ -2382,23 +2382,19 @@ static inline void tcg_out_op(TCGContext *s, TCGOpcode opc,
switch (opc) {
case INDEX_op_goto_tb:
- if (s->tb_jmp_insn_offset) {
- /* direct jump method */
- int gap;
- /* jump displacement must be aligned for atomic patching;
+ qemu_build_assert(TCG_TARGET_HAS_direct_jump);
+ {
+ /*
+ * Jump displacement must be aligned for atomic patching;
* see if we need to add extra nops before jump
*/
- gap = QEMU_ALIGN_PTR_UP(s->code_ptr + 1, 4) - s->code_ptr;
+ int gap = QEMU_ALIGN_PTR_UP(s->code_ptr + 1, 4) - s->code_ptr;
if (gap != 1) {
tcg_out_nopn(s, gap - 1);
}
tcg_out8(s, OPC_JMP_long); /* jmp im */
s->tb_jmp_insn_offset[a0] = tcg_current_code_size(s);
tcg_out32(s, 0);
- } else {
- /* indirect jump method */
- tcg_out_modrm_offset(s, OPC_GRP5, EXT5_JMPN_Ev, -1,
- (intptr_t)(s->tb_jmp_target_addr + a0));
}
set_jmp_reset_offset(s, a0);
break;
--
2.34.1
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH 03/22] tcg/ppc: Remove unused goto_tb code for indirect jump
2022-12-06 4:16 [PATCH 00/22] tcg: exit_tb tidy, goto_tb reorg Richard Henderson
2022-12-06 4:16 ` [PATCH 01/22] tcg: Split out tcg_out_exit_tb Richard Henderson
2022-12-06 4:16 ` [PATCH 02/22] tcg/i386: Remove unused goto_tb code for indirect jump Richard Henderson
@ 2022-12-06 4:16 ` Richard Henderson
2022-12-06 4:16 ` [PATCH 04/22] tcg/sparc64: " Richard Henderson
` (18 subsequent siblings)
21 siblings, 0 replies; 35+ messages in thread
From: Richard Henderson @ 2022-12-06 4:16 UTC (permalink / raw)
To: qemu-devel; +Cc: r
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
tcg/ppc/tcg-target.c.inc | 32 +++++++++++++-------------------
1 file changed, 13 insertions(+), 19 deletions(-)
diff --git a/tcg/ppc/tcg-target.c.inc b/tcg/ppc/tcg-target.c.inc
index 186599dae0..3f9ee4b39a 100644
--- a/tcg/ppc/tcg-target.c.inc
+++ b/tcg/ppc/tcg-target.c.inc
@@ -2630,27 +2630,21 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
switch (opc) {
case INDEX_op_goto_tb:
- if (s->tb_jmp_insn_offset) {
- /* Direct jump. */
- if (TCG_TARGET_REG_BITS == 64) {
- /* Ensure the next insns are 8 or 16-byte aligned. */
- while ((uintptr_t)s->code_ptr & (have_isa_2_07 ? 15 : 7)) {
- tcg_out32(s, NOP);
- }
- s->tb_jmp_insn_offset[args[0]] = tcg_current_code_size(s);
- tcg_out32(s, ADDIS | TAI(TCG_REG_TB, TCG_REG_TB, 0));
- tcg_out32(s, ADDI | TAI(TCG_REG_TB, TCG_REG_TB, 0));
- } else {
- s->tb_jmp_insn_offset[args[0]] = tcg_current_code_size(s);
- tcg_out32(s, B);
- s->tb_jmp_reset_offset[args[0]] = tcg_current_code_size(s);
- break;
+ qemu_build_assert(TCG_TARGET_HAS_direct_jump);
+ /* Direct jump. */
+ if (TCG_TARGET_REG_BITS == 64) {
+ /* Ensure the next insns are 8 or 16-byte aligned. */
+ while ((uintptr_t)s->code_ptr & (have_isa_2_07 ? 15 : 7)) {
+ tcg_out32(s, NOP);
}
+ s->tb_jmp_insn_offset[args[0]] = tcg_current_code_size(s);
+ tcg_out32(s, ADDIS | TAI(TCG_REG_TB, TCG_REG_TB, 0));
+ tcg_out32(s, ADDI | TAI(TCG_REG_TB, TCG_REG_TB, 0));
} else {
- /* Indirect jump. */
- tcg_debug_assert(s->tb_jmp_insn_offset == NULL);
- tcg_out_ld(s, TCG_TYPE_PTR, TCG_REG_TB, 0,
- (intptr_t)(s->tb_jmp_insn_offset + args[0]));
+ s->tb_jmp_insn_offset[args[0]] = tcg_current_code_size(s);
+ tcg_out32(s, B);
+ s->tb_jmp_reset_offset[args[0]] = tcg_current_code_size(s);
+ break;
}
tcg_out32(s, MTSPR | RS(TCG_REG_TB) | CTR);
tcg_out32(s, BCCTR | BO_ALWAYS);
--
2.34.1
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH 04/22] tcg/sparc64: Remove unused goto_tb code for indirect jump
2022-12-06 4:16 [PATCH 00/22] tcg: exit_tb tidy, goto_tb reorg Richard Henderson
` (2 preceding siblings ...)
2022-12-06 4:16 ` [PATCH 03/22] tcg/ppc: " Richard Henderson
@ 2022-12-06 4:16 ` Richard Henderson
2022-12-06 4:16 ` [PATCH 05/22] tcg: Replace asserts on tcg_jmp_insn_offset Richard Henderson
` (17 subsequent siblings)
21 siblings, 0 replies; 35+ messages in thread
From: Richard Henderson @ 2022-12-06 4:16 UTC (permalink / raw)
To: qemu-devel; +Cc: r
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
tcg/sparc64/tcg-target.c.inc | 41 +++++++++++-------------------------
1 file changed, 12 insertions(+), 29 deletions(-)
diff --git a/tcg/sparc64/tcg-target.c.inc b/tcg/sparc64/tcg-target.c.inc
index 1e3351a4e8..f035bf7dd2 100644
--- a/tcg/sparc64/tcg-target.c.inc
+++ b/tcg/sparc64/tcg-target.c.inc
@@ -537,17 +537,6 @@ static bool tcg_out_sti(TCGContext *s, TCGType type, TCGArg val,
return false;
}
-static void tcg_out_ld_ptr(TCGContext *s, TCGReg ret, const void *arg)
-{
- intptr_t diff = tcg_tbrel_diff(s, arg);
- if (USE_REG_TB && check_fit_ptr(diff, 13)) {
- tcg_out_ld(s, TCG_TYPE_PTR, ret, TCG_REG_TB, diff);
- return;
- }
- tcg_out_movi(s, TCG_TYPE_PTR, ret, (uintptr_t)arg & ~0x3ff);
- tcg_out_ld(s, TCG_TYPE_PTR, ret, ret, (uintptr_t)arg & 0x3ff);
-}
-
static void tcg_out_sety(TCGContext *s, TCGReg rs)
{
tcg_out32(s, WRY | INSN_RS1(TCG_REG_G0) | INSN_RS2(rs));
@@ -1462,27 +1451,21 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
switch (opc) {
case INDEX_op_goto_tb:
- if (s->tb_jmp_insn_offset) {
- /* direct jump method */
- if (USE_REG_TB) {
- /* make sure the patch is 8-byte aligned. */
- if ((intptr_t)s->code_ptr & 4) {
- tcg_out_nop(s);
- }
- s->tb_jmp_insn_offset[a0] = tcg_current_code_size(s);
- tcg_out_sethi(s, TCG_REG_T1, 0);
- tcg_out_arithi(s, TCG_REG_T1, TCG_REG_T1, 0, ARITH_OR);
- tcg_out_arith(s, TCG_REG_G0, TCG_REG_TB, TCG_REG_T1, JMPL);
- tcg_out_arith(s, TCG_REG_TB, TCG_REG_TB, TCG_REG_T1, ARITH_ADD);
- } else {
- s->tb_jmp_insn_offset[a0] = tcg_current_code_size(s);
- tcg_out32(s, CALL);
+ qemu_build_assert(TCG_TARGET_HAS_direct_jump);
+ /* Direct jump. */
+ if (USE_REG_TB) {
+ /* make sure the patch is 8-byte aligned. */
+ if ((intptr_t)s->code_ptr & 4) {
tcg_out_nop(s);
}
+ s->tb_jmp_insn_offset[a0] = tcg_current_code_size(s);
+ tcg_out_sethi(s, TCG_REG_T1, 0);
+ tcg_out_arithi(s, TCG_REG_T1, TCG_REG_T1, 0, ARITH_OR);
+ tcg_out_arith(s, TCG_REG_G0, TCG_REG_TB, TCG_REG_T1, JMPL);
+ tcg_out_arith(s, TCG_REG_TB, TCG_REG_TB, TCG_REG_T1, ARITH_ADD);
} else {
- /* indirect jump method */
- tcg_out_ld_ptr(s, TCG_REG_TB, s->tb_jmp_target_addr + a0);
- tcg_out_arithi(s, TCG_REG_G0, TCG_REG_TB, 0, JMPL);
+ s->tb_jmp_insn_offset[a0] = tcg_current_code_size(s);
+ tcg_out32(s, CALL);
tcg_out_nop(s);
}
set_jmp_reset_offset(s, a0);
--
2.34.1
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH 05/22] tcg: Replace asserts on tcg_jmp_insn_offset
2022-12-06 4:16 [PATCH 00/22] tcg: exit_tb tidy, goto_tb reorg Richard Henderson
` (3 preceding siblings ...)
2022-12-06 4:16 ` [PATCH 04/22] tcg/sparc64: " Richard Henderson
@ 2022-12-06 4:16 ` Richard Henderson
2022-12-06 7:27 ` Philippe Mathieu-Daudé
2022-12-06 4:16 ` [PATCH 06/22] tcg: Introduce set_jmp_insn_offset Richard Henderson
` (16 subsequent siblings)
21 siblings, 1 reply; 35+ messages in thread
From: Richard Henderson @ 2022-12-06 4:16 UTC (permalink / raw)
To: qemu-devel; +Cc: r
Test TCG_TARGET_HAS_direct_jump instead of testing an
implementation pointer.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
tcg/aarch64/tcg-target.c.inc | 2 +-
tcg/arm/tcg-target.c.inc | 2 +-
tcg/loongarch64/tcg-target.c.inc | 2 +-
tcg/mips/tcg-target.c.inc | 2 +-
tcg/riscv/tcg-target.c.inc | 2 +-
tcg/s390x/tcg-target.c.inc | 2 +-
tcg/tci/tcg-target.c.inc | 2 +-
7 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/tcg/aarch64/tcg-target.c.inc b/tcg/aarch64/tcg-target.c.inc
index 16c5e33b69..8e97da3a39 100644
--- a/tcg/aarch64/tcg-target.c.inc
+++ b/tcg/aarch64/tcg-target.c.inc
@@ -1917,7 +1917,7 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
switch (opc) {
case INDEX_op_goto_tb:
- tcg_debug_assert(s->tb_jmp_insn_offset != NULL);
+ qemu_build_assert(TCG_TARGET_HAS_direct_jump);
/*
* Ensure that ADRP+ADD are 8-byte aligned so that an atomic
* write can be used to patch the target address.
diff --git a/tcg/arm/tcg-target.c.inc b/tcg/arm/tcg-target.c.inc
index e5c2eae5a5..31f8c5b7a7 100644
--- a/tcg/arm/tcg-target.c.inc
+++ b/tcg/arm/tcg-target.c.inc
@@ -1947,7 +1947,7 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
intptr_t ptr, dif, dil;
TCGReg base = TCG_REG_PC;
- tcg_debug_assert(s->tb_jmp_insn_offset == 0);
+ qemu_build_assert(!TCG_TARGET_HAS_direct_jump);
ptr = (intptr_t)tcg_splitwx_to_rx(s->tb_jmp_target_addr + args[0]);
dif = tcg_pcrel_diff(s, (void *)ptr) - 8;
dil = sextract32(dif, 0, 12);
diff --git a/tcg/loongarch64/tcg-target.c.inc b/tcg/loongarch64/tcg-target.c.inc
index 41fc5ffa91..78398684cd 100644
--- a/tcg/loongarch64/tcg-target.c.inc
+++ b/tcg/loongarch64/tcg-target.c.inc
@@ -1089,7 +1089,7 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
switch (opc) {
case INDEX_op_goto_tb:
- tcg_debug_assert(s->tb_jmp_insn_offset != NULL);
+ qemu_build_assert(TCG_TARGET_HAS_direct_jump);
/*
* Ensure that patch area is 8-byte aligned so that an
* atomic write can be used to patch the target address.
diff --git a/tcg/mips/tcg-target.c.inc b/tcg/mips/tcg-target.c.inc
index 819648b100..5fc96e4406 100644
--- a/tcg/mips/tcg-target.c.inc
+++ b/tcg/mips/tcg-target.c.inc
@@ -1986,7 +1986,7 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
switch (opc) {
case INDEX_op_goto_tb:
/* indirect jump method */
- tcg_debug_assert(s->tb_jmp_insn_offset == 0);
+ qemu_build_assert(!TCG_TARGET_HAS_direct_jump);
tcg_out_ld(s, TCG_TYPE_PTR, TCG_TMP0, TCG_REG_ZERO,
(uintptr_t)(s->tb_jmp_target_addr + a0));
tcg_out_opc_reg(s, OPC_JR, 0, TCG_TMP0, 0);
diff --git a/tcg/riscv/tcg-target.c.inc b/tcg/riscv/tcg-target.c.inc
index 1381c835af..1e8406b8c1 100644
--- a/tcg/riscv/tcg-target.c.inc
+++ b/tcg/riscv/tcg-target.c.inc
@@ -1286,7 +1286,7 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
switch (opc) {
case INDEX_op_goto_tb:
- assert(s->tb_jmp_insn_offset == 0);
+ qemu_build_assert(!TCG_TARGET_HAS_direct_jump);
/* indirect jump method */
tcg_out_ld(s, TCG_TYPE_PTR, TCG_REG_TMP0, TCG_REG_ZERO,
(uintptr_t)(s->tb_jmp_target_addr + a0));
diff --git a/tcg/s390x/tcg-target.c.inc b/tcg/s390x/tcg-target.c.inc
index 9568452773..9498694564 100644
--- a/tcg/s390x/tcg-target.c.inc
+++ b/tcg/s390x/tcg-target.c.inc
@@ -2095,7 +2095,7 @@ static inline void tcg_out_op(TCGContext *s, TCGOpcode opc,
switch (opc) {
case INDEX_op_goto_tb:
a0 = args[0];
- if (s->tb_jmp_insn_offset) {
+ if (TCG_TARGET_HAS_direct_jump) {
/*
* branch displacement must be aligned for atomic patching;
* see if we need to add extra nop before branch
diff --git a/tcg/tci/tcg-target.c.inc b/tcg/tci/tcg-target.c.inc
index 34583a3499..d1cc41261a 100644
--- a/tcg/tci/tcg-target.c.inc
+++ b/tcg/tci/tcg-target.c.inc
@@ -606,7 +606,7 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
switch (opc) {
case INDEX_op_goto_tb:
- tcg_debug_assert(s->tb_jmp_insn_offset == 0);
+ qemu_build_assert(!TCG_TARGET_HAS_direct_jump);
/* indirect jump method. */
tcg_out_op_p(s, opc, s->tb_jmp_target_addr + args[0]);
set_jmp_reset_offset(s, args[0]);
--
2.34.1
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH 06/22] tcg: Introduce set_jmp_insn_offset
2022-12-06 4:16 [PATCH 00/22] tcg: exit_tb tidy, goto_tb reorg Richard Henderson
` (4 preceding siblings ...)
2022-12-06 4:16 ` [PATCH 05/22] tcg: Replace asserts on tcg_jmp_insn_offset Richard Henderson
@ 2022-12-06 4:16 ` Richard Henderson
2022-12-06 4:17 ` [PATCH 07/22] tcg: Introduce get_jmp_target_addr Richard Henderson
` (15 subsequent siblings)
21 siblings, 0 replies; 35+ messages in thread
From: Richard Henderson @ 2022-12-06 4:16 UTC (permalink / raw)
To: qemu-devel; +Cc: r
Similar to the existing set_jmp_reset_offset. Move any assert for
TCG_TARGET_HAS_direct_jump into the new function (which now cannot
be build-time). Will be unused if TCG_TARGET_HAS_direct_jump is
constant 0, but we can't test for constant in the preprocessor,
so just mark it G_GNUC_UNUSED.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
tcg/tcg.c | 10 ++++++++++
tcg/aarch64/tcg-target.c.inc | 3 +--
tcg/i386/tcg-target.c.inc | 3 +--
tcg/loongarch64/tcg-target.c.inc | 3 +--
tcg/ppc/tcg-target.c.inc | 7 +++----
tcg/s390x/tcg-target.c.inc | 2 +-
tcg/sparc64/tcg-target.c.inc | 5 ++---
7 files changed, 19 insertions(+), 14 deletions(-)
diff --git a/tcg/tcg.c b/tcg/tcg.c
index b53961baf7..df5a6cedf0 100644
--- a/tcg/tcg.c
+++ b/tcg/tcg.c
@@ -320,6 +320,16 @@ static void set_jmp_reset_offset(TCGContext *s, int which)
s->tb_jmp_reset_offset[which] = tcg_current_code_size(s);
}
+static void G_GNUC_UNUSED set_jmp_insn_offset(TCGContext *s, int which)
+{
+ /*
+ * We will check for overflow at the end of the opcode loop in
+ * tcg_gen_code, where we bound tcg_current_code_size to UINT16_MAX.
+ */
+ tcg_debug_assert(TCG_TARGET_HAS_direct_jump);
+ s->tb_jmp_insn_offset[which] = tcg_current_code_size(s);
+}
+
/* Signal overflow, starting over with fewer guest insns. */
static G_NORETURN
void tcg_raise_tb_overflow(TCGContext *s)
diff --git a/tcg/aarch64/tcg-target.c.inc b/tcg/aarch64/tcg-target.c.inc
index 8e97da3a39..3aa89d6060 100644
--- a/tcg/aarch64/tcg-target.c.inc
+++ b/tcg/aarch64/tcg-target.c.inc
@@ -1917,7 +1917,6 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
switch (opc) {
case INDEX_op_goto_tb:
- qemu_build_assert(TCG_TARGET_HAS_direct_jump);
/*
* Ensure that ADRP+ADD are 8-byte aligned so that an atomic
* write can be used to patch the target address.
@@ -1925,7 +1924,7 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
if ((uintptr_t)s->code_ptr & 7) {
tcg_out32(s, NOP);
}
- s->tb_jmp_insn_offset[a0] = tcg_current_code_size(s);
+ set_jmp_insn_offset(s, a0);
/*
* actual branch destination will be patched by
* tb_target_set_jmp_target later
diff --git a/tcg/i386/tcg-target.c.inc b/tcg/i386/tcg-target.c.inc
index f3a40fc428..3d4cf71552 100644
--- a/tcg/i386/tcg-target.c.inc
+++ b/tcg/i386/tcg-target.c.inc
@@ -2382,7 +2382,6 @@ static inline void tcg_out_op(TCGContext *s, TCGOpcode opc,
switch (opc) {
case INDEX_op_goto_tb:
- qemu_build_assert(TCG_TARGET_HAS_direct_jump);
{
/*
* Jump displacement must be aligned for atomic patching;
@@ -2393,7 +2392,7 @@ static inline void tcg_out_op(TCGContext *s, TCGOpcode opc,
tcg_out_nopn(s, gap - 1);
}
tcg_out8(s, OPC_JMP_long); /* jmp im */
- s->tb_jmp_insn_offset[a0] = tcg_current_code_size(s);
+ set_jmp_insn_offset(s, a0);
tcg_out32(s, 0);
}
set_jmp_reset_offset(s, a0);
diff --git a/tcg/loongarch64/tcg-target.c.inc b/tcg/loongarch64/tcg-target.c.inc
index 78398684cd..dd4934d6d5 100644
--- a/tcg/loongarch64/tcg-target.c.inc
+++ b/tcg/loongarch64/tcg-target.c.inc
@@ -1089,7 +1089,6 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
switch (opc) {
case INDEX_op_goto_tb:
- qemu_build_assert(TCG_TARGET_HAS_direct_jump);
/*
* Ensure that patch area is 8-byte aligned so that an
* atomic write can be used to patch the target address.
@@ -1097,7 +1096,7 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
if ((uintptr_t)s->code_ptr & 7) {
tcg_out_nop(s);
}
- s->tb_jmp_insn_offset[a0] = tcg_current_code_size(s);
+ set_jmp_insn_offset(s, a0);
/*
* actual branch destination will be patched by
* tb_target_set_jmp_target later
diff --git a/tcg/ppc/tcg-target.c.inc b/tcg/ppc/tcg-target.c.inc
index 3f9ee4b39a..86684e1c84 100644
--- a/tcg/ppc/tcg-target.c.inc
+++ b/tcg/ppc/tcg-target.c.inc
@@ -2630,20 +2630,19 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
switch (opc) {
case INDEX_op_goto_tb:
- qemu_build_assert(TCG_TARGET_HAS_direct_jump);
/* Direct jump. */
if (TCG_TARGET_REG_BITS == 64) {
/* Ensure the next insns are 8 or 16-byte aligned. */
while ((uintptr_t)s->code_ptr & (have_isa_2_07 ? 15 : 7)) {
tcg_out32(s, NOP);
}
- s->tb_jmp_insn_offset[args[0]] = tcg_current_code_size(s);
+ set_jmp_insn_offset(s, args[0]);
tcg_out32(s, ADDIS | TAI(TCG_REG_TB, TCG_REG_TB, 0));
tcg_out32(s, ADDI | TAI(TCG_REG_TB, TCG_REG_TB, 0));
} else {
- s->tb_jmp_insn_offset[args[0]] = tcg_current_code_size(s);
+ set_jmp_insn_offset(s, args[0]);
tcg_out32(s, B);
- s->tb_jmp_reset_offset[args[0]] = tcg_current_code_size(s);
+ set_jmp_reset_offset(s, args[0]);
break;
}
tcg_out32(s, MTSPR | RS(TCG_REG_TB) | CTR);
diff --git a/tcg/s390x/tcg-target.c.inc b/tcg/s390x/tcg-target.c.inc
index 9498694564..ad6433fcf2 100644
--- a/tcg/s390x/tcg-target.c.inc
+++ b/tcg/s390x/tcg-target.c.inc
@@ -2105,7 +2105,7 @@ static inline void tcg_out_op(TCGContext *s, TCGOpcode opc,
}
tcg_debug_assert(!USE_REG_TB);
tcg_out16(s, RIL_BRCL | (S390_CC_ALWAYS << 4));
- s->tb_jmp_insn_offset[a0] = tcg_current_code_size(s);
+ set_jmp_insn_offset(s, a0);
s->code_ptr += 2;
} else {
/* load address stored at s->tb_jmp_target_addr + a0 */
diff --git a/tcg/sparc64/tcg-target.c.inc b/tcg/sparc64/tcg-target.c.inc
index f035bf7dd2..f0c6332ef1 100644
--- a/tcg/sparc64/tcg-target.c.inc
+++ b/tcg/sparc64/tcg-target.c.inc
@@ -1451,20 +1451,19 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
switch (opc) {
case INDEX_op_goto_tb:
- qemu_build_assert(TCG_TARGET_HAS_direct_jump);
/* Direct jump. */
if (USE_REG_TB) {
/* make sure the patch is 8-byte aligned. */
if ((intptr_t)s->code_ptr & 4) {
tcg_out_nop(s);
}
- s->tb_jmp_insn_offset[a0] = tcg_current_code_size(s);
+ set_jmp_insn_offset(s, a0);
tcg_out_sethi(s, TCG_REG_T1, 0);
tcg_out_arithi(s, TCG_REG_T1, TCG_REG_T1, 0, ARITH_OR);
tcg_out_arith(s, TCG_REG_G0, TCG_REG_TB, TCG_REG_T1, JMPL);
tcg_out_arith(s, TCG_REG_TB, TCG_REG_TB, TCG_REG_T1, ARITH_ADD);
} else {
- s->tb_jmp_insn_offset[a0] = tcg_current_code_size(s);
+ set_jmp_insn_offset(s, a0);
tcg_out32(s, CALL);
tcg_out_nop(s);
}
--
2.34.1
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH 07/22] tcg: Introduce get_jmp_target_addr
2022-12-06 4:16 [PATCH 00/22] tcg: exit_tb tidy, goto_tb reorg Richard Henderson
` (5 preceding siblings ...)
2022-12-06 4:16 ` [PATCH 06/22] tcg: Introduce set_jmp_insn_offset Richard Henderson
@ 2022-12-06 4:17 ` Richard Henderson
2022-12-06 7:29 ` Philippe Mathieu-Daudé
2022-12-06 4:17 ` [PATCH 08/22] tcg: Split out tcg_out_goto_tb Richard Henderson
` (14 subsequent siblings)
21 siblings, 1 reply; 35+ messages in thread
From: Richard Henderson @ 2022-12-06 4:17 UTC (permalink / raw)
To: qemu-devel; +Cc: r
Similar to the existing set_jmp_reset_offset. Include the
rw->rx address space coversion done by arm and s390x, and
forgotten by mips and riscv.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
tcg/tcg.c | 9 +++++++++
tcg/arm/tcg-target.c.inc | 2 +-
tcg/mips/tcg-target.c.inc | 2 +-
tcg/riscv/tcg-target.c.inc | 2 +-
tcg/s390x/tcg-target.c.inc | 2 +-
tcg/tci/tcg-target.c.inc | 2 +-
6 files changed, 14 insertions(+), 5 deletions(-)
diff --git a/tcg/tcg.c b/tcg/tcg.c
index df5a6cedf0..8bde8eb0d4 100644
--- a/tcg/tcg.c
+++ b/tcg/tcg.c
@@ -330,6 +330,15 @@ static void G_GNUC_UNUSED set_jmp_insn_offset(TCGContext *s, int which)
s->tb_jmp_insn_offset[which] = tcg_current_code_size(s);
}
+static uintptr_t G_GNUC_UNUSED get_jmp_target_addr(TCGContext *s, int which)
+{
+ /*
+ * Return the read-execute version of the pointer, for the benefit
+ * of any pc-relative addressing mode.
+ */
+ return (uintptr_t)tcg_splitwx_to_rx(s->tb_jmp_target_addr + which);
+}
+
/* Signal overflow, starting over with fewer guest insns. */
static G_NORETURN
void tcg_raise_tb_overflow(TCGContext *s)
diff --git a/tcg/arm/tcg-target.c.inc b/tcg/arm/tcg-target.c.inc
index 31f8c5b7a7..3a545f3707 100644
--- a/tcg/arm/tcg-target.c.inc
+++ b/tcg/arm/tcg-target.c.inc
@@ -1948,7 +1948,7 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
TCGReg base = TCG_REG_PC;
qemu_build_assert(!TCG_TARGET_HAS_direct_jump);
- ptr = (intptr_t)tcg_splitwx_to_rx(s->tb_jmp_target_addr + args[0]);
+ ptr = get_jmp_target_addr(s, args[0]);
dif = tcg_pcrel_diff(s, (void *)ptr) - 8;
dil = sextract32(dif, 0, 12);
if (dif != dil) {
diff --git a/tcg/mips/tcg-target.c.inc b/tcg/mips/tcg-target.c.inc
index 5fc96e4406..6b9c070f6b 100644
--- a/tcg/mips/tcg-target.c.inc
+++ b/tcg/mips/tcg-target.c.inc
@@ -1988,7 +1988,7 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
/* indirect jump method */
qemu_build_assert(!TCG_TARGET_HAS_direct_jump);
tcg_out_ld(s, TCG_TYPE_PTR, TCG_TMP0, TCG_REG_ZERO,
- (uintptr_t)(s->tb_jmp_target_addr + a0));
+ get_jmp_target_addr(s, a0));
tcg_out_opc_reg(s, OPC_JR, 0, TCG_TMP0, 0);
tcg_out_nop(s);
set_jmp_reset_offset(s, a0);
diff --git a/tcg/riscv/tcg-target.c.inc b/tcg/riscv/tcg-target.c.inc
index 1e8406b8c1..091c44d071 100644
--- a/tcg/riscv/tcg-target.c.inc
+++ b/tcg/riscv/tcg-target.c.inc
@@ -1289,7 +1289,7 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
qemu_build_assert(!TCG_TARGET_HAS_direct_jump);
/* indirect jump method */
tcg_out_ld(s, TCG_TYPE_PTR, TCG_REG_TMP0, TCG_REG_ZERO,
- (uintptr_t)(s->tb_jmp_target_addr + a0));
+ get_jmp_target_addr(s, a0));
tcg_out_opc_imm(s, OPC_JALR, TCG_REG_ZERO, TCG_REG_TMP0, 0);
set_jmp_reset_offset(s, a0);
break;
diff --git a/tcg/s390x/tcg-target.c.inc b/tcg/s390x/tcg-target.c.inc
index ad6433fcf2..dba02db6ee 100644
--- a/tcg/s390x/tcg-target.c.inc
+++ b/tcg/s390x/tcg-target.c.inc
@@ -2110,7 +2110,7 @@ static inline void tcg_out_op(TCGContext *s, TCGOpcode opc,
} else {
/* load address stored at s->tb_jmp_target_addr + a0 */
tcg_out_ld_abs(s, TCG_TYPE_PTR, TCG_REG_TB,
- tcg_splitwx_to_rx(s->tb_jmp_target_addr + a0));
+ (const void *)get_jmp_target_addr(s, a0));
/* and go there */
tcg_out_insn(s, RR, BCR, S390_CC_ALWAYS, TCG_REG_TB);
}
diff --git a/tcg/tci/tcg-target.c.inc b/tcg/tci/tcg-target.c.inc
index d1cc41261a..baf3480c67 100644
--- a/tcg/tci/tcg-target.c.inc
+++ b/tcg/tci/tcg-target.c.inc
@@ -608,7 +608,7 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
case INDEX_op_goto_tb:
qemu_build_assert(!TCG_TARGET_HAS_direct_jump);
/* indirect jump method. */
- tcg_out_op_p(s, opc, s->tb_jmp_target_addr + args[0]);
+ tcg_out_op_p(s, opc, (void *)get_jmp_target_addr(s, args[0]));
set_jmp_reset_offset(s, args[0]);
break;
--
2.34.1
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH 08/22] tcg: Split out tcg_out_goto_tb
2022-12-06 4:16 [PATCH 00/22] tcg: exit_tb tidy, goto_tb reorg Richard Henderson
` (6 preceding siblings ...)
2022-12-06 4:17 ` [PATCH 07/22] tcg: Introduce get_jmp_target_addr Richard Henderson
@ 2022-12-06 4:17 ` Richard Henderson
2022-12-06 7:36 ` Philippe Mathieu-Daudé
2022-12-06 4:17 ` [PATCH 09/22] tcg: Rename TB_JMP_RESET_OFFSET_INVALID to TB_JMP_OFFSET_INVALID Richard Henderson
` (13 subsequent siblings)
21 siblings, 1 reply; 35+ messages in thread
From: Richard Henderson @ 2022-12-06 4:17 UTC (permalink / raw)
To: qemu-devel; +Cc: r
The INDEX_op_goto_tb opcode needs no register allocation.
Split out a dedicated helper function for it.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
tcg/tcg.c | 4 ++
tcg/aarch64/tcg-target.c.inc | 40 +++++++++---------
tcg/arm/tcg-target.c.inc | 49 +++++++++++-----------
tcg/i386/tcg-target.c.inc | 33 +++++++--------
tcg/loongarch64/tcg-target.c.inc | 38 +++++++++--------
tcg/mips/tcg-target.c.inc | 21 ++++++----
tcg/ppc/tcg-target.c.inc | 52 ++++++++++++------------
tcg/riscv/tcg-target.c.inc | 20 +++++----
tcg/s390x/tcg-target.c.inc | 70 ++++++++++++++++----------------
tcg/sparc64/tcg-target.c.inc | 68 ++++++++++++++++---------------
tcg/tci/tcg-target.c.inc | 16 ++++----
11 files changed, 219 insertions(+), 192 deletions(-)
diff --git a/tcg/tcg.c b/tcg/tcg.c
index 8bde8eb0d4..d334f95375 100644
--- a/tcg/tcg.c
+++ b/tcg/tcg.c
@@ -108,6 +108,7 @@ static bool tcg_out_mov(TCGContext *s, TCGType type, TCGReg ret, TCGReg arg);
static void tcg_out_movi(TCGContext *s, TCGType type,
TCGReg ret, tcg_target_long arg);
static void tcg_out_exit_tb(TCGContext *s, uintptr_t arg);
+static void tcg_out_goto_tb(TCGContext *s, int which);
static void tcg_out_op(TCGContext *s, TCGOpcode opc,
const TCGArg args[TCG_MAX_OP_ARGS],
const int const_args[TCG_MAX_OP_ARGS]);
@@ -4325,6 +4326,9 @@ int tcg_gen_code(TCGContext *s, TranslationBlock *tb, target_ulong pc_start)
case INDEX_op_exit_tb:
tcg_out_exit_tb(s, op->args[0]);
break;
+ case INDEX_op_goto_tb:
+ tcg_out_goto_tb(s, op->args[0]);
+ break;
case INDEX_op_dup2_vec:
if (tcg_reg_alloc_dup2(s, op)) {
break;
diff --git a/tcg/aarch64/tcg-target.c.inc b/tcg/aarch64/tcg-target.c.inc
index 3aa89d6060..072d32f355 100644
--- a/tcg/aarch64/tcg-target.c.inc
+++ b/tcg/aarch64/tcg-target.c.inc
@@ -1897,6 +1897,26 @@ static void tcg_out_exit_tb(TCGContext *s, uintptr_t a0)
}
}
+static void tcg_out_goto_tb(TCGContext *s, int which)
+{
+ /*
+ * Ensure that ADRP+ADD are 8-byte aligned so that an atomic
+ * write can be used to patch the target address.
+ */
+ if ((uintptr_t)s->code_ptr & 7) {
+ tcg_out32(s, NOP);
+ }
+ set_jmp_insn_offset(s, which);
+ /*
+ * actual branch destination will be patched by
+ * tb_target_set_jmp_target later
+ */
+ tcg_out_insn(s, 3406, ADRP, TCG_REG_TMP, 0);
+ tcg_out_insn(s, 3401, ADDI, TCG_TYPE_I64, TCG_REG_TMP, TCG_REG_TMP, 0);
+ tcg_out_insn(s, 3207, BR, TCG_REG_TMP);
+ set_jmp_reset_offset(s, which);
+}
+
static void tcg_out_op(TCGContext *s, TCGOpcode opc,
const TCGArg args[TCG_MAX_OP_ARGS],
const int const_args[TCG_MAX_OP_ARGS])
@@ -1916,25 +1936,6 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
#define REG0(I) (const_args[I] ? TCG_REG_XZR : (TCGReg)args[I])
switch (opc) {
- case INDEX_op_goto_tb:
- /*
- * Ensure that ADRP+ADD are 8-byte aligned so that an atomic
- * write can be used to patch the target address.
- */
- if ((uintptr_t)s->code_ptr & 7) {
- tcg_out32(s, NOP);
- }
- set_jmp_insn_offset(s, a0);
- /*
- * actual branch destination will be patched by
- * tb_target_set_jmp_target later
- */
- tcg_out_insn(s, 3406, ADRP, TCG_REG_TMP, 0);
- tcg_out_insn(s, 3401, ADDI, TCG_TYPE_I64, TCG_REG_TMP, TCG_REG_TMP, 0);
- tcg_out_insn(s, 3207, BR, TCG_REG_TMP);
- set_jmp_reset_offset(s, a0);
- break;
-
case INDEX_op_goto_ptr:
tcg_out_insn(s, 3207, BR, a0);
break;
@@ -2305,6 +2306,7 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
case INDEX_op_mov_i64:
case INDEX_op_call: /* Always emitted via tcg_out_call. */
case INDEX_op_exit_tb: /* Always emitted via tcg_out_exit_tb. */
+ case INDEX_op_goto_tb: /* Always emitted via tcg_out_goto_tb. */
default:
g_assert_not_reached();
}
diff --git a/tcg/arm/tcg-target.c.inc b/tcg/arm/tcg-target.c.inc
index 3a545f3707..0afc286c8e 100644
--- a/tcg/arm/tcg-target.c.inc
+++ b/tcg/arm/tcg-target.c.inc
@@ -1933,6 +1933,31 @@ static void tcg_out_exit_tb(TCGContext *s, uintptr_t arg)
tcg_out_epilogue(s);
}
+static void tcg_out_goto_tb(TCGContext *s, int which)
+{
+ /* Indirect jump method */
+ intptr_t ptr, dif, dil;
+ TCGReg base = TCG_REG_PC;
+
+ qemu_build_assert(!TCG_TARGET_HAS_direct_jump);
+ ptr = get_jmp_target_addr(s, which);
+ dif = tcg_pcrel_diff(s, (void *)ptr) - 8;
+ dil = sextract32(dif, 0, 12);
+ if (dif != dil) {
+ /*
+ * The TB is close, but outside the 12 bits addressable by
+ * the load. We can extend this to 20 bits with a sub of a
+ * shifted immediate from pc. In the vastly unlikely event
+ * the code requires more than 1MB, we'll use 2 insns and
+ * be no worse off.
+ */
+ base = TCG_REG_R0;
+ tcg_out_movi32(s, COND_AL, base, ptr - dil);
+ }
+ tcg_out_ld32_12(s, COND_AL, TCG_REG_PC, base, dil);
+ set_jmp_reset_offset(s, which);
+}
+
static void tcg_out_op(TCGContext *s, TCGOpcode opc,
const TCGArg args[TCG_MAX_OP_ARGS],
const int const_args[TCG_MAX_OP_ARGS])
@@ -1941,29 +1966,6 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
int c;
switch (opc) {
- case INDEX_op_goto_tb:
- {
- /* Indirect jump method */
- intptr_t ptr, dif, dil;
- TCGReg base = TCG_REG_PC;
-
- qemu_build_assert(!TCG_TARGET_HAS_direct_jump);
- ptr = get_jmp_target_addr(s, args[0]);
- dif = tcg_pcrel_diff(s, (void *)ptr) - 8;
- dil = sextract32(dif, 0, 12);
- if (dif != dil) {
- /* The TB is close, but outside the 12 bits addressable by
- the load. We can extend this to 20 bits with a sub of a
- shifted immediate from pc. In the vastly unlikely event
- the code requires more than 1MB, we'll use 2 insns and
- be no worse off. */
- base = TCG_REG_R0;
- tcg_out_movi32(s, COND_AL, base, ptr - dil);
- }
- tcg_out_ld32_12(s, COND_AL, TCG_REG_PC, base, dil);
- set_jmp_reset_offset(s, args[0]);
- }
- break;
case INDEX_op_goto_ptr:
tcg_out_b_reg(s, COND_AL, args[0]);
break;
@@ -2253,6 +2255,7 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
case INDEX_op_mov_i32: /* Always emitted via tcg_out_mov. */
case INDEX_op_call: /* Always emitted via tcg_out_call. */
case INDEX_op_exit_tb: /* Always emitted via tcg_out_exit_tb. */
+ case INDEX_op_goto_tb: /* Always emitted via tcg_out_goto_tb. */
default:
tcg_abort();
}
diff --git a/tcg/i386/tcg-target.c.inc b/tcg/i386/tcg-target.c.inc
index 3d4cf71552..d3cef242ef 100644
--- a/tcg/i386/tcg-target.c.inc
+++ b/tcg/i386/tcg-target.c.inc
@@ -2357,6 +2357,22 @@ static void tcg_out_exit_tb(TCGContext *s, uintptr_t a0)
}
}
+static void tcg_out_goto_tb(TCGContext *s, int which)
+{
+ /*
+ * Jump displacement must be aligned for atomic patching;
+ * see if we need to add extra nops before jump
+ */
+ int gap = QEMU_ALIGN_PTR_UP(s->code_ptr + 1, 4) - s->code_ptr;
+ if (gap != 1) {
+ tcg_out_nopn(s, gap - 1);
+ }
+ tcg_out8(s, OPC_JMP_long); /* jmp im */
+ set_jmp_insn_offset(s, which);
+ tcg_out32(s, 0);
+ set_jmp_reset_offset(s, which);
+}
+
static inline void tcg_out_op(TCGContext *s, TCGOpcode opc,
const TCGArg args[TCG_MAX_OP_ARGS],
const int const_args[TCG_MAX_OP_ARGS])
@@ -2381,22 +2397,6 @@ static inline void tcg_out_op(TCGContext *s, TCGOpcode opc,
const_a2 = const_args[2];
switch (opc) {
- case INDEX_op_goto_tb:
- {
- /*
- * Jump displacement must be aligned for atomic patching;
- * see if we need to add extra nops before jump
- */
- int gap = QEMU_ALIGN_PTR_UP(s->code_ptr + 1, 4) - s->code_ptr;
- if (gap != 1) {
- tcg_out_nopn(s, gap - 1);
- }
- tcg_out8(s, OPC_JMP_long); /* jmp im */
- set_jmp_insn_offset(s, a0);
- tcg_out32(s, 0);
- }
- set_jmp_reset_offset(s, a0);
- break;
case INDEX_op_goto_ptr:
/* jmp to the given host address (could be epilogue) */
tcg_out_modrm(s, OPC_GRP5, EXT5_JMPN_Ev, a0);
@@ -2791,6 +2791,7 @@ static inline void tcg_out_op(TCGContext *s, TCGOpcode opc,
case INDEX_op_mov_i64:
case INDEX_op_call: /* Always emitted via tcg_out_call. */
case INDEX_op_exit_tb: /* Always emitted via tcg_out_exit_tb. */
+ case INDEX_op_goto_tb: /* Always emitted via tcg_out_goto_tb. */
default:
tcg_abort();
}
diff --git a/tcg/loongarch64/tcg-target.c.inc b/tcg/loongarch64/tcg-target.c.inc
index dd4934d6d5..dbe64b7d8b 100644
--- a/tcg/loongarch64/tcg-target.c.inc
+++ b/tcg/loongarch64/tcg-target.c.inc
@@ -1078,6 +1078,25 @@ static void tcg_out_exit_tb(TCGContext *s, uintptr_t a0)
}
}
+static void tcg_out_goto_tb(TCGContext *s, int which)
+{
+ /*
+ * Ensure that patch area is 8-byte aligned so that an
+ * atomic write can be used to patch the target address.
+ */
+ if ((uintptr_t)s->code_ptr & 7) {
+ tcg_out_nop(s);
+ }
+ set_jmp_insn_offset(s, which);
+ /*
+ * actual branch destination will be patched by
+ * tb_target_set_jmp_target later
+ */
+ tcg_out_opc_pcaddu18i(s, TCG_REG_TMP0, 0);
+ tcg_out_opc_jirl(s, TCG_REG_ZERO, TCG_REG_TMP0, 0);
+ set_jmp_reset_offset(s, which);
+}
+
static void tcg_out_op(TCGContext *s, TCGOpcode opc,
const TCGArg args[TCG_MAX_OP_ARGS],
const int const_args[TCG_MAX_OP_ARGS])
@@ -1088,24 +1107,6 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
int c2 = const_args[2];
switch (opc) {
- case INDEX_op_goto_tb:
- /*
- * Ensure that patch area is 8-byte aligned so that an
- * atomic write can be used to patch the target address.
- */
- if ((uintptr_t)s->code_ptr & 7) {
- tcg_out_nop(s);
- }
- set_jmp_insn_offset(s, a0);
- /*
- * actual branch destination will be patched by
- * tb_target_set_jmp_target later
- */
- tcg_out_opc_pcaddu18i(s, TCG_REG_TMP0, 0);
- tcg_out_opc_jirl(s, TCG_REG_ZERO, TCG_REG_TMP0, 0);
- set_jmp_reset_offset(s, a0);
- break;
-
case INDEX_op_mb:
tcg_out_mb(s, a0);
break;
@@ -1500,6 +1501,7 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
case INDEX_op_mov_i64:
case INDEX_op_call: /* Always emitted via tcg_out_call. */
case INDEX_op_exit_tb: /* Always emitted via tcg_out_exit_tb. */
+ case INDEX_op_goto_tb: /* Always emitted via tcg_out_goto_tb. */
default:
g_assert_not_reached();
}
diff --git a/tcg/mips/tcg-target.c.inc b/tcg/mips/tcg-target.c.inc
index 6b9c070f6b..605aa7714a 100644
--- a/tcg/mips/tcg-target.c.inc
+++ b/tcg/mips/tcg-target.c.inc
@@ -1965,6 +1965,17 @@ static void tcg_out_exit_tb(TCGContext *s, uintptr_t a0)
tcg_out_opc_imm(s, OPC_ORI, TCG_REG_V0, b0, a0 & 0xffff);
}
+static void tcg_out_goto_tb(TCGContext *s, int which)
+{
+ /* indirect jump method */
+ qemu_build_assert(!TCG_TARGET_HAS_direct_jump);
+ tcg_out_ld(s, TCG_TYPE_PTR, TCG_TMP0, TCG_REG_ZERO,
+ get_jmp_target_addr(s, which));
+ tcg_out_opc_reg(s, OPC_JR, 0, TCG_TMP0, 0);
+ tcg_out_nop(s);
+ set_jmp_reset_offset(s, which);
+}
+
static void tcg_out_op(TCGContext *s, TCGOpcode opc,
const TCGArg args[TCG_MAX_OP_ARGS],
const int const_args[TCG_MAX_OP_ARGS])
@@ -1984,15 +1995,6 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
c2 = const_args[2];
switch (opc) {
- case INDEX_op_goto_tb:
- /* indirect jump method */
- qemu_build_assert(!TCG_TARGET_HAS_direct_jump);
- tcg_out_ld(s, TCG_TYPE_PTR, TCG_TMP0, TCG_REG_ZERO,
- get_jmp_target_addr(s, a0));
- tcg_out_opc_reg(s, OPC_JR, 0, TCG_TMP0, 0);
- tcg_out_nop(s);
- set_jmp_reset_offset(s, a0);
- break;
case INDEX_op_goto_ptr:
/* jmp to the given host address (could be epilogue) */
tcg_out_opc_reg(s, OPC_JR, 0, a0, 0);
@@ -2401,6 +2403,7 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
case INDEX_op_mov_i64:
case INDEX_op_call: /* Always emitted via tcg_out_call. */
case INDEX_op_exit_tb: /* Always emitted via tcg_out_exit_tb. */
+ case INDEX_op_goto_tb: /* Always emitted via tcg_out_goto_tb. */
default:
tcg_abort();
}
diff --git a/tcg/ppc/tcg-target.c.inc b/tcg/ppc/tcg-target.c.inc
index 86684e1c84..ec13ea3171 100644
--- a/tcg/ppc/tcg-target.c.inc
+++ b/tcg/ppc/tcg-target.c.inc
@@ -2622,6 +2622,32 @@ static void tcg_out_exit_tb(TCGContext *s, uintptr_t arg)
tcg_out_b(s, 0, tcg_code_gen_epilogue);
}
+static void tcg_out_goto_tb(TCGContext *s, int which)
+{
+ /* Direct jump. */
+ if (TCG_TARGET_REG_BITS == 64) {
+ /* Ensure the next insns are 8 or 16-byte aligned. */
+ while ((uintptr_t)s->code_ptr & (have_isa_2_07 ? 15 : 7)) {
+ tcg_out32(s, NOP);
+ }
+ set_jmp_insn_offset(s, which);
+ tcg_out32(s, ADDIS | TAI(TCG_REG_TB, TCG_REG_TB, 0));
+ tcg_out32(s, ADDI | TAI(TCG_REG_TB, TCG_REG_TB, 0));
+ tcg_out32(s, MTSPR | RS(TCG_REG_TB) | CTR);
+ tcg_out32(s, BCCTR | BO_ALWAYS);
+ set_jmp_reset_offset(s, which);
+ if (USE_REG_TB) {
+ /* For the unlinked case, need to reset TCG_REG_TB. */
+ tcg_out_mem_long(s, ADDI, ADD, TCG_REG_TB, TCG_REG_TB,
+ -tcg_current_code_size(s));
+ }
+ } else {
+ set_jmp_insn_offset(s, which);
+ tcg_out32(s, B);
+ set_jmp_reset_offset(s, which);
+ }
+}
+
static void tcg_out_op(TCGContext *s, TCGOpcode opc,
const TCGArg args[TCG_MAX_OP_ARGS],
const int const_args[TCG_MAX_OP_ARGS])
@@ -2629,31 +2655,6 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
TCGArg a0, a1, a2;
switch (opc) {
- case INDEX_op_goto_tb:
- /* Direct jump. */
- if (TCG_TARGET_REG_BITS == 64) {
- /* Ensure the next insns are 8 or 16-byte aligned. */
- while ((uintptr_t)s->code_ptr & (have_isa_2_07 ? 15 : 7)) {
- tcg_out32(s, NOP);
- }
- set_jmp_insn_offset(s, args[0]);
- tcg_out32(s, ADDIS | TAI(TCG_REG_TB, TCG_REG_TB, 0));
- tcg_out32(s, ADDI | TAI(TCG_REG_TB, TCG_REG_TB, 0));
- } else {
- set_jmp_insn_offset(s, args[0]);
- tcg_out32(s, B);
- set_jmp_reset_offset(s, args[0]);
- break;
- }
- tcg_out32(s, MTSPR | RS(TCG_REG_TB) | CTR);
- tcg_out32(s, BCCTR | BO_ALWAYS);
- set_jmp_reset_offset(s, args[0]);
- if (USE_REG_TB) {
- /* For the unlinked case, need to reset TCG_REG_TB. */
- tcg_out_mem_long(s, ADDI, ADD, TCG_REG_TB, TCG_REG_TB,
- -tcg_current_code_size(s));
- }
- break;
case INDEX_op_goto_ptr:
tcg_out32(s, MTSPR | RS(args[0]) | CTR);
if (USE_REG_TB) {
@@ -3181,6 +3182,7 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
case INDEX_op_mov_i64:
case INDEX_op_call: /* Always emitted via tcg_out_call. */
case INDEX_op_exit_tb: /* Always emitted via tcg_out_exit_tb. */
+ case INDEX_op_goto_tb: /* Always emitted via tcg_out_goto_tb. */
default:
tcg_abort();
}
diff --git a/tcg/riscv/tcg-target.c.inc b/tcg/riscv/tcg-target.c.inc
index 091c44d071..e73dfb4cbb 100644
--- a/tcg/riscv/tcg-target.c.inc
+++ b/tcg/riscv/tcg-target.c.inc
@@ -1275,6 +1275,16 @@ static void tcg_out_exit_tb(TCGContext *s, uintptr_t a0)
}
}
+static void tcg_out_goto_tb(TCGContext *s, int which)
+{
+ qemu_build_assert(!TCG_TARGET_HAS_direct_jump);
+ /* indirect jump method */
+ tcg_out_ld(s, TCG_TYPE_PTR, TCG_REG_TMP0, TCG_REG_ZERO,
+ get_jmp_target_addr(s, which));
+ tcg_out_opc_imm(s, OPC_JALR, TCG_REG_ZERO, TCG_REG_TMP0, 0);
+ set_jmp_reset_offset(s, which);
+}
+
static void tcg_out_op(TCGContext *s, TCGOpcode opc,
const TCGArg args[TCG_MAX_OP_ARGS],
const int const_args[TCG_MAX_OP_ARGS])
@@ -1285,15 +1295,6 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
int c2 = const_args[2];
switch (opc) {
- case INDEX_op_goto_tb:
- qemu_build_assert(!TCG_TARGET_HAS_direct_jump);
- /* indirect jump method */
- tcg_out_ld(s, TCG_TYPE_PTR, TCG_REG_TMP0, TCG_REG_ZERO,
- get_jmp_target_addr(s, a0));
- tcg_out_opc_imm(s, OPC_JALR, TCG_REG_ZERO, TCG_REG_TMP0, 0);
- set_jmp_reset_offset(s, a0);
- break;
-
case INDEX_op_goto_ptr:
tcg_out_opc_imm(s, OPC_JALR, TCG_REG_ZERO, a0, 0);
break;
@@ -1594,6 +1595,7 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
case INDEX_op_mov_i64:
case INDEX_op_call: /* Always emitted via tcg_out_call. */
case INDEX_op_exit_tb: /* Always emitted via tcg_out_exit_tb. */
+ case INDEX_op_goto_tb: /* Always emitted via tcg_out_goto_tb. */
default:
g_assert_not_reached();
}
diff --git a/tcg/s390x/tcg-target.c.inc b/tcg/s390x/tcg-target.c.inc
index dba02db6ee..a60edc18cf 100644
--- a/tcg/s390x/tcg-target.c.inc
+++ b/tcg/s390x/tcg-target.c.inc
@@ -2081,6 +2081,41 @@ static void tcg_out_exit_tb(TCGContext *s, uintptr_t a0)
}
}
+static void tcg_out_goto_tb(TCGContext *s, int which)
+{
+ if (TCG_TARGET_HAS_direct_jump) {
+ /*
+ * Branch displacement must be aligned for atomic patching;
+ * see if we need to add extra nop before branch
+ */
+ if (!QEMU_PTR_IS_ALIGNED(s->code_ptr + 1, 4)) {
+ tcg_out16(s, NOP);
+ }
+ tcg_debug_assert(!USE_REG_TB);
+ tcg_out16(s, RIL_BRCL | (S390_CC_ALWAYS << 4));
+ set_jmp_insn_offset(s, which);
+ s->code_ptr += 2;
+ } else {
+ /* Load address stored in the TB. */
+ tcg_out_ld_abs(s, TCG_TYPE_PTR, TCG_REG_TB,
+ (const void *)get_jmp_target_addr(s, which));
+ /* and go there */
+ tcg_out_insn(s, RR, BCR, S390_CC_ALWAYS, TCG_REG_TB);
+ }
+ set_jmp_reset_offset(s, which);
+
+ /*
+ * For the unlinked path of goto_tb, we need to reset TCG_REG_TB
+ * to the beginning of this TB.
+ */
+ if (USE_REG_TB) {
+ int ofs = -tcg_current_code_size(s);
+ /* All TB are restricted to 64KiB by unwind info. */
+ tcg_debug_assert(ofs == sextract64(ofs, 0, 20));
+ tcg_out_insn(s, RXY, LAY, TCG_REG_TB, TCG_REG_TB, TCG_REG_NONE, ofs);
+ }
+}
+
# define OP_32_64(x) \
case glue(glue(INDEX_op_,x),_i32): \
case glue(glue(INDEX_op_,x),_i64)
@@ -2093,40 +2128,6 @@ static inline void tcg_out_op(TCGContext *s, TCGOpcode opc,
TCGArg a0, a1, a2;
switch (opc) {
- case INDEX_op_goto_tb:
- a0 = args[0];
- if (TCG_TARGET_HAS_direct_jump) {
- /*
- * branch displacement must be aligned for atomic patching;
- * see if we need to add extra nop before branch
- */
- if (!QEMU_PTR_IS_ALIGNED(s->code_ptr + 1, 4)) {
- tcg_out16(s, NOP);
- }
- tcg_debug_assert(!USE_REG_TB);
- tcg_out16(s, RIL_BRCL | (S390_CC_ALWAYS << 4));
- set_jmp_insn_offset(s, a0);
- s->code_ptr += 2;
- } else {
- /* load address stored at s->tb_jmp_target_addr + a0 */
- tcg_out_ld_abs(s, TCG_TYPE_PTR, TCG_REG_TB,
- (const void *)get_jmp_target_addr(s, a0));
- /* and go there */
- tcg_out_insn(s, RR, BCR, S390_CC_ALWAYS, TCG_REG_TB);
- }
- set_jmp_reset_offset(s, a0);
-
- /* For the unlinked path of goto_tb, we need to reset
- TCG_REG_TB to the beginning of this TB. */
- if (USE_REG_TB) {
- int ofs = -tcg_current_code_size(s);
- /* All TB are restricted to 64KiB by unwind info. */
- tcg_debug_assert(ofs == sextract64(ofs, 0, 20));
- tcg_out_insn(s, RXY, LAY, TCG_REG_TB,
- TCG_REG_TB, TCG_REG_NONE, ofs);
- }
- break;
-
case INDEX_op_goto_ptr:
a0 = args[0];
if (USE_REG_TB) {
@@ -2662,6 +2663,7 @@ static inline void tcg_out_op(TCGContext *s, TCGOpcode opc,
case INDEX_op_mov_i64:
case INDEX_op_call: /* Always emitted via tcg_out_call. */
case INDEX_op_exit_tb: /* Always emitted via tcg_out_exit_tb. */
+ case INDEX_op_goto_tb: /* Always emitted via tcg_out_goto_tb. */
default:
tcg_abort();
}
diff --git a/tcg/sparc64/tcg-target.c.inc b/tcg/sparc64/tcg-target.c.inc
index f0c6332ef1..ed8efa413c 100644
--- a/tcg/sparc64/tcg-target.c.inc
+++ b/tcg/sparc64/tcg-target.c.inc
@@ -1436,6 +1436,41 @@ static void tcg_out_exit_tb(TCGContext *s, uintptr_t a0)
tcg_out_arithi(s, TCG_REG_O0, TCG_REG_O0, a0 & 0x3ff, ARITH_OR);
}
+static void tcg_out_goto_tb(TCGContext *s, int which)
+{
+ /* Direct jump. */
+ if (USE_REG_TB) {
+ /* make sure the patch is 8-byte aligned. */
+ if ((intptr_t)s->code_ptr & 4) {
+ tcg_out_nop(s);
+ }
+ set_jmp_insn_offset(s, which);
+ tcg_out_sethi(s, TCG_REG_T1, 0);
+ tcg_out_arithi(s, TCG_REG_T1, TCG_REG_T1, 0, ARITH_OR);
+ tcg_out_arith(s, TCG_REG_G0, TCG_REG_TB, TCG_REG_T1, JMPL);
+ tcg_out_arith(s, TCG_REG_TB, TCG_REG_TB, TCG_REG_T1, ARITH_ADD);
+ } else {
+ set_jmp_insn_offset(s, which);
+ tcg_out32(s, CALL);
+ tcg_out_nop(s);
+ }
+ set_jmp_reset_offset(s, which);
+
+ /*
+ * For the unlinked path of goto_tb, we need to reset TCG_REG_TB
+ * to the beginning of this TB.
+ */
+ if (USE_REG_TB) {
+ int c = -tcg_current_code_size(s);
+ if (check_fit_i32(c, 13)) {
+ tcg_out_arithi(s, TCG_REG_TB, TCG_REG_TB, c, ARITH_ADD);
+ } else {
+ tcg_out_movi(s, TCG_TYPE_PTR, TCG_REG_T1, c);
+ tcg_out_arith(s, TCG_REG_TB, TCG_REG_TB, TCG_REG_T1, ARITH_ADD);
+ }
+ }
+}
+
static void tcg_out_op(TCGContext *s, TCGOpcode opc,
const TCGArg args[TCG_MAX_OP_ARGS],
const int const_args[TCG_MAX_OP_ARGS])
@@ -1450,38 +1485,6 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
c2 = const_args[2];
switch (opc) {
- case INDEX_op_goto_tb:
- /* Direct jump. */
- if (USE_REG_TB) {
- /* make sure the patch is 8-byte aligned. */
- if ((intptr_t)s->code_ptr & 4) {
- tcg_out_nop(s);
- }
- set_jmp_insn_offset(s, a0);
- tcg_out_sethi(s, TCG_REG_T1, 0);
- tcg_out_arithi(s, TCG_REG_T1, TCG_REG_T1, 0, ARITH_OR);
- tcg_out_arith(s, TCG_REG_G0, TCG_REG_TB, TCG_REG_T1, JMPL);
- tcg_out_arith(s, TCG_REG_TB, TCG_REG_TB, TCG_REG_T1, ARITH_ADD);
- } else {
- set_jmp_insn_offset(s, a0);
- tcg_out32(s, CALL);
- tcg_out_nop(s);
- }
- set_jmp_reset_offset(s, a0);
-
- /* For the unlinked path of goto_tb, we need to reset
- TCG_REG_TB to the beginning of this TB. */
- if (USE_REG_TB) {
- c = -tcg_current_code_size(s);
- if (check_fit_i32(c, 13)) {
- tcg_out_arithi(s, TCG_REG_TB, TCG_REG_TB, c, ARITH_ADD);
- } else {
- tcg_out_movi(s, TCG_TYPE_PTR, TCG_REG_T1, c);
- tcg_out_arith(s, TCG_REG_TB, TCG_REG_TB,
- TCG_REG_T1, ARITH_ADD);
- }
- }
- break;
case INDEX_op_goto_ptr:
tcg_out_arithi(s, TCG_REG_G0, a0, 0, JMPL);
if (USE_REG_TB) {
@@ -1700,6 +1703,7 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
case INDEX_op_mov_i64:
case INDEX_op_call: /* Always emitted via tcg_out_call. */
case INDEX_op_exit_tb: /* Always emitted via tcg_out_exit_tb. */
+ case INDEX_op_goto_tb: /* Always emitted via tcg_out_goto_tb. */
default:
tcg_abort();
}
diff --git a/tcg/tci/tcg-target.c.inc b/tcg/tci/tcg-target.c.inc
index baf3480c67..cf84ff3133 100644
--- a/tcg/tci/tcg-target.c.inc
+++ b/tcg/tci/tcg-target.c.inc
@@ -598,6 +598,14 @@ static void tcg_out_exit_tb(TCGContext *s, uintptr_t arg)
tcg_out_op_p(s, INDEX_op_exit_tb, (void *)arg);
}
+static void tcg_out_goto_tb(TCGContext *s, int which)
+{
+ qemu_build_assert(!TCG_TARGET_HAS_direct_jump);
+ /* indirect jump method. */
+ tcg_out_op_p(s, INDEX_op_goto_tb, (void *)get_jmp_target_addr(s, which));
+ set_jmp_reset_offset(s, which);
+}
+
static void tcg_out_op(TCGContext *s, TCGOpcode opc,
const TCGArg args[TCG_MAX_OP_ARGS],
const int const_args[TCG_MAX_OP_ARGS])
@@ -605,13 +613,6 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
TCGOpcode exts;
switch (opc) {
- case INDEX_op_goto_tb:
- qemu_build_assert(!TCG_TARGET_HAS_direct_jump);
- /* indirect jump method. */
- tcg_out_op_p(s, opc, (void *)get_jmp_target_addr(s, args[0]));
- set_jmp_reset_offset(s, args[0]);
- break;
-
case INDEX_op_goto_ptr:
tcg_out_op_r(s, opc, args[0]);
break;
@@ -784,6 +785,7 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
case INDEX_op_mov_i64:
case INDEX_op_call: /* Always emitted via tcg_out_call. */
case INDEX_op_exit_tb: /* Always emitted via tcg_out_exit_tb. */
+ case INDEX_op_goto_tb: /* Always emitted via tcg_out_goto_tb. */
default:
tcg_abort();
}
--
2.34.1
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH 09/22] tcg: Rename TB_JMP_RESET_OFFSET_INVALID to TB_JMP_OFFSET_INVALID
2022-12-06 4:16 [PATCH 00/22] tcg: exit_tb tidy, goto_tb reorg Richard Henderson
` (7 preceding siblings ...)
2022-12-06 4:17 ` [PATCH 08/22] tcg: Split out tcg_out_goto_tb Richard Henderson
@ 2022-12-06 4:17 ` Richard Henderson
2022-12-06 7:36 ` Philippe Mathieu-Daudé
2022-12-06 4:17 ` [PATCH 10/22] tcg: Add gen_tb to TCGContext Richard Henderson
` (12 subsequent siblings)
21 siblings, 1 reply; 35+ messages in thread
From: Richard Henderson @ 2022-12-06 4:17 UTC (permalink / raw)
To: qemu-devel; +Cc: r
This will shortly be used for more than reset.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
include/exec/exec-all.h | 2 +-
accel/tcg/translate-all.c | 8 ++++----
tcg/tcg.c | 4 ++--
3 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h
index 9b7bfbf09a..7566ad9e7f 100644
--- a/include/exec/exec-all.h
+++ b/include/exec/exec-all.h
@@ -575,8 +575,8 @@ struct TranslationBlock {
* setting one of the jump targets (or patching the jump instruction). Only
* two of such jumps are supported.
*/
+#define TB_JMP_OFFSET_INVALID 0xffff /* indicates no jump generated */
uint16_t jmp_reset_offset[2]; /* offset of original jump target */
-#define TB_JMP_RESET_OFFSET_INVALID 0xffff /* indicates no jump generated */
uintptr_t jmp_target_arg[2]; /* target address or offset */
/*
diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c
index ac3ee3740c..9cf88da6cb 100644
--- a/accel/tcg/translate-all.c
+++ b/accel/tcg/translate-all.c
@@ -979,10 +979,10 @@ TranslationBlock *tb_gen_code(CPUState *cpu,
tb->jmp_dest[1] = (uintptr_t)NULL;
/* init original jump addresses which have been set during tcg_gen_code() */
- if (tb->jmp_reset_offset[0] != TB_JMP_RESET_OFFSET_INVALID) {
+ if (tb->jmp_reset_offset[0] != TB_JMP_OFFSET_INVALID) {
tb_reset_jump(tb, 0);
}
- if (tb->jmp_reset_offset[1] != TB_JMP_RESET_OFFSET_INVALID) {
+ if (tb->jmp_reset_offset[1] != TB_JMP_OFFSET_INVALID) {
tb_reset_jump(tb, 1);
}
@@ -1164,9 +1164,9 @@ static gboolean tb_tree_stats_iter(gpointer key, gpointer value, gpointer data)
if (tb_page_addr1(tb) != -1) {
tst->cross_page++;
}
- if (tb->jmp_reset_offset[0] != TB_JMP_RESET_OFFSET_INVALID) {
+ if (tb->jmp_reset_offset[0] != TB_JMP_OFFSET_INVALID) {
tst->direct_jmp_count++;
- if (tb->jmp_reset_offset[1] != TB_JMP_RESET_OFFSET_INVALID) {
+ if (tb->jmp_reset_offset[1] != TB_JMP_OFFSET_INVALID) {
tst->direct_jmp2_count++;
}
}
diff --git a/tcg/tcg.c b/tcg/tcg.c
index d334f95375..37957208c7 100644
--- a/tcg/tcg.c
+++ b/tcg/tcg.c
@@ -4250,8 +4250,8 @@ int tcg_gen_code(TCGContext *s, TranslationBlock *tb, target_ulong pc_start)
#endif
/* Initialize goto_tb jump offsets. */
- tb->jmp_reset_offset[0] = TB_JMP_RESET_OFFSET_INVALID;
- tb->jmp_reset_offset[1] = TB_JMP_RESET_OFFSET_INVALID;
+ tb->jmp_reset_offset[0] = TB_JMP_OFFSET_INVALID;
+ tb->jmp_reset_offset[1] = TB_JMP_OFFSET_INVALID;
tcg_ctx->tb_jmp_reset_offset = tb->jmp_reset_offset;
if (TCG_TARGET_HAS_direct_jump) {
tcg_ctx->tb_jmp_insn_offset = tb->jmp_target_arg;
--
2.34.1
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH 10/22] tcg: Add gen_tb to TCGContext
2022-12-06 4:16 [PATCH 00/22] tcg: exit_tb tidy, goto_tb reorg Richard Henderson
` (8 preceding siblings ...)
2022-12-06 4:17 ` [PATCH 09/22] tcg: Rename TB_JMP_RESET_OFFSET_INVALID to TB_JMP_OFFSET_INVALID Richard Henderson
@ 2022-12-06 4:17 ` Richard Henderson
2022-12-06 4:17 ` [PATCH 11/22] tcg: Add TranslationBlock.jmp_insn_offset Richard Henderson
` (11 subsequent siblings)
21 siblings, 0 replies; 35+ messages in thread
From: Richard Henderson @ 2022-12-06 4:17 UTC (permalink / raw)
To: qemu-devel; +Cc: r
This can replace four other variables that are references
into the TranslationBlock structure.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
include/tcg/tcg.h | 11 +++--------
accel/tcg/translate-all.c | 2 +-
tcg/tcg-op.c | 14 +++++++-------
tcg/tcg.c | 14 +++-----------
4 files changed, 14 insertions(+), 27 deletions(-)
diff --git a/include/tcg/tcg.h b/include/tcg/tcg.h
index d84bae6e3f..993aafa1a2 100644
--- a/include/tcg/tcg.h
+++ b/include/tcg/tcg.h
@@ -544,20 +544,15 @@ struct TCGContext {
int nb_indirects;
int nb_ops;
- /* goto_tb support */
- tcg_insn_unit *code_buf;
- uint16_t *tb_jmp_reset_offset; /* tb->jmp_reset_offset */
- uintptr_t *tb_jmp_insn_offset; /* tb->jmp_target_arg if direct_jump */
- uintptr_t *tb_jmp_target_addr; /* tb->jmp_target_arg if !direct_jump */
-
TCGRegSet reserved_regs;
- uint32_t tb_cflags; /* cflags of the current TB */
intptr_t current_frame_offset;
intptr_t frame_start;
intptr_t frame_end;
TCGTemp *frame_temp;
- tcg_insn_unit *code_ptr;
+ TranslationBlock *gen_tb; /* tb for which code is being generated */
+ tcg_insn_unit *code_buf; /* pointer for start of tb */
+ tcg_insn_unit *code_ptr; /* pointer for running end of tb */
#ifdef CONFIG_PROFILER
TCGProfile prof;
diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c
index 9cf88da6cb..94238a1926 100644
--- a/accel/tcg/translate-all.c
+++ b/accel/tcg/translate-all.c
@@ -827,7 +827,7 @@ TranslationBlock *tb_gen_code(CPUState *cpu,
tb->trace_vcpu_dstate = *cpu->trace_dstate;
tb_set_page_addr0(tb, phys_pc);
tb_set_page_addr1(tb, -1);
- tcg_ctx->tb_cflags = cflags;
+ tcg_ctx->gen_tb = tb;
tb_overflow:
#ifdef CONFIG_PROFILER
diff --git a/tcg/tcg-op.c b/tcg/tcg-op.c
index 019fab00cc..585f33ffaf 100644
--- a/tcg/tcg-op.c
+++ b/tcg/tcg-op.c
@@ -94,7 +94,7 @@ void tcg_gen_op6(TCGOpcode opc, TCGArg a1, TCGArg a2, TCGArg a3,
void tcg_gen_mb(TCGBar mb_type)
{
- if (tcg_ctx->tb_cflags & CF_PARALLEL) {
+ if (tcg_ctx->gen_tb->cflags & CF_PARALLEL) {
tcg_gen_op1(INDEX_op_mb, mb_type);
}
}
@@ -2763,7 +2763,7 @@ void tcg_gen_exit_tb(const TranslationBlock *tb, unsigned idx)
void tcg_gen_goto_tb(unsigned idx)
{
/* We tested CF_NO_GOTO_TB in translator_use_goto_tb. */
- tcg_debug_assert(!(tcg_ctx->tb_cflags & CF_NO_GOTO_TB));
+ tcg_debug_assert(!(tcg_ctx->gen_tb->cflags & CF_NO_GOTO_TB));
/* We only support two chained exits. */
tcg_debug_assert(idx <= TB_EXIT_IDXMAX);
#ifdef CONFIG_DEBUG_TCG
@@ -2779,7 +2779,7 @@ void tcg_gen_lookup_and_goto_ptr(void)
{
TCGv_ptr ptr;
- if (tcg_ctx->tb_cflags & CF_NO_GOTO_PTR) {
+ if (tcg_ctx->gen_tb->cflags & CF_NO_GOTO_PTR) {
tcg_gen_exit_tb(NULL, 0);
return;
}
@@ -3146,7 +3146,7 @@ void tcg_gen_atomic_cmpxchg_i32(TCGv_i32 retv, TCGv addr, TCGv_i32 cmpv,
{
memop = tcg_canonicalize_memop(memop, 0, 0);
- if (!(tcg_ctx->tb_cflags & CF_PARALLEL)) {
+ if (!(tcg_ctx->gen_tb->cflags & CF_PARALLEL)) {
TCGv_i32 t1 = tcg_temp_new_i32();
TCGv_i32 t2 = tcg_temp_new_i32();
@@ -3184,7 +3184,7 @@ void tcg_gen_atomic_cmpxchg_i64(TCGv_i64 retv, TCGv addr, TCGv_i64 cmpv,
{
memop = tcg_canonicalize_memop(memop, 1, 0);
- if (!(tcg_ctx->tb_cflags & CF_PARALLEL)) {
+ if (!(tcg_ctx->gen_tb->cflags & CF_PARALLEL)) {
TCGv_i64 t1 = tcg_temp_new_i64();
TCGv_i64 t2 = tcg_temp_new_i64();
@@ -3345,7 +3345,7 @@ static void * const table_##NAME[(MO_SIZE | MO_BSWAP) + 1] = { \
void tcg_gen_atomic_##NAME##_i32 \
(TCGv_i32 ret, TCGv addr, TCGv_i32 val, TCGArg idx, MemOp memop) \
{ \
- if (tcg_ctx->tb_cflags & CF_PARALLEL) { \
+ if (tcg_ctx->gen_tb->cflags & CF_PARALLEL) { \
do_atomic_op_i32(ret, addr, val, idx, memop, table_##NAME); \
} else { \
do_nonatomic_op_i32(ret, addr, val, idx, memop, NEW, \
@@ -3355,7 +3355,7 @@ void tcg_gen_atomic_##NAME##_i32 \
void tcg_gen_atomic_##NAME##_i64 \
(TCGv_i64 ret, TCGv addr, TCGv_i64 val, TCGArg idx, MemOp memop) \
{ \
- if (tcg_ctx->tb_cflags & CF_PARALLEL) { \
+ if (tcg_ctx->gen_tb->cflags & CF_PARALLEL) { \
do_atomic_op_i64(ret, addr, val, idx, memop, table_##NAME); \
} else { \
do_nonatomic_op_i64(ret, addr, val, idx, memop, NEW, \
diff --git a/tcg/tcg.c b/tcg/tcg.c
index 37957208c7..c7ad46ff45 100644
--- a/tcg/tcg.c
+++ b/tcg/tcg.c
@@ -318,7 +318,7 @@ static void set_jmp_reset_offset(TCGContext *s, int which)
* We will check for overflow at the end of the opcode loop in
* tcg_gen_code, where we bound tcg_current_code_size to UINT16_MAX.
*/
- s->tb_jmp_reset_offset[which] = tcg_current_code_size(s);
+ s->gen_tb->jmp_reset_offset[which] = tcg_current_code_size(s);
}
static void G_GNUC_UNUSED set_jmp_insn_offset(TCGContext *s, int which)
@@ -328,7 +328,7 @@ static void G_GNUC_UNUSED set_jmp_insn_offset(TCGContext *s, int which)
* tcg_gen_code, where we bound tcg_current_code_size to UINT16_MAX.
*/
tcg_debug_assert(TCG_TARGET_HAS_direct_jump);
- s->tb_jmp_insn_offset[which] = tcg_current_code_size(s);
+ s->gen_tb->jmp_target_arg[which] = tcg_current_code_size(s);
}
static uintptr_t G_GNUC_UNUSED get_jmp_target_addr(TCGContext *s, int which)
@@ -337,7 +337,7 @@ static uintptr_t G_GNUC_UNUSED get_jmp_target_addr(TCGContext *s, int which)
* Return the read-execute version of the pointer, for the benefit
* of any pc-relative addressing mode.
*/
- return (uintptr_t)tcg_splitwx_to_rx(s->tb_jmp_target_addr + which);
+ return (uintptr_t)tcg_splitwx_to_rx(s->gen_tb->jmp_target_arg + which);
}
/* Signal overflow, starting over with fewer guest insns. */
@@ -4252,14 +4252,6 @@ int tcg_gen_code(TCGContext *s, TranslationBlock *tb, target_ulong pc_start)
/* Initialize goto_tb jump offsets. */
tb->jmp_reset_offset[0] = TB_JMP_OFFSET_INVALID;
tb->jmp_reset_offset[1] = TB_JMP_OFFSET_INVALID;
- tcg_ctx->tb_jmp_reset_offset = tb->jmp_reset_offset;
- if (TCG_TARGET_HAS_direct_jump) {
- tcg_ctx->tb_jmp_insn_offset = tb->jmp_target_arg;
- tcg_ctx->tb_jmp_target_addr = NULL;
- } else {
- tcg_ctx->tb_jmp_insn_offset = NULL;
- tcg_ctx->tb_jmp_target_addr = tb->jmp_target_arg;
- }
tcg_reg_alloc_start(s);
--
2.34.1
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH 11/22] tcg: Add TranslationBlock.jmp_insn_offset
2022-12-06 4:16 [PATCH 00/22] tcg: exit_tb tidy, goto_tb reorg Richard Henderson
` (9 preceding siblings ...)
2022-12-06 4:17 ` [PATCH 10/22] tcg: Add gen_tb to TCGContext Richard Henderson
@ 2022-12-06 4:17 ` Richard Henderson
2022-12-06 4:17 ` [PATCH 12/22] tcg: Change tb_target_set_jmp_target arguments Richard Henderson
` (10 subsequent siblings)
21 siblings, 0 replies; 35+ messages in thread
From: Richard Henderson @ 2022-12-06 4:17 UTC (permalink / raw)
To: qemu-devel; +Cc: r
Stop overloading jmp_target_arg for both offset and address,
depending on TCG_TARGET_HAS_direct_jump. Instead, add a new
field to hold the jump insn offset and always set the target
address in jmp_target_addr[]. This will allow a tcg backend
to use either direct or indirect depending on displacement.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
include/exec/exec-all.h | 3 ++-
accel/tcg/cpu-exec.c | 5 ++---
tcg/tcg.c | 6 ++++--
3 files changed, 8 insertions(+), 6 deletions(-)
diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h
index 7566ad9e7f..bb01508b4b 100644
--- a/include/exec/exec-all.h
+++ b/include/exec/exec-all.h
@@ -577,7 +577,8 @@ struct TranslationBlock {
*/
#define TB_JMP_OFFSET_INVALID 0xffff /* indicates no jump generated */
uint16_t jmp_reset_offset[2]; /* offset of original jump target */
- uintptr_t jmp_target_arg[2]; /* target address or offset */
+ uint16_t jmp_insn_offset[2]; /* offset of direct jump insn */
+ uintptr_t jmp_target_addr[2]; /* target address */
/*
* Each TB has a NULL-terminated list (jmp_list_head) of incoming jumps.
diff --git a/accel/tcg/cpu-exec.c b/accel/tcg/cpu-exec.c
index 356fe348de..a87fbf74f4 100644
--- a/accel/tcg/cpu-exec.c
+++ b/accel/tcg/cpu-exec.c
@@ -572,14 +572,13 @@ void cpu_exec_step_atomic(CPUState *cpu)
void tb_set_jmp_target(TranslationBlock *tb, int n, uintptr_t addr)
{
+ tb->jmp_target_addr[n] = addr;
if (TCG_TARGET_HAS_direct_jump) {
- uintptr_t offset = tb->jmp_target_arg[n];
+ uintptr_t offset = tb->jmp_insn_offset[n];
uintptr_t tc_ptr = (uintptr_t)tb->tc.ptr;
uintptr_t jmp_rx = tc_ptr + offset;
uintptr_t jmp_rw = jmp_rx - tcg_splitwx_diff;
tb_target_set_jmp_target(tc_ptr, jmp_rx, jmp_rw, addr);
- } else {
- tb->jmp_target_arg[n] = addr;
}
}
diff --git a/tcg/tcg.c b/tcg/tcg.c
index c7ad46ff45..c103dd0037 100644
--- a/tcg/tcg.c
+++ b/tcg/tcg.c
@@ -328,7 +328,7 @@ static void G_GNUC_UNUSED set_jmp_insn_offset(TCGContext *s, int which)
* tcg_gen_code, where we bound tcg_current_code_size to UINT16_MAX.
*/
tcg_debug_assert(TCG_TARGET_HAS_direct_jump);
- s->gen_tb->jmp_target_arg[which] = tcg_current_code_size(s);
+ s->gen_tb->jmp_insn_offset[which] = tcg_current_code_size(s);
}
static uintptr_t G_GNUC_UNUSED get_jmp_target_addr(TCGContext *s, int which)
@@ -337,7 +337,7 @@ static uintptr_t G_GNUC_UNUSED get_jmp_target_addr(TCGContext *s, int which)
* Return the read-execute version of the pointer, for the benefit
* of any pc-relative addressing mode.
*/
- return (uintptr_t)tcg_splitwx_to_rx(s->gen_tb->jmp_target_arg + which);
+ return (uintptr_t)tcg_splitwx_to_rx(&s->gen_tb->jmp_target_addr[which]);
}
/* Signal overflow, starting over with fewer guest insns. */
@@ -4252,6 +4252,8 @@ int tcg_gen_code(TCGContext *s, TranslationBlock *tb, target_ulong pc_start)
/* Initialize goto_tb jump offsets. */
tb->jmp_reset_offset[0] = TB_JMP_OFFSET_INVALID;
tb->jmp_reset_offset[1] = TB_JMP_OFFSET_INVALID;
+ tb->jmp_insn_offset[0] = TB_JMP_OFFSET_INVALID;
+ tb->jmp_insn_offset[1] = TB_JMP_OFFSET_INVALID;
tcg_reg_alloc_start(s);
--
2.34.1
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH 12/22] tcg: Change tb_target_set_jmp_target arguments
2022-12-06 4:16 [PATCH 00/22] tcg: exit_tb tidy, goto_tb reorg Richard Henderson
` (10 preceding siblings ...)
2022-12-06 4:17 ` [PATCH 11/22] tcg: Add TranslationBlock.jmp_insn_offset Richard Henderson
@ 2022-12-06 4:17 ` Richard Henderson
2022-12-06 4:17 ` [PATCH 13/22] tcg: Move tb_target_set_jmp_target declaration to tcg.h Richard Henderson
` (9 subsequent siblings)
21 siblings, 0 replies; 35+ messages in thread
From: Richard Henderson @ 2022-12-06 4:17 UTC (permalink / raw)
To: qemu-devel; +Cc: r
Replace 'tc_ptr' and 'addr' with 'tb' and 'n'.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
tcg/aarch64/tcg-target.h | 3 ++-
tcg/arm/tcg-target.h | 3 ++-
tcg/i386/tcg-target.h | 9 ++-------
tcg/loongarch64/tcg-target.h | 3 ++-
tcg/mips/tcg-target.h | 3 ++-
tcg/ppc/tcg-target.h | 3 ++-
tcg/riscv/tcg-target.h | 3 ++-
tcg/s390x/tcg-target.h | 10 ++--------
tcg/sparc64/tcg-target.h | 3 ++-
tcg/tci/tcg-target.h | 3 ++-
accel/tcg/cpu-exec.c | 6 +++---
tcg/aarch64/tcg-target.c.inc | 5 +++--
tcg/i386/tcg-target.c.inc | 9 +++++++++
tcg/loongarch64/tcg-target.c.inc | 5 +++--
tcg/ppc/tcg-target.c.inc | 7 ++++---
tcg/s390x/tcg-target.c.inc | 10 ++++++++++
tcg/sparc64/tcg-target.c.inc | 7 ++++---
17 files changed, 56 insertions(+), 36 deletions(-)
diff --git a/tcg/aarch64/tcg-target.h b/tcg/aarch64/tcg-target.h
index 485f685bd2..9b0927012c 100644
--- a/tcg/aarch64/tcg-target.h
+++ b/tcg/aarch64/tcg-target.h
@@ -152,7 +152,8 @@ typedef enum {
#define TCG_TARGET_DEFAULT_MO (0)
#define TCG_TARGET_HAS_MEMORY_BSWAP 0
-void tb_target_set_jmp_target(uintptr_t, uintptr_t, uintptr_t, uintptr_t);
+void tb_target_set_jmp_target(const TranslationBlock *, int,
+ uintptr_t, uintptr_t);
#define TCG_TARGET_NEED_LDST_LABELS
#define TCG_TARGET_NEED_POOL_LABELS
diff --git a/tcg/arm/tcg-target.h b/tcg/arm/tcg-target.h
index 7e96495392..743a725aa7 100644
--- a/tcg/arm/tcg-target.h
+++ b/tcg/arm/tcg-target.h
@@ -152,7 +152,8 @@ extern bool use_neon_instructions;
#define TCG_TARGET_HAS_MEMORY_BSWAP 0
/* not defined -- call should be eliminated at compile time */
-void tb_target_set_jmp_target(uintptr_t, uintptr_t, uintptr_t, uintptr_t);
+void tb_target_set_jmp_target(const TranslationBlock *tb, int n,
+ uintptr_t, uintptr_t);
#define TCG_TARGET_NEED_LDST_LABELS
#define TCG_TARGET_NEED_POOL_LABELS
diff --git a/tcg/i386/tcg-target.h b/tcg/i386/tcg-target.h
index 00fcbe297d..b64317bf40 100644
--- a/tcg/i386/tcg-target.h
+++ b/tcg/i386/tcg-target.h
@@ -218,13 +218,8 @@ extern bool have_movbe;
#define TCG_TARGET_extract_i64_valid(ofs, len) \
(((ofs) == 8 && (len) == 8) || ((ofs) + (len)) == 32)
-static inline void tb_target_set_jmp_target(uintptr_t tc_ptr, uintptr_t jmp_rx,
- uintptr_t jmp_rw, uintptr_t addr)
-{
- /* patch the branch destination */
- qatomic_set((int32_t *)jmp_rw, addr - (jmp_rx + 4));
- /* no need to flush icache explicitly */
-}
+void tb_target_set_jmp_target(const TranslationBlock *, int,
+ uintptr_t, uintptr_t);
/* This defines the natural memory order supported by this
* architecture before guarantees made by various barrier
diff --git a/tcg/loongarch64/tcg-target.h b/tcg/loongarch64/tcg-target.h
index a659c8d6fd..6a6c8d6941 100644
--- a/tcg/loongarch64/tcg-target.h
+++ b/tcg/loongarch64/tcg-target.h
@@ -170,7 +170,8 @@ typedef enum {
#define TCG_TARGET_HAS_muluh_i64 1
#define TCG_TARGET_HAS_mulsh_i64 1
-void tb_target_set_jmp_target(uintptr_t, uintptr_t, uintptr_t, uintptr_t);
+void tb_target_set_jmp_target(const TranslationBlock *tb, int n,
+ uintptr_t, uintptr_t);
#define TCG_TARGET_DEFAULT_MO (0)
diff --git a/tcg/mips/tcg-target.h b/tcg/mips/tcg-target.h
index 7669213175..57154ec808 100644
--- a/tcg/mips/tcg-target.h
+++ b/tcg/mips/tcg-target.h
@@ -204,7 +204,8 @@ extern bool use_mips32r2_instructions;
#define TCG_TARGET_HAS_MEMORY_BSWAP 1
/* not defined -- call should be eliminated at compile time */
-void tb_target_set_jmp_target(uintptr_t, uintptr_t, uintptr_t, uintptr_t)
+void tb_target_set_jmp_target(const TranslationBlock *tb, int n,
+ uintptr_t, uintptr_t)
QEMU_ERROR("code path is reachable");
#define TCG_TARGET_NEED_LDST_LABELS
diff --git a/tcg/ppc/tcg-target.h b/tcg/ppc/tcg-target.h
index b5cd225cfa..02764c3331 100644
--- a/tcg/ppc/tcg-target.h
+++ b/tcg/ppc/tcg-target.h
@@ -180,7 +180,8 @@ extern bool have_vsx;
#define TCG_TARGET_HAS_bitsel_vec have_vsx
#define TCG_TARGET_HAS_cmpsel_vec 0
-void tb_target_set_jmp_target(uintptr_t, uintptr_t, uintptr_t, uintptr_t);
+void tb_target_set_jmp_target(const TranslationBlock *tb, int n,
+ uintptr_t, uintptr_t);
#define TCG_TARGET_DEFAULT_MO (0)
#define TCG_TARGET_HAS_MEMORY_BSWAP 1
diff --git a/tcg/riscv/tcg-target.h b/tcg/riscv/tcg-target.h
index 11c9b3e4f4..21d455a081 100644
--- a/tcg/riscv/tcg-target.h
+++ b/tcg/riscv/tcg-target.h
@@ -161,7 +161,8 @@ typedef enum {
#endif
/* not defined -- call should be eliminated at compile time */
-void tb_target_set_jmp_target(uintptr_t, uintptr_t, uintptr_t, uintptr_t);
+void tb_target_set_jmp_target(const TranslationBlock *tb, int n,
+ uintptr_t, uintptr_t);
#define TCG_TARGET_DEFAULT_MO (0)
diff --git a/tcg/s390x/tcg-target.h b/tcg/s390x/tcg-target.h
index 23e2063667..d9a45e20a8 100644
--- a/tcg/s390x/tcg-target.h
+++ b/tcg/s390x/tcg-target.h
@@ -172,14 +172,8 @@ extern uint64_t s390_facilities[3];
#define TCG_TARGET_DEFAULT_MO (TCG_MO_ALL & ~TCG_MO_ST_LD)
-static inline void tb_target_set_jmp_target(uintptr_t tc_ptr, uintptr_t jmp_rx,
- uintptr_t jmp_rw, uintptr_t addr)
-{
- /* patch the branch destination */
- intptr_t disp = addr - (jmp_rx - 2);
- qatomic_set((int32_t *)jmp_rw, disp / 2);
- /* no need to flush icache explicitly */
-}
+void tb_target_set_jmp_target(const TranslationBlock *tb, int n,
+ uintptr_t jmp_rx, uintptr_t jmp_rw);
#define TCG_TARGET_NEED_LDST_LABELS
#define TCG_TARGET_NEED_POOL_LABELS
diff --git a/tcg/sparc64/tcg-target.h b/tcg/sparc64/tcg-target.h
index 8655acdbe5..f70fa9e350 100644
--- a/tcg/sparc64/tcg-target.h
+++ b/tcg/sparc64/tcg-target.h
@@ -154,7 +154,8 @@ extern bool use_vis3_instructions;
#define TCG_TARGET_DEFAULT_MO (0)
#define TCG_TARGET_HAS_MEMORY_BSWAP 1
-void tb_target_set_jmp_target(uintptr_t, uintptr_t, uintptr_t, uintptr_t);
+void tb_target_set_jmp_target(const TranslationBlock *tb, int n,
+ uintptr_t, uintptr_t);
#define TCG_TARGET_NEED_POOL_LABELS
diff --git a/tcg/tci/tcg-target.h b/tcg/tci/tcg-target.h
index ceb36c4f7a..5d1a77d078 100644
--- a/tcg/tci/tcg-target.h
+++ b/tcg/tci/tcg-target.h
@@ -170,6 +170,7 @@ typedef enum {
#define TCG_TARGET_HAS_MEMORY_BSWAP 1
/* not defined -- call should be eliminated at compile time */
-void tb_target_set_jmp_target(uintptr_t, uintptr_t, uintptr_t, uintptr_t);
+void tb_target_set_jmp_target(const TranslationBlock *tb, int n,
+ uintptr_t, uintptr_t);
#endif /* TCG_TARGET_H */
diff --git a/accel/tcg/cpu-exec.c b/accel/tcg/cpu-exec.c
index a87fbf74f4..ac5b581e52 100644
--- a/accel/tcg/cpu-exec.c
+++ b/accel/tcg/cpu-exec.c
@@ -574,11 +574,11 @@ void tb_set_jmp_target(TranslationBlock *tb, int n, uintptr_t addr)
{
tb->jmp_target_addr[n] = addr;
if (TCG_TARGET_HAS_direct_jump) {
+ const TranslationBlock *c_tb = tcg_splitwx_to_rx(tb);
uintptr_t offset = tb->jmp_insn_offset[n];
- uintptr_t tc_ptr = (uintptr_t)tb->tc.ptr;
- uintptr_t jmp_rx = tc_ptr + offset;
+ uintptr_t jmp_rx = (uintptr_t)tb->tc.ptr + offset;
uintptr_t jmp_rw = jmp_rx - tcg_splitwx_diff;
- tb_target_set_jmp_target(tc_ptr, jmp_rx, jmp_rw, addr);
+ tb_target_set_jmp_target(c_tb, n, jmp_rx, jmp_rw);
}
}
diff --git a/tcg/aarch64/tcg-target.c.inc b/tcg/aarch64/tcg-target.c.inc
index 072d32f355..af30f9db69 100644
--- a/tcg/aarch64/tcg-target.c.inc
+++ b/tcg/aarch64/tcg-target.c.inc
@@ -1352,9 +1352,10 @@ static void tcg_out_call(TCGContext *s, const tcg_insn_unit *target)
}
}
-void tb_target_set_jmp_target(uintptr_t tc_ptr, uintptr_t jmp_rx,
- uintptr_t jmp_rw, uintptr_t addr)
+void tb_target_set_jmp_target(const TranslationBlock *tb, int n,
+ uintptr_t jmp_rx, uintptr_t jmp_rw)
{
+ uintptr_t addr = tb->jmp_target_addr[n];
tcg_insn_unit i1, i2;
TCGType rt = TCG_TYPE_I64;
TCGReg rd = TCG_REG_TMP;
diff --git a/tcg/i386/tcg-target.c.inc b/tcg/i386/tcg-target.c.inc
index d3cef242ef..6650e2eda3 100644
--- a/tcg/i386/tcg-target.c.inc
+++ b/tcg/i386/tcg-target.c.inc
@@ -2373,6 +2373,15 @@ static void tcg_out_goto_tb(TCGContext *s, int which)
set_jmp_reset_offset(s, which);
}
+void tb_target_set_jmp_target(const TranslationBlock *tb, int n,
+ uintptr_t jmp_rx, uintptr_t jmp_rw)
+{
+ /* patch the branch destination */
+ uintptr_t addr = tb->jmp_target_addr[n];
+ qatomic_set((int32_t *)jmp_rw, addr - (jmp_rx + 4));
+ /* no need to flush icache explicitly */
+}
+
static inline void tcg_out_op(TCGContext *s, TCGOpcode opc,
const TCGArg args[TCG_MAX_OP_ARGS],
const int const_args[TCG_MAX_OP_ARGS])
diff --git a/tcg/loongarch64/tcg-target.c.inc b/tcg/loongarch64/tcg-target.c.inc
index dbe64b7d8b..c05b19a084 100644
--- a/tcg/loongarch64/tcg-target.c.inc
+++ b/tcg/loongarch64/tcg-target.c.inc
@@ -1038,11 +1038,12 @@ static void tcg_out_nop(TCGContext *s)
tcg_out32(s, NOP);
}
-void tb_target_set_jmp_target(uintptr_t tc_ptr, uintptr_t jmp_rx,
- uintptr_t jmp_rw, uintptr_t addr)
+void tb_target_set_jmp_target(const TranslationBlock *tb, int n,
+ uintptr_t jmp_rx, uintptr_t jmp_rw)
{
tcg_insn_unit i1, i2;
ptrdiff_t upper, lower;
+ uintptr_t addr = tb->jmp_target_addr[n];
ptrdiff_t offset = (ptrdiff_t)(addr - jmp_rx) >> 2;
if (offset == sextreg(offset, 0, 26)) {
diff --git a/tcg/ppc/tcg-target.c.inc b/tcg/ppc/tcg-target.c.inc
index ec13ea3171..592b8d6498 100644
--- a/tcg/ppc/tcg-target.c.inc
+++ b/tcg/ppc/tcg-target.c.inc
@@ -1886,11 +1886,12 @@ static inline void ppc64_replace4(uintptr_t rx, uintptr_t rw,
flush_idcache_range(rx, rw, 16);
}
-void tb_target_set_jmp_target(uintptr_t tc_ptr, uintptr_t jmp_rx,
- uintptr_t jmp_rw, uintptr_t addr)
+void tb_target_set_jmp_target(const TranslationBlock *tb, int n,
+ uintptr_t jmp_rx, uintptr_t jmp_rw)
{
tcg_insn_unit i0, i1, i2, i3;
- intptr_t tb_diff = addr - tc_ptr;
+ uintptr_t addr = tb->jmp_target_addr[n];
+ intptr_t tb_diff = addr - (uintptr_t)tb->tc.ptr;
intptr_t br_diff = addr - (jmp_rx + 4);
intptr_t lo, hi;
diff --git a/tcg/s390x/tcg-target.c.inc b/tcg/s390x/tcg-target.c.inc
index a60edc18cf..82435ecadb 100644
--- a/tcg/s390x/tcg-target.c.inc
+++ b/tcg/s390x/tcg-target.c.inc
@@ -2116,6 +2116,16 @@ static void tcg_out_goto_tb(TCGContext *s, int which)
}
}
+void tb_target_set_jmp_target(const TranslationBlock *tb, int n,
+ uintptr_t jmp_rx, uintptr_t jmp_rw)
+{
+ /* patch the branch destination */
+ uintptr_t addr = tb->jmp_target_addr[n];
+ intptr_t disp = addr - (jmp_rx - 2);
+ qatomic_set((int32_t *)jmp_rw, disp / 2);
+ /* no need to flush icache explicitly */
+}
+
# define OP_32_64(x) \
case glue(glue(INDEX_op_,x),_i32): \
case glue(glue(INDEX_op_,x),_i64)
diff --git a/tcg/sparc64/tcg-target.c.inc b/tcg/sparc64/tcg-target.c.inc
index ed8efa413c..96d58f30b1 100644
--- a/tcg/sparc64/tcg-target.c.inc
+++ b/tcg/sparc64/tcg-target.c.inc
@@ -1884,10 +1884,11 @@ void tcg_register_jit(const void *buf, size_t buf_size)
tcg_register_jit_int(buf, buf_size, &debug_frame, sizeof(debug_frame));
}
-void tb_target_set_jmp_target(uintptr_t tc_ptr, uintptr_t jmp_rx,
- uintptr_t jmp_rw, uintptr_t addr)
+void tb_target_set_jmp_target(const TranslationBlock *tb, int n,
+ uintptr_t jmp_rx, uintptr_t jmp_rw)
{
- intptr_t tb_disp = addr - tc_ptr;
+ uintptr_t addr = tb->jmp_target_addr[n];
+ intptr_t tb_disp = addr - (uintptr_t)tb->tc.ptr;
intptr_t br_disp = addr - jmp_rx;
tcg_insn_unit i1, i2;
--
2.34.1
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH 13/22] tcg: Move tb_target_set_jmp_target declaration to tcg.h
2022-12-06 4:16 [PATCH 00/22] tcg: exit_tb tidy, goto_tb reorg Richard Henderson
` (11 preceding siblings ...)
2022-12-06 4:17 ` [PATCH 12/22] tcg: Change tb_target_set_jmp_target arguments Richard Henderson
@ 2022-12-06 4:17 ` Richard Henderson
2022-12-06 7:38 ` Philippe Mathieu-Daudé
2022-12-06 4:17 ` [PATCH 14/22] tcg: Always define tb_target_set_jmp_target Richard Henderson
` (8 subsequent siblings)
21 siblings, 1 reply; 35+ messages in thread
From: Richard Henderson @ 2022-12-06 4:17 UTC (permalink / raw)
To: qemu-devel; +Cc: r
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
include/tcg/tcg.h | 3 +++
tcg/aarch64/tcg-target.h | 4 ----
tcg/arm/tcg-target.h | 5 -----
tcg/i386/tcg-target.h | 3 ---
tcg/loongarch64/tcg-target.h | 3 ---
tcg/mips/tcg-target.h | 5 -----
tcg/ppc/tcg-target.h | 4 ----
tcg/riscv/tcg-target.h | 4 ----
tcg/s390x/tcg-target.h | 4 ----
tcg/sparc64/tcg-target.h | 4 ----
tcg/tci/tcg-target.h | 4 ----
11 files changed, 3 insertions(+), 40 deletions(-)
diff --git a/include/tcg/tcg.h b/include/tcg/tcg.h
index 993aafa1a2..6f3b602564 100644
--- a/include/tcg/tcg.h
+++ b/include/tcg/tcg.h
@@ -837,6 +837,9 @@ void tcg_func_start(TCGContext *s);
int tcg_gen_code(TCGContext *s, TranslationBlock *tb, target_ulong pc_start);
+void tb_target_set_jmp_target(const TranslationBlock *, int,
+ uintptr_t, uintptr_t);
+
void tcg_set_frame(TCGContext *s, TCGReg reg, intptr_t start, intptr_t size);
TCGTemp *tcg_global_mem_new_internal(TCGType, TCGv_ptr,
diff --git a/tcg/aarch64/tcg-target.h b/tcg/aarch64/tcg-target.h
index 9b0927012c..dc16fd0da6 100644
--- a/tcg/aarch64/tcg-target.h
+++ b/tcg/aarch64/tcg-target.h
@@ -151,10 +151,6 @@ typedef enum {
#define TCG_TARGET_DEFAULT_MO (0)
#define TCG_TARGET_HAS_MEMORY_BSWAP 0
-
-void tb_target_set_jmp_target(const TranslationBlock *, int,
- uintptr_t, uintptr_t);
-
#define TCG_TARGET_NEED_LDST_LABELS
#define TCG_TARGET_NEED_POOL_LABELS
diff --git a/tcg/arm/tcg-target.h b/tcg/arm/tcg-target.h
index 743a725aa7..13ad721438 100644
--- a/tcg/arm/tcg-target.h
+++ b/tcg/arm/tcg-target.h
@@ -150,11 +150,6 @@ extern bool use_neon_instructions;
#define TCG_TARGET_DEFAULT_MO (0)
#define TCG_TARGET_HAS_MEMORY_BSWAP 0
-
-/* not defined -- call should be eliminated at compile time */
-void tb_target_set_jmp_target(const TranslationBlock *tb, int n,
- uintptr_t, uintptr_t);
-
#define TCG_TARGET_NEED_LDST_LABELS
#define TCG_TARGET_NEED_POOL_LABELS
diff --git a/tcg/i386/tcg-target.h b/tcg/i386/tcg-target.h
index b64317bf40..4b4ceacfa5 100644
--- a/tcg/i386/tcg-target.h
+++ b/tcg/i386/tcg-target.h
@@ -218,9 +218,6 @@ extern bool have_movbe;
#define TCG_TARGET_extract_i64_valid(ofs, len) \
(((ofs) == 8 && (len) == 8) || ((ofs) + (len)) == 32)
-void tb_target_set_jmp_target(const TranslationBlock *, int,
- uintptr_t, uintptr_t);
-
/* This defines the natural memory order supported by this
* architecture before guarantees made by various barrier
* instructions.
diff --git a/tcg/loongarch64/tcg-target.h b/tcg/loongarch64/tcg-target.h
index 6a6c8d6941..87f40d935c 100644
--- a/tcg/loongarch64/tcg-target.h
+++ b/tcg/loongarch64/tcg-target.h
@@ -170,9 +170,6 @@ typedef enum {
#define TCG_TARGET_HAS_muluh_i64 1
#define TCG_TARGET_HAS_mulsh_i64 1
-void tb_target_set_jmp_target(const TranslationBlock *tb, int n,
- uintptr_t, uintptr_t);
-
#define TCG_TARGET_DEFAULT_MO (0)
#define TCG_TARGET_NEED_LDST_LABELS
diff --git a/tcg/mips/tcg-target.h b/tcg/mips/tcg-target.h
index 57154ec808..31236d8e81 100644
--- a/tcg/mips/tcg-target.h
+++ b/tcg/mips/tcg-target.h
@@ -203,11 +203,6 @@ extern bool use_mips32r2_instructions;
#define TCG_TARGET_DEFAULT_MO (0)
#define TCG_TARGET_HAS_MEMORY_BSWAP 1
-/* not defined -- call should be eliminated at compile time */
-void tb_target_set_jmp_target(const TranslationBlock *tb, int n,
- uintptr_t, uintptr_t)
- QEMU_ERROR("code path is reachable");
-
#define TCG_TARGET_NEED_LDST_LABELS
#endif
diff --git a/tcg/ppc/tcg-target.h b/tcg/ppc/tcg-target.h
index 02764c3331..5ffb41fb57 100644
--- a/tcg/ppc/tcg-target.h
+++ b/tcg/ppc/tcg-target.h
@@ -180,12 +180,8 @@ extern bool have_vsx;
#define TCG_TARGET_HAS_bitsel_vec have_vsx
#define TCG_TARGET_HAS_cmpsel_vec 0
-void tb_target_set_jmp_target(const TranslationBlock *tb, int n,
- uintptr_t, uintptr_t);
-
#define TCG_TARGET_DEFAULT_MO (0)
#define TCG_TARGET_HAS_MEMORY_BSWAP 1
-
#define TCG_TARGET_NEED_LDST_LABELS
#define TCG_TARGET_NEED_POOL_LABELS
diff --git a/tcg/riscv/tcg-target.h b/tcg/riscv/tcg-target.h
index 21d455a081..ff2f861e82 100644
--- a/tcg/riscv/tcg-target.h
+++ b/tcg/riscv/tcg-target.h
@@ -160,10 +160,6 @@ typedef enum {
#define TCG_TARGET_HAS_mulsh_i64 1
#endif
-/* not defined -- call should be eliminated at compile time */
-void tb_target_set_jmp_target(const TranslationBlock *tb, int n,
- uintptr_t, uintptr_t);
-
#define TCG_TARGET_DEFAULT_MO (0)
#define TCG_TARGET_NEED_LDST_LABELS
diff --git a/tcg/s390x/tcg-target.h b/tcg/s390x/tcg-target.h
index d9a45e20a8..274cb3cc5d 100644
--- a/tcg/s390x/tcg-target.h
+++ b/tcg/s390x/tcg-target.h
@@ -171,10 +171,6 @@ extern uint64_t s390_facilities[3];
#define TCG_TARGET_HAS_MEMORY_BSWAP 1
#define TCG_TARGET_DEFAULT_MO (TCG_MO_ALL & ~TCG_MO_ST_LD)
-
-void tb_target_set_jmp_target(const TranslationBlock *tb, int n,
- uintptr_t jmp_rx, uintptr_t jmp_rw);
-
#define TCG_TARGET_NEED_LDST_LABELS
#define TCG_TARGET_NEED_POOL_LABELS
diff --git a/tcg/sparc64/tcg-target.h b/tcg/sparc64/tcg-target.h
index f70fa9e350..ca7e4da6d3 100644
--- a/tcg/sparc64/tcg-target.h
+++ b/tcg/sparc64/tcg-target.h
@@ -153,10 +153,6 @@ extern bool use_vis3_instructions;
#define TCG_TARGET_DEFAULT_MO (0)
#define TCG_TARGET_HAS_MEMORY_BSWAP 1
-
-void tb_target_set_jmp_target(const TranslationBlock *tb, int n,
- uintptr_t, uintptr_t);
-
#define TCG_TARGET_NEED_POOL_LABELS
#endif
diff --git a/tcg/tci/tcg-target.h b/tcg/tci/tcg-target.h
index 5d1a77d078..4b4b26a89f 100644
--- a/tcg/tci/tcg-target.h
+++ b/tcg/tci/tcg-target.h
@@ -169,8 +169,4 @@ typedef enum {
#define TCG_TARGET_HAS_MEMORY_BSWAP 1
-/* not defined -- call should be eliminated at compile time */
-void tb_target_set_jmp_target(const TranslationBlock *tb, int n,
- uintptr_t, uintptr_t);
-
#endif /* TCG_TARGET_H */
--
2.34.1
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH 14/22] tcg: Always define tb_target_set_jmp_target
2022-12-06 4:16 [PATCH 00/22] tcg: exit_tb tidy, goto_tb reorg Richard Henderson
` (12 preceding siblings ...)
2022-12-06 4:17 ` [PATCH 13/22] tcg: Move tb_target_set_jmp_target declaration to tcg.h Richard Henderson
@ 2022-12-06 4:17 ` Richard Henderson
2022-12-06 7:39 ` Philippe Mathieu-Daudé
2022-12-06 4:17 ` [PATCH 15/22] tcg: Remove TCG_TARGET_HAS_direct_jump Richard Henderson
` (7 subsequent siblings)
21 siblings, 1 reply; 35+ messages in thread
From: Richard Henderson @ 2022-12-06 4:17 UTC (permalink / raw)
To: qemu-devel; +Cc: r
Install empty versions for !TCG_TARGET_HAS_direct_jump hosts.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
tcg/arm/tcg-target.c.inc | 6 ++++++
tcg/mips/tcg-target.c.inc | 6 ++++++
tcg/riscv/tcg-target.c.inc | 6 ++++++
tcg/tci/tcg-target.c.inc | 6 ++++++
4 files changed, 24 insertions(+)
diff --git a/tcg/arm/tcg-target.c.inc b/tcg/arm/tcg-target.c.inc
index 0afc286c8e..f5103dcc6d 100644
--- a/tcg/arm/tcg-target.c.inc
+++ b/tcg/arm/tcg-target.c.inc
@@ -1958,6 +1958,12 @@ static void tcg_out_goto_tb(TCGContext *s, int which)
set_jmp_reset_offset(s, which);
}
+void tb_target_set_jmp_target(const TranslationBlock *tb, int n,
+ uintptr_t jmp_rx, uintptr_t jmp_rw)
+{
+ /* Always indirect, nothing to do */
+}
+
static void tcg_out_op(TCGContext *s, TCGOpcode opc,
const TCGArg args[TCG_MAX_OP_ARGS],
const int const_args[TCG_MAX_OP_ARGS])
diff --git a/tcg/mips/tcg-target.c.inc b/tcg/mips/tcg-target.c.inc
index 605aa7714a..dfb31a17bb 100644
--- a/tcg/mips/tcg-target.c.inc
+++ b/tcg/mips/tcg-target.c.inc
@@ -1976,6 +1976,12 @@ static void tcg_out_goto_tb(TCGContext *s, int which)
set_jmp_reset_offset(s, which);
}
+void tb_target_set_jmp_target(const TranslationBlock *tb, int n,
+ uintptr_t jmp_rx, uintptr_t jmp_rw)
+{
+ /* Always indirect, nothing to do */
+}
+
static void tcg_out_op(TCGContext *s, TCGOpcode opc,
const TCGArg args[TCG_MAX_OP_ARGS],
const int const_args[TCG_MAX_OP_ARGS])
diff --git a/tcg/riscv/tcg-target.c.inc b/tcg/riscv/tcg-target.c.inc
index e73dfb4cbb..06308c5243 100644
--- a/tcg/riscv/tcg-target.c.inc
+++ b/tcg/riscv/tcg-target.c.inc
@@ -1285,6 +1285,12 @@ static void tcg_out_goto_tb(TCGContext *s, int which)
set_jmp_reset_offset(s, which);
}
+void tb_target_set_jmp_target(const TranslationBlock *tb, int n,
+ uintptr_t jmp_rx, uintptr_t jmp_rw)
+{
+ /* Always indirect, nothing to do */
+}
+
static void tcg_out_op(TCGContext *s, TCGOpcode opc,
const TCGArg args[TCG_MAX_OP_ARGS],
const int const_args[TCG_MAX_OP_ARGS])
diff --git a/tcg/tci/tcg-target.c.inc b/tcg/tci/tcg-target.c.inc
index cf84ff3133..31094067b7 100644
--- a/tcg/tci/tcg-target.c.inc
+++ b/tcg/tci/tcg-target.c.inc
@@ -606,6 +606,12 @@ static void tcg_out_goto_tb(TCGContext *s, int which)
set_jmp_reset_offset(s, which);
}
+void tb_target_set_jmp_target(const TranslationBlock *tb, int n,
+ uintptr_t jmp_rx, uintptr_t jmp_rw)
+{
+ /* Always indirect, nothing to do */
+}
+
static void tcg_out_op(TCGContext *s, TCGOpcode opc,
const TCGArg args[TCG_MAX_OP_ARGS],
const int const_args[TCG_MAX_OP_ARGS])
--
2.34.1
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH 15/22] tcg: Remove TCG_TARGET_HAS_direct_jump
2022-12-06 4:16 [PATCH 00/22] tcg: exit_tb tidy, goto_tb reorg Richard Henderson
` (13 preceding siblings ...)
2022-12-06 4:17 ` [PATCH 14/22] tcg: Always define tb_target_set_jmp_target Richard Henderson
@ 2022-12-06 4:17 ` Richard Henderson
2022-12-06 4:17 ` [PATCH 16/22] tcg/aarch64: Reorg goto_tb implementation Richard Henderson
` (6 subsequent siblings)
21 siblings, 0 replies; 35+ messages in thread
From: Richard Henderson @ 2022-12-06 4:17 UTC (permalink / raw)
To: qemu-devel; +Cc: r
We now have the option to generate direct or indirect
goto_tb depending on the dynamic displacement, thus
the define is no longer necessary or completely accurate.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
tcg/aarch64/tcg-target.h | 1 -
tcg/arm/tcg-target.h | 1 -
tcg/i386/tcg-target.h | 1 -
tcg/loongarch64/tcg-target.h | 1 -
tcg/mips/tcg-target.h | 1 -
tcg/ppc/tcg-target.h | 1 -
tcg/riscv/tcg-target.h | 1 -
tcg/s390x/tcg-target.h | 1 -
tcg/sparc64/tcg-target.h | 1 -
tcg/tci/tcg-target.h | 1 -
accel/tcg/cpu-exec.c | 13 ++++++-------
tcg/tcg.c | 1 -
tcg/arm/tcg-target.c.inc | 1 -
tcg/mips/tcg-target.c.inc | 1 -
tcg/riscv/tcg-target.c.inc | 1 -
tcg/s390x/tcg-target.c.inc | 5 ++++-
tcg/tci/tcg-target.c.inc | 1 -
17 files changed, 10 insertions(+), 23 deletions(-)
diff --git a/tcg/aarch64/tcg-target.h b/tcg/aarch64/tcg-target.h
index dc16fd0da6..c8202e4bc5 100644
--- a/tcg/aarch64/tcg-target.h
+++ b/tcg/aarch64/tcg-target.h
@@ -123,7 +123,6 @@ typedef enum {
#define TCG_TARGET_HAS_muls2_i64 0
#define TCG_TARGET_HAS_muluh_i64 1
#define TCG_TARGET_HAS_mulsh_i64 1
-#define TCG_TARGET_HAS_direct_jump 1
#define TCG_TARGET_HAS_v64 1
#define TCG_TARGET_HAS_v128 1
diff --git a/tcg/arm/tcg-target.h b/tcg/arm/tcg-target.h
index 13ad721438..879eecd93d 100644
--- a/tcg/arm/tcg-target.h
+++ b/tcg/arm/tcg-target.h
@@ -121,7 +121,6 @@ extern bool use_neon_instructions;
#define TCG_TARGET_HAS_mulsh_i32 0
#define TCG_TARGET_HAS_div_i32 use_idiv_instructions
#define TCG_TARGET_HAS_rem_i32 0
-#define TCG_TARGET_HAS_direct_jump 0
#define TCG_TARGET_HAS_qemu_st8_i32 0
#define TCG_TARGET_HAS_v64 use_neon_instructions
diff --git a/tcg/i386/tcg-target.h b/tcg/i386/tcg-target.h
index 4b4ceacfa5..b1ffd47493 100644
--- a/tcg/i386/tcg-target.h
+++ b/tcg/i386/tcg-target.h
@@ -139,7 +139,6 @@ extern bool have_movbe;
#define TCG_TARGET_HAS_muls2_i32 1
#define TCG_TARGET_HAS_muluh_i32 0
#define TCG_TARGET_HAS_mulsh_i32 0
-#define TCG_TARGET_HAS_direct_jump 1
#if TCG_TARGET_REG_BITS == 64
/* Keep target addresses zero-extended in a register. */
diff --git a/tcg/loongarch64/tcg-target.h b/tcg/loongarch64/tcg-target.h
index 87f40d935c..086c90bda3 100644
--- a/tcg/loongarch64/tcg-target.h
+++ b/tcg/loongarch64/tcg-target.h
@@ -127,7 +127,6 @@ typedef enum {
#define TCG_TARGET_HAS_clz_i32 1
#define TCG_TARGET_HAS_ctz_i32 1
#define TCG_TARGET_HAS_ctpop_i32 0
-#define TCG_TARGET_HAS_direct_jump 1
#define TCG_TARGET_HAS_brcond2 0
#define TCG_TARGET_HAS_setcond2 0
#define TCG_TARGET_HAS_qemu_st8_i32 0
diff --git a/tcg/mips/tcg-target.h b/tcg/mips/tcg-target.h
index 31236d8e81..cc0a6f301a 100644
--- a/tcg/mips/tcg-target.h
+++ b/tcg/mips/tcg-target.h
@@ -132,7 +132,6 @@ extern bool use_mips32r2_instructions;
#define TCG_TARGET_HAS_muluh_i32 1
#define TCG_TARGET_HAS_mulsh_i32 1
#define TCG_TARGET_HAS_bswap32_i32 1
-#define TCG_TARGET_HAS_direct_jump 0
#if TCG_TARGET_REG_BITS == 64
#define TCG_TARGET_HAS_add2_i32 0
diff --git a/tcg/ppc/tcg-target.h b/tcg/ppc/tcg-target.h
index 5ffb41fb57..f253184915 100644
--- a/tcg/ppc/tcg-target.h
+++ b/tcg/ppc/tcg-target.h
@@ -108,7 +108,6 @@ extern bool have_vsx;
#define TCG_TARGET_HAS_muls2_i32 0
#define TCG_TARGET_HAS_muluh_i32 1
#define TCG_TARGET_HAS_mulsh_i32 1
-#define TCG_TARGET_HAS_direct_jump 1
#define TCG_TARGET_HAS_qemu_st8_i32 0
#if TCG_TARGET_REG_BITS == 64
diff --git a/tcg/riscv/tcg-target.h b/tcg/riscv/tcg-target.h
index ff2f861e82..56f7bc3346 100644
--- a/tcg/riscv/tcg-target.h
+++ b/tcg/riscv/tcg-target.h
@@ -116,7 +116,6 @@ typedef enum {
#define TCG_TARGET_HAS_clz_i32 0
#define TCG_TARGET_HAS_ctz_i32 0
#define TCG_TARGET_HAS_ctpop_i32 0
-#define TCG_TARGET_HAS_direct_jump 0
#define TCG_TARGET_HAS_brcond2 1
#define TCG_TARGET_HAS_setcond2 1
#define TCG_TARGET_HAS_qemu_st8_i32 0
diff --git a/tcg/s390x/tcg-target.h b/tcg/s390x/tcg-target.h
index 274cb3cc5d..2c7c14055b 100644
--- a/tcg/s390x/tcg-target.h
+++ b/tcg/s390x/tcg-target.h
@@ -103,7 +103,6 @@ extern uint64_t s390_facilities[3];
#define TCG_TARGET_HAS_mulsh_i32 0
#define TCG_TARGET_HAS_extrl_i64_i32 0
#define TCG_TARGET_HAS_extrh_i64_i32 0
-#define TCG_TARGET_HAS_direct_jump HAVE_FACILITY(GEN_INST_EXT)
#define TCG_TARGET_HAS_qemu_st8_i32 0
#define TCG_TARGET_HAS_div2_i64 1
diff --git a/tcg/sparc64/tcg-target.h b/tcg/sparc64/tcg-target.h
index ca7e4da6d3..21d3c59bd7 100644
--- a/tcg/sparc64/tcg-target.h
+++ b/tcg/sparc64/tcg-target.h
@@ -110,7 +110,6 @@ extern bool use_vis3_instructions;
#define TCG_TARGET_HAS_muls2_i32 1
#define TCG_TARGET_HAS_muluh_i32 0
#define TCG_TARGET_HAS_mulsh_i32 0
-#define TCG_TARGET_HAS_direct_jump 1
#define TCG_TARGET_HAS_qemu_st8_i32 0
#define TCG_TARGET_HAS_extrl_i64_i32 1
diff --git a/tcg/tci/tcg-target.h b/tcg/tci/tcg-target.h
index 4b4b26a89f..c9d2fe5b94 100644
--- a/tcg/tci/tcg-target.h
+++ b/tcg/tci/tcg-target.h
@@ -82,7 +82,6 @@
#define TCG_TARGET_HAS_muls2_i32 1
#define TCG_TARGET_HAS_muluh_i32 0
#define TCG_TARGET_HAS_mulsh_i32 0
-#define TCG_TARGET_HAS_direct_jump 0
#define TCG_TARGET_HAS_qemu_st8_i32 0
#if TCG_TARGET_REG_BITS == 64
diff --git a/accel/tcg/cpu-exec.c b/accel/tcg/cpu-exec.c
index ac5b581e52..0892c6534a 100644
--- a/accel/tcg/cpu-exec.c
+++ b/accel/tcg/cpu-exec.c
@@ -572,14 +572,13 @@ void cpu_exec_step_atomic(CPUState *cpu)
void tb_set_jmp_target(TranslationBlock *tb, int n, uintptr_t addr)
{
+ const TranslationBlock *c_tb = tcg_splitwx_to_rx(tb);
+ uintptr_t offset = tb->jmp_insn_offset[n];
+ uintptr_t jmp_rx = (uintptr_t)tb->tc.ptr + offset;
+ uintptr_t jmp_rw = jmp_rx - tcg_splitwx_diff;
+
tb->jmp_target_addr[n] = addr;
- if (TCG_TARGET_HAS_direct_jump) {
- const TranslationBlock *c_tb = tcg_splitwx_to_rx(tb);
- uintptr_t offset = tb->jmp_insn_offset[n];
- uintptr_t jmp_rx = (uintptr_t)tb->tc.ptr + offset;
- uintptr_t jmp_rw = jmp_rx - tcg_splitwx_diff;
- tb_target_set_jmp_target(c_tb, n, jmp_rx, jmp_rw);
- }
+ tb_target_set_jmp_target(c_tb, n, jmp_rx, jmp_rw);
}
static inline void tb_add_jump(TranslationBlock *tb, int n,
diff --git a/tcg/tcg.c b/tcg/tcg.c
index c103dd0037..b09500ffb1 100644
--- a/tcg/tcg.c
+++ b/tcg/tcg.c
@@ -327,7 +327,6 @@ static void G_GNUC_UNUSED set_jmp_insn_offset(TCGContext *s, int which)
* We will check for overflow at the end of the opcode loop in
* tcg_gen_code, where we bound tcg_current_code_size to UINT16_MAX.
*/
- tcg_debug_assert(TCG_TARGET_HAS_direct_jump);
s->gen_tb->jmp_insn_offset[which] = tcg_current_code_size(s);
}
diff --git a/tcg/arm/tcg-target.c.inc b/tcg/arm/tcg-target.c.inc
index f5103dcc6d..a44d1d969f 100644
--- a/tcg/arm/tcg-target.c.inc
+++ b/tcg/arm/tcg-target.c.inc
@@ -1939,7 +1939,6 @@ static void tcg_out_goto_tb(TCGContext *s, int which)
intptr_t ptr, dif, dil;
TCGReg base = TCG_REG_PC;
- qemu_build_assert(!TCG_TARGET_HAS_direct_jump);
ptr = get_jmp_target_addr(s, which);
dif = tcg_pcrel_diff(s, (void *)ptr) - 8;
dil = sextract32(dif, 0, 12);
diff --git a/tcg/mips/tcg-target.c.inc b/tcg/mips/tcg-target.c.inc
index dfb31a17bb..d52b6a902b 100644
--- a/tcg/mips/tcg-target.c.inc
+++ b/tcg/mips/tcg-target.c.inc
@@ -1968,7 +1968,6 @@ static void tcg_out_exit_tb(TCGContext *s, uintptr_t a0)
static void tcg_out_goto_tb(TCGContext *s, int which)
{
/* indirect jump method */
- qemu_build_assert(!TCG_TARGET_HAS_direct_jump);
tcg_out_ld(s, TCG_TYPE_PTR, TCG_TMP0, TCG_REG_ZERO,
get_jmp_target_addr(s, which));
tcg_out_opc_reg(s, OPC_JR, 0, TCG_TMP0, 0);
diff --git a/tcg/riscv/tcg-target.c.inc b/tcg/riscv/tcg-target.c.inc
index 06308c5243..37baae9cda 100644
--- a/tcg/riscv/tcg-target.c.inc
+++ b/tcg/riscv/tcg-target.c.inc
@@ -1277,7 +1277,6 @@ static void tcg_out_exit_tb(TCGContext *s, uintptr_t a0)
static void tcg_out_goto_tb(TCGContext *s, int which)
{
- qemu_build_assert(!TCG_TARGET_HAS_direct_jump);
/* indirect jump method */
tcg_out_ld(s, TCG_TYPE_PTR, TCG_REG_TMP0, TCG_REG_ZERO,
get_jmp_target_addr(s, which));
diff --git a/tcg/s390x/tcg-target.c.inc b/tcg/s390x/tcg-target.c.inc
index 82435ecadb..cf96ccb053 100644
--- a/tcg/s390x/tcg-target.c.inc
+++ b/tcg/s390x/tcg-target.c.inc
@@ -2083,7 +2083,7 @@ static void tcg_out_exit_tb(TCGContext *s, uintptr_t a0)
static void tcg_out_goto_tb(TCGContext *s, int which)
{
- if (TCG_TARGET_HAS_direct_jump) {
+ if (HAVE_FACILITY(GEN_INST_EXT)) {
/*
* Branch displacement must be aligned for atomic patching;
* see if we need to add extra nop before branch
@@ -2119,6 +2119,9 @@ static void tcg_out_goto_tb(TCGContext *s, int which)
void tb_target_set_jmp_target(const TranslationBlock *tb, int n,
uintptr_t jmp_rx, uintptr_t jmp_rw)
{
+ if (!HAVE_FACILITY(GEN_INST_EXT)) {
+ return;
+ }
/* patch the branch destination */
uintptr_t addr = tb->jmp_target_addr[n];
intptr_t disp = addr - (jmp_rx - 2);
diff --git a/tcg/tci/tcg-target.c.inc b/tcg/tci/tcg-target.c.inc
index 31094067b7..434fcc74dc 100644
--- a/tcg/tci/tcg-target.c.inc
+++ b/tcg/tci/tcg-target.c.inc
@@ -600,7 +600,6 @@ static void tcg_out_exit_tb(TCGContext *s, uintptr_t arg)
static void tcg_out_goto_tb(TCGContext *s, int which)
{
- qemu_build_assert(!TCG_TARGET_HAS_direct_jump);
/* indirect jump method. */
tcg_out_op_p(s, INDEX_op_goto_tb, (void *)get_jmp_target_addr(s, which));
set_jmp_reset_offset(s, which);
--
2.34.1
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH 16/22] tcg/aarch64: Reorg goto_tb implementation
2022-12-06 4:16 [PATCH 00/22] tcg: exit_tb tidy, goto_tb reorg Richard Henderson
` (14 preceding siblings ...)
2022-12-06 4:17 ` [PATCH 15/22] tcg: Remove TCG_TARGET_HAS_direct_jump Richard Henderson
@ 2022-12-06 4:17 ` Richard Henderson
2022-12-06 4:17 ` [PATCH 17/22] tcg/ppc: " Richard Henderson
` (5 subsequent siblings)
21 siblings, 0 replies; 35+ messages in thread
From: Richard Henderson @ 2022-12-06 4:17 UTC (permalink / raw)
To: qemu-devel; +Cc: r
The old implementation replaces two insns, swapping between
b <dest>
nop
br x30
and
adrp x30, <dest>
addi x30, x30, lo12:<dest>
br x30
There is a race condition in which a thread could be stopped at
the PC of the second insn, and when restarted does not see the
complete address computation and branches to nowhere.
The new implemetation replaces only one insn, swapping between
b <dest>
br tmp
and
ldr tmp, <jmp_addr>
br tmp
Reported-by: hev <r@hev.cc>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
tcg/aarch64/tcg-target.h | 2 +-
tcg/aarch64/tcg-target.c.inc | 64 +++++++++++++++---------------------
2 files changed, 27 insertions(+), 39 deletions(-)
diff --git a/tcg/aarch64/tcg-target.h b/tcg/aarch64/tcg-target.h
index c8202e4bc5..9b8835eaa3 100644
--- a/tcg/aarch64/tcg-target.h
+++ b/tcg/aarch64/tcg-target.h
@@ -15,7 +15,7 @@
#define TCG_TARGET_INSN_UNIT_SIZE 4
#define TCG_TARGET_TLB_DISPLACEMENT_BITS 24
-#define MAX_CODE_GEN_BUFFER_SIZE (2 * GiB)
+#define MAX_CODE_GEN_BUFFER_SIZE ((size_t)-1)
#undef TCG_TARGET_STACK_GROWSUP
typedef enum {
diff --git a/tcg/aarch64/tcg-target.c.inc b/tcg/aarch64/tcg-target.c.inc
index af30f9db69..fe8e8636aa 100644
--- a/tcg/aarch64/tcg-target.c.inc
+++ b/tcg/aarch64/tcg-target.c.inc
@@ -1352,33 +1352,6 @@ static void tcg_out_call(TCGContext *s, const tcg_insn_unit *target)
}
}
-void tb_target_set_jmp_target(const TranslationBlock *tb, int n,
- uintptr_t jmp_rx, uintptr_t jmp_rw)
-{
- uintptr_t addr = tb->jmp_target_addr[n];
- tcg_insn_unit i1, i2;
- TCGType rt = TCG_TYPE_I64;
- TCGReg rd = TCG_REG_TMP;
- uint64_t pair;
-
- ptrdiff_t offset = addr - jmp_rx;
-
- if (offset == sextract64(offset, 0, 26)) {
- i1 = I3206_B | ((offset >> 2) & 0x3ffffff);
- i2 = NOP;
- } else {
- offset = (addr >> 12) - (jmp_rx >> 12);
-
- /* patch ADRP */
- i1 = I3406_ADRP | (offset & 3) << 29 | (offset & 0x1ffffc) << (5 - 2) | rd;
- /* patch ADDI */
- i2 = I3401_ADDI | rt << 31 | (addr & 0xfff) << 10 | rd << 5 | rd;
- }
- pair = (uint64_t)i2 << 32 | i1;
- qatomic_set((uint64_t *)jmp_rw, pair);
- flush_idcache_range(jmp_rx, jmp_rw, 8);
-}
-
static inline void tcg_out_goto_label(TCGContext *s, TCGLabel *l)
{
if (!l->has_value) {
@@ -1901,23 +1874,38 @@ static void tcg_out_exit_tb(TCGContext *s, uintptr_t a0)
static void tcg_out_goto_tb(TCGContext *s, int which)
{
/*
- * Ensure that ADRP+ADD are 8-byte aligned so that an atomic
- * write can be used to patch the target address.
+ * Direct branch, or indirect address load, will be patched
+ * by tb_target_set_jmp_target. Assert indirect load offset
+ * in range early, regardless of direct branch distance.
*/
- if ((uintptr_t)s->code_ptr & 7) {
- tcg_out32(s, NOP);
- }
+ intptr_t i_off = tcg_pcrel_diff(s, (void *)get_jmp_target_addr(s, which));
+ tcg_debug_assert(i_off == sextract64(i_off, 0, 21));
+
set_jmp_insn_offset(s, which);
- /*
- * actual branch destination will be patched by
- * tb_target_set_jmp_target later
- */
- tcg_out_insn(s, 3406, ADRP, TCG_REG_TMP, 0);
- tcg_out_insn(s, 3401, ADDI, TCG_TYPE_I64, TCG_REG_TMP, TCG_REG_TMP, 0);
+ tcg_out32(s, I3206_B);
tcg_out_insn(s, 3207, BR, TCG_REG_TMP);
set_jmp_reset_offset(s, which);
}
+void tb_target_set_jmp_target(const TranslationBlock *tb, int n,
+ uintptr_t jmp_rx, uintptr_t jmp_rw)
+{
+ uintptr_t d_addr = tb->jmp_target_addr[n];
+ uintptr_t i_addr = (uintptr_t)&tb->jmp_target_addr[n];
+ ptrdiff_t d_offset = d_addr - jmp_rx;
+ ptrdiff_t i_offset = i_addr - jmp_rx;
+ tcg_insn_unit insn;
+
+ /* Either directly branch, or indirect branch load. */
+ if (d_offset == sextract64(d_offset, 0, 26)) {
+ insn = I3206_B | ((d_offset >> 2) & 0x3ffffff);
+ } else {
+ insn = I3305_LDR | TCG_REG_TMP | (((i_offset >> 2) & 0x7ffff) << 5);
+ }
+ qatomic_set((uint32_t *)jmp_rw, insn);
+ flush_idcache_range(jmp_rx, jmp_rw, 4);
+}
+
static void tcg_out_op(TCGContext *s, TCGOpcode opc,
const TCGArg args[TCG_MAX_OP_ARGS],
const int const_args[TCG_MAX_OP_ARGS])
--
2.34.1
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH 17/22] tcg/ppc: Reorg goto_tb implementation
2022-12-06 4:16 [PATCH 00/22] tcg: exit_tb tidy, goto_tb reorg Richard Henderson
` (15 preceding siblings ...)
2022-12-06 4:17 ` [PATCH 16/22] tcg/aarch64: Reorg goto_tb implementation Richard Henderson
@ 2022-12-06 4:17 ` Richard Henderson
2022-12-06 4:17 ` [PATCH 18/22] tcg/sparc64: Remove USE_REG_TB Richard Henderson
` (4 subsequent siblings)
21 siblings, 0 replies; 35+ messages in thread
From: Richard Henderson @ 2022-12-06 4:17 UTC (permalink / raw)
To: qemu-devel; +Cc: r
The old ppc64 implementation replaces 2 or 4 insns, which leaves a race
condition in which a thread could be stopped at a PC in the middle of
the sequence, and when restarted does not see the complete address
computation and branches to nowhere.
The new implemetation replaces only one insn, swapping between
b <dest>
and
mtctr r31
falling through to a general-case indirect branch.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
tcg/ppc/tcg-target.h | 3 +-
tcg/ppc/tcg-target.c.inc | 158 +++++++++++----------------------------
2 files changed, 44 insertions(+), 117 deletions(-)
diff --git a/tcg/ppc/tcg-target.h b/tcg/ppc/tcg-target.h
index f253184915..af81c5a57f 100644
--- a/tcg/ppc/tcg-target.h
+++ b/tcg/ppc/tcg-target.h
@@ -27,11 +27,10 @@
#ifdef _ARCH_PPC64
# define TCG_TARGET_REG_BITS 64
-# define MAX_CODE_GEN_BUFFER_SIZE (2 * GiB)
#else
# define TCG_TARGET_REG_BITS 32
-# define MAX_CODE_GEN_BUFFER_SIZE (32 * MiB)
#endif
+#define MAX_CODE_GEN_BUFFER_SIZE ((size_t)-1)
#define TCG_TARGET_NB_REGS 64
#define TCG_TARGET_INSN_UNIT_SIZE 4
diff --git a/tcg/ppc/tcg-target.c.inc b/tcg/ppc/tcg-target.c.inc
index 592b8d6498..755b954447 100644
--- a/tcg/ppc/tcg-target.c.inc
+++ b/tcg/ppc/tcg-target.c.inc
@@ -1847,104 +1847,6 @@ static void tcg_out_mb(TCGContext *s, TCGArg a0)
tcg_out32(s, insn);
}
-static inline uint64_t make_pair(tcg_insn_unit i1, tcg_insn_unit i2)
-{
- if (HOST_BIG_ENDIAN) {
- return (uint64_t)i1 << 32 | i2;
- }
- return (uint64_t)i2 << 32 | i1;
-}
-
-static inline void ppc64_replace2(uintptr_t rx, uintptr_t rw,
- tcg_insn_unit i0, tcg_insn_unit i1)
-{
-#if TCG_TARGET_REG_BITS == 64
- qatomic_set((uint64_t *)rw, make_pair(i0, i1));
- flush_idcache_range(rx, rw, 8);
-#else
- qemu_build_not_reached();
-#endif
-}
-
-static inline void ppc64_replace4(uintptr_t rx, uintptr_t rw,
- tcg_insn_unit i0, tcg_insn_unit i1,
- tcg_insn_unit i2, tcg_insn_unit i3)
-{
- uint64_t p[2];
-
- p[!HOST_BIG_ENDIAN] = make_pair(i0, i1);
- p[HOST_BIG_ENDIAN] = make_pair(i2, i3);
-
- /*
- * There's no convenient way to get the compiler to allocate a pair
- * of registers at an even index, so copy into r6/r7 and clobber.
- */
- asm("mr %%r6, %1\n\t"
- "mr %%r7, %2\n\t"
- "stq %%r6, %0"
- : "=Q"(*(__int128 *)rw) : "r"(p[0]), "r"(p[1]) : "r6", "r7");
- flush_idcache_range(rx, rw, 16);
-}
-
-void tb_target_set_jmp_target(const TranslationBlock *tb, int n,
- uintptr_t jmp_rx, uintptr_t jmp_rw)
-{
- tcg_insn_unit i0, i1, i2, i3;
- uintptr_t addr = tb->jmp_target_addr[n];
- intptr_t tb_diff = addr - (uintptr_t)tb->tc.ptr;
- intptr_t br_diff = addr - (jmp_rx + 4);
- intptr_t lo, hi;
-
- if (TCG_TARGET_REG_BITS == 32) {
- intptr_t diff = addr - jmp_rx;
- tcg_debug_assert(in_range_b(diff));
- qatomic_set((uint32_t *)jmp_rw, B | (diff & 0x3fffffc));
- flush_idcache_range(jmp_rx, jmp_rw, 4);
- return;
- }
-
- /*
- * For 16-bit displacements, we can use a single add + branch.
- * This happens quite often.
- */
- if (tb_diff == (int16_t)tb_diff) {
- i0 = ADDI | TAI(TCG_REG_TB, TCG_REG_TB, tb_diff);
- i1 = B | (br_diff & 0x3fffffc);
- ppc64_replace2(jmp_rx, jmp_rw, i0, i1);
- return;
- }
-
- lo = (int16_t)tb_diff;
- hi = (int32_t)(tb_diff - lo);
- assert(tb_diff == hi + lo);
- i0 = ADDIS | TAI(TCG_REG_TB, TCG_REG_TB, hi >> 16);
- i1 = ADDI | TAI(TCG_REG_TB, TCG_REG_TB, lo);
-
- /*
- * Without stq from 2.07, we can only update two insns,
- * and those must be the ones that load the target address.
- */
- if (!have_isa_2_07) {
- ppc64_replace2(jmp_rx, jmp_rw, i0, i1);
- return;
- }
-
- /*
- * For 26-bit displacements, we can use a direct branch.
- * Otherwise we still need the indirect branch, which we
- * must restore after a potential direct branch write.
- */
- br_diff -= 4;
- if (in_range_b(br_diff)) {
- i2 = B | (br_diff & 0x3fffffc);
- i3 = NOP;
- } else {
- i2 = MTSPR | RS(TCG_REG_TB) | CTR;
- i3 = BCCTR | BO_ALWAYS;
- }
- ppc64_replace4(jmp_rx, jmp_rw, i0, i1, i2, i3);
-}
-
static void tcg_out_call_int(TCGContext *s, int lk,
const tcg_insn_unit *target)
{
@@ -2625,30 +2527,56 @@ static void tcg_out_exit_tb(TCGContext *s, uintptr_t arg)
static void tcg_out_goto_tb(TCGContext *s, int which)
{
- /* Direct jump. */
- if (TCG_TARGET_REG_BITS == 64) {
- /* Ensure the next insns are 8 or 16-byte aligned. */
- while ((uintptr_t)s->code_ptr & (have_isa_2_07 ? 15 : 7)) {
- tcg_out32(s, NOP);
- }
+ uintptr_t ptr = get_jmp_target_addr(s, which);
+
+ if (USE_REG_TB) {
+ ptrdiff_t offset = tcg_tbrel_diff(s, (void *)ptr);
+ tcg_out_mem_long(s, LD, LDX, TCG_REG_TB, TCG_REG_TB, offset);
+
+ /* Direct branch will be patched by tb_target_set_jmp_target. */
set_jmp_insn_offset(s, which);
- tcg_out32(s, ADDIS | TAI(TCG_REG_TB, TCG_REG_TB, 0));
- tcg_out32(s, ADDI | TAI(TCG_REG_TB, TCG_REG_TB, 0));
tcg_out32(s, MTSPR | RS(TCG_REG_TB) | CTR);
+
+ /* When branch is out of range, fall through to indirect. */
+ tcg_out32(s, BCCTR | BO_ALWAYS);
+
+ /* For the unlinked case, need to reset TCG_REG_TB. */
+ set_jmp_reset_offset(s, which);
+ tcg_out_mem_long(s, ADDI, ADD, TCG_REG_TB, TCG_REG_TB,
+ -tcg_current_code_size(s));
+ } else {
+ /* Direct branch will be patched by tb_target_set_jmp_target. */
+ set_jmp_insn_offset(s, which);
+ tcg_out32(s, NOP);
+
+ /* When branch is out of range, fall through to indirect. */
+ tcg_out_movi(s, TCG_TYPE_PTR, TCG_REG_TMP1, ptr - (int16_t)ptr);
+ tcg_out_ld(s, TCG_TYPE_PTR, TCG_REG_TMP1, TCG_REG_TMP1, (int16_t)ptr);
+ tcg_out32(s, MTSPR | RS(TCG_REG_TMP1) | CTR);
tcg_out32(s, BCCTR | BO_ALWAYS);
set_jmp_reset_offset(s, which);
- if (USE_REG_TB) {
- /* For the unlinked case, need to reset TCG_REG_TB. */
- tcg_out_mem_long(s, ADDI, ADD, TCG_REG_TB, TCG_REG_TB,
- -tcg_current_code_size(s));
- }
- } else {
- set_jmp_insn_offset(s, which);
- tcg_out32(s, B);
- set_jmp_reset_offset(s, which);
}
}
+void tb_target_set_jmp_target(const TranslationBlock *tb, int n,
+ uintptr_t jmp_rx, uintptr_t jmp_rw)
+{
+ uintptr_t addr = tb->jmp_target_addr[n];
+ intptr_t diff = addr - jmp_rx;
+ tcg_insn_unit insn;
+
+ if (in_range_b(diff)) {
+ insn = B | (diff & 0x3fffffc);
+ } else if (USE_REG_TB) {
+ insn = MTSPR | RS(TCG_REG_TB) | CTR;
+ } else {
+ insn = NOP;
+ }
+
+ qatomic_set((uint32_t *)jmp_rw, insn);
+ flush_idcache_range(jmp_rx, jmp_rw, 4);
+}
+
static void tcg_out_op(TCGContext *s, TCGOpcode opc,
const TCGArg args[TCG_MAX_OP_ARGS],
const int const_args[TCG_MAX_OP_ARGS])
--
2.34.1
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH 18/22] tcg/sparc64: Remove USE_REG_TB
2022-12-06 4:16 [PATCH 00/22] tcg: exit_tb tidy, goto_tb reorg Richard Henderson
` (16 preceding siblings ...)
2022-12-06 4:17 ` [PATCH 17/22] tcg/ppc: " Richard Henderson
@ 2022-12-06 4:17 ` Richard Henderson
2022-12-06 7:44 ` Philippe Mathieu-Daudé
2022-12-06 4:17 ` [PATCH 19/22] tcg/sparc64: Reorg goto_tb implementation Richard Henderson
` (3 subsequent siblings)
21 siblings, 1 reply; 35+ messages in thread
From: Richard Henderson @ 2022-12-06 4:17 UTC (permalink / raw)
To: qemu-devel; +Cc: r
This is always true for sparc64, so this is dead since 3a5f6805c7ca.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
tcg/sparc64/tcg-target.c.inc | 57 ++++++++++++++----------------------
1 file changed, 22 insertions(+), 35 deletions(-)
diff --git a/tcg/sparc64/tcg-target.c.inc b/tcg/sparc64/tcg-target.c.inc
index 96d58f30b1..1ae9615ef0 100644
--- a/tcg/sparc64/tcg-target.c.inc
+++ b/tcg/sparc64/tcg-target.c.inc
@@ -92,7 +92,6 @@ static const char * const tcg_target_reg_names[TCG_TARGET_NB_REGS] = {
#endif
#define TCG_REG_TB TCG_REG_I1
-#define USE_REG_TB (sizeof(void *) > 4)
static const int tcg_target_reg_alloc_order[] = {
TCG_REG_L0,
@@ -439,7 +438,7 @@ static void tcg_out_movi_int(TCGContext *s, TCGType type, TCGReg ret,
}
/* A 13-bit constant relative to the TB. */
- if (!in_prologue && USE_REG_TB) {
+ if (!in_prologue) {
test = tcg_tbrel_diff(s, (void *)arg);
if (check_fit_ptr(test, 13)) {
tcg_out_arithi(s, ret, TCG_REG_TB, test, ARITH_ADD);
@@ -468,7 +467,7 @@ static void tcg_out_movi_int(TCGContext *s, TCGType type, TCGReg ret,
}
/* Use the constant pool, if possible. */
- if (!in_prologue && USE_REG_TB) {
+ if (!in_prologue) {
new_pool_label(s, arg, R_SPARC_13, s->code_ptr,
tcg_tbrel_diff(s, NULL));
tcg_out32(s, LDX | INSN_RD(ret) | INSN_RS1(TCG_REG_TB));
@@ -1014,10 +1013,8 @@ static void tcg_target_qemu_prologue(TCGContext *s)
#endif
/* We choose TCG_REG_TB such that no move is required. */
- if (USE_REG_TB) {
- QEMU_BUILD_BUG_ON(TCG_REG_TB != TCG_REG_I1);
- tcg_regset_set_reg(s->reserved_regs, TCG_REG_TB);
- }
+ QEMU_BUILD_BUG_ON(TCG_REG_TB != TCG_REG_I1);
+ tcg_regset_set_reg(s->reserved_regs, TCG_REG_TB);
tcg_out_arithi(s, TCG_REG_G0, TCG_REG_I1, 0, JMPL);
/* delay slot */
@@ -1422,7 +1419,7 @@ static void tcg_out_exit_tb(TCGContext *s, uintptr_t a0)
tcg_out_arithi(s, TCG_REG_G0, TCG_REG_I7, 8, RETURN);
tcg_out_movi_imm13(s, TCG_REG_O0, a0);
return;
- } else if (USE_REG_TB) {
+ } else {
intptr_t tb_diff = tcg_tbrel_diff(s, (void *)a0);
if (check_fit_ptr(tb_diff, 13)) {
tcg_out_arithi(s, TCG_REG_G0, TCG_REG_I7, 8, RETURN);
@@ -1438,36 +1435,30 @@ static void tcg_out_exit_tb(TCGContext *s, uintptr_t a0)
static void tcg_out_goto_tb(TCGContext *s, int which)
{
+ int c;
+
/* Direct jump. */
- if (USE_REG_TB) {
- /* make sure the patch is 8-byte aligned. */
- if ((intptr_t)s->code_ptr & 4) {
- tcg_out_nop(s);
- }
- set_jmp_insn_offset(s, which);
- tcg_out_sethi(s, TCG_REG_T1, 0);
- tcg_out_arithi(s, TCG_REG_T1, TCG_REG_T1, 0, ARITH_OR);
- tcg_out_arith(s, TCG_REG_G0, TCG_REG_TB, TCG_REG_T1, JMPL);
- tcg_out_arith(s, TCG_REG_TB, TCG_REG_TB, TCG_REG_T1, ARITH_ADD);
- } else {
- set_jmp_insn_offset(s, which);
- tcg_out32(s, CALL);
+ /* make sure the patch is 8-byte aligned. */
+ if ((intptr_t)s->code_ptr & 4) {
tcg_out_nop(s);
}
+ set_jmp_insn_offset(s, which);
+ tcg_out_sethi(s, TCG_REG_T1, 0);
+ tcg_out_arithi(s, TCG_REG_T1, TCG_REG_T1, 0, ARITH_OR);
+ tcg_out_arith(s, TCG_REG_G0, TCG_REG_TB, TCG_REG_T1, JMPL);
+ tcg_out_arith(s, TCG_REG_TB, TCG_REG_TB, TCG_REG_T1, ARITH_ADD);
set_jmp_reset_offset(s, which);
/*
* For the unlinked path of goto_tb, we need to reset TCG_REG_TB
* to the beginning of this TB.
*/
- if (USE_REG_TB) {
- int c = -tcg_current_code_size(s);
- if (check_fit_i32(c, 13)) {
- tcg_out_arithi(s, TCG_REG_TB, TCG_REG_TB, c, ARITH_ADD);
- } else {
- tcg_out_movi(s, TCG_TYPE_PTR, TCG_REG_T1, c);
- tcg_out_arith(s, TCG_REG_TB, TCG_REG_TB, TCG_REG_T1, ARITH_ADD);
- }
+ c = -tcg_current_code_size(s);
+ if (check_fit_i32(c, 13)) {
+ tcg_out_arithi(s, TCG_REG_TB, TCG_REG_TB, c, ARITH_ADD);
+ } else {
+ tcg_out_movi(s, TCG_TYPE_PTR, TCG_REG_T1, c);
+ tcg_out_arith(s, TCG_REG_TB, TCG_REG_TB, TCG_REG_T1, ARITH_ADD);
}
}
@@ -1487,11 +1478,7 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc,
switch (opc) {
case INDEX_op_goto_ptr:
tcg_out_arithi(s, TCG_REG_G0, a0, 0, JMPL);
- if (USE_REG_TB) {
- tcg_out_mov_delay(s, TCG_REG_TB, a0);
- } else {
- tcg_out_nop(s);
- }
+ tcg_out_mov_delay(s, TCG_REG_TB, a0);
break;
case INDEX_op_br:
tcg_out_bpcc(s, COND_A, BPCC_PT, arg_label(a0));
@@ -1897,7 +1884,7 @@ void tb_target_set_jmp_target(const TranslationBlock *tb, int n,
tcg_debug_assert(tb_disp == (int32_t)tb_disp);
tcg_debug_assert(br_disp == (int32_t)br_disp);
- if (!USE_REG_TB) {
+ if (0) {
qatomic_set((uint32_t *)jmp_rw,
deposit32(CALL, 0, 30, br_disp >> 2));
flush_idcache_range(jmp_rx, jmp_rw, 4);
--
2.34.1
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH 19/22] tcg/sparc64: Reorg goto_tb implementation
2022-12-06 4:16 [PATCH 00/22] tcg: exit_tb tidy, goto_tb reorg Richard Henderson
` (17 preceding siblings ...)
2022-12-06 4:17 ` [PATCH 18/22] tcg/sparc64: Remove USE_REG_TB Richard Henderson
@ 2022-12-06 4:17 ` Richard Henderson
2022-12-06 4:17 ` [PATCH 20/22] tcg/arm: Implement direct branch for goto_tb Richard Henderson
` (2 subsequent siblings)
21 siblings, 0 replies; 35+ messages in thread
From: Richard Henderson @ 2022-12-06 4:17 UTC (permalink / raw)
To: qemu-devel; +Cc: r
The old sparc64 implementation may replace two insns, which leaves
a race condition in which a thread could be stopped at a PC in the
middle of the sequence, and when restarted does not see the complete
address computation and branches to nowhere.
The new implemetation replaces only one insn, swapping between a
direct branch and a direct call. The TCG_REG_TB register is loaded
from tb->jmp_target_addr[] in the delay slot.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
tcg/sparc64/tcg-target.c.inc | 93 ++++++++++++++----------------------
1 file changed, 37 insertions(+), 56 deletions(-)
diff --git a/tcg/sparc64/tcg-target.c.inc b/tcg/sparc64/tcg-target.c.inc
index 1ae9615ef0..07ebea7a6e 100644
--- a/tcg/sparc64/tcg-target.c.inc
+++ b/tcg/sparc64/tcg-target.c.inc
@@ -1435,33 +1435,56 @@ static void tcg_out_exit_tb(TCGContext *s, uintptr_t a0)
static void tcg_out_goto_tb(TCGContext *s, int which)
{
- int c;
+ ptrdiff_t off = tcg_tbrel_diff(s, (void *)get_jmp_target_addr(s, which));
- /* Direct jump. */
- /* make sure the patch is 8-byte aligned. */
- if ((intptr_t)s->code_ptr & 4) {
- tcg_out_nop(s);
- }
+ /* Direct branch will be patched by tb_target_set_jmp_target. */
set_jmp_insn_offset(s, which);
- tcg_out_sethi(s, TCG_REG_T1, 0);
- tcg_out_arithi(s, TCG_REG_T1, TCG_REG_T1, 0, ARITH_OR);
- tcg_out_arith(s, TCG_REG_G0, TCG_REG_TB, TCG_REG_T1, JMPL);
- tcg_out_arith(s, TCG_REG_TB, TCG_REG_TB, TCG_REG_T1, ARITH_ADD);
+ tcg_out32(s, CALL);
+ /* delay slot */
+ tcg_debug_assert(check_fit_ptr(off, 13));
+ tcg_out_ld(s, TCG_TYPE_PTR, TCG_REG_TB, TCG_REG_TB, off);
set_jmp_reset_offset(s, which);
/*
* For the unlinked path of goto_tb, we need to reset TCG_REG_TB
* to the beginning of this TB.
*/
- c = -tcg_current_code_size(s);
- if (check_fit_i32(c, 13)) {
- tcg_out_arithi(s, TCG_REG_TB, TCG_REG_TB, c, ARITH_ADD);
+ off = -tcg_current_code_size(s);
+ if (check_fit_i32(off, 13)) {
+ tcg_out_arithi(s, TCG_REG_TB, TCG_REG_TB, off, ARITH_ADD);
} else {
- tcg_out_movi(s, TCG_TYPE_PTR, TCG_REG_T1, c);
+ tcg_out_movi(s, TCG_TYPE_PTR, TCG_REG_T1, off);
tcg_out_arith(s, TCG_REG_TB, TCG_REG_TB, TCG_REG_T1, ARITH_ADD);
}
}
+void tb_target_set_jmp_target(const TranslationBlock *tb, int n,
+ uintptr_t jmp_rx, uintptr_t jmp_rw)
+{
+ uintptr_t addr = tb->jmp_target_addr[n];
+ intptr_t br_disp = (intptr_t)(addr - jmp_rx) >> 2;
+ tcg_insn_unit insn;
+
+ br_disp >>= 2;
+ if (check_fit_ptr(br_disp, 19)) {
+ /* ba,pt %icc, addr */
+ insn = deposit32(INSN_OP(0) | INSN_OP2(1) | INSN_COND(COND_A)
+ | BPCC_ICC | BPCC_PT, 0, 19, br_disp);
+ } else if (check_fit_ptr(br_disp, 22)) {
+ /* ba addr */
+ insn = deposit32(INSN_OP(0) | INSN_OP2(2) | INSN_COND(COND_A),
+ 0, 22, br_disp);
+ } else {
+ /* The code_gen_buffer can't be larger than 2GB. */
+ tcg_debug_assert(check_fit_ptr(br_disp, 30));
+ /* call addr */
+ insn = deposit32(CALL, 0, 30, br_disp);
+ }
+
+ qatomic_set((uint32_t *)jmp_rw, insn);
+ flush_idcache_range(jmp_rx, jmp_rw, 4);
+}
+
static void tcg_out_op(TCGContext *s, TCGOpcode opc,
const TCGArg args[TCG_MAX_OP_ARGS],
const int const_args[TCG_MAX_OP_ARGS])
@@ -1871,45 +1894,3 @@ void tcg_register_jit(const void *buf, size_t buf_size)
tcg_register_jit_int(buf, buf_size, &debug_frame, sizeof(debug_frame));
}
-void tb_target_set_jmp_target(const TranslationBlock *tb, int n,
- uintptr_t jmp_rx, uintptr_t jmp_rw)
-{
- uintptr_t addr = tb->jmp_target_addr[n];
- intptr_t tb_disp = addr - (uintptr_t)tb->tc.ptr;
- intptr_t br_disp = addr - jmp_rx;
- tcg_insn_unit i1, i2;
-
- /* We can reach the entire address space for ILP32.
- For LP64, the code_gen_buffer can't be larger than 2GB. */
- tcg_debug_assert(tb_disp == (int32_t)tb_disp);
- tcg_debug_assert(br_disp == (int32_t)br_disp);
-
- if (0) {
- qatomic_set((uint32_t *)jmp_rw,
- deposit32(CALL, 0, 30, br_disp >> 2));
- flush_idcache_range(jmp_rx, jmp_rw, 4);
- return;
- }
-
- /* This does not exercise the range of the branch, but we do
- still need to be able to load the new value of TCG_REG_TB.
- But this does still happen quite often. */
- if (check_fit_ptr(tb_disp, 13)) {
- /* ba,pt %icc, addr */
- i1 = (INSN_OP(0) | INSN_OP2(1) | INSN_COND(COND_A)
- | BPCC_ICC | BPCC_PT | INSN_OFF19(br_disp));
- i2 = (ARITH_ADD | INSN_RD(TCG_REG_TB) | INSN_RS1(TCG_REG_TB)
- | INSN_IMM13(tb_disp));
- } else if (tb_disp >= 0) {
- i1 = SETHI | INSN_RD(TCG_REG_T1) | ((tb_disp & 0xfffffc00) >> 10);
- i2 = (ARITH_OR | INSN_RD(TCG_REG_T1) | INSN_RS1(TCG_REG_T1)
- | INSN_IMM13(tb_disp & 0x3ff));
- } else {
- i1 = SETHI | INSN_RD(TCG_REG_T1) | ((~tb_disp & 0xfffffc00) >> 10);
- i2 = (ARITH_XOR | INSN_RD(TCG_REG_T1) | INSN_RS1(TCG_REG_T1)
- | INSN_IMM13((tb_disp & 0x3ff) | -0x400));
- }
-
- qatomic_set((uint64_t *)jmp_rw, deposit64(i2, 32, 32, i1));
- flush_idcache_range(jmp_rx, jmp_rw, 8);
-}
--
2.34.1
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH 20/22] tcg/arm: Implement direct branch for goto_tb
2022-12-06 4:16 [PATCH 00/22] tcg: exit_tb tidy, goto_tb reorg Richard Henderson
` (18 preceding siblings ...)
2022-12-06 4:17 ` [PATCH 19/22] tcg/sparc64: Reorg goto_tb implementation Richard Henderson
@ 2022-12-06 4:17 ` Richard Henderson
2022-12-06 4:17 ` [PATCH 21/22] tcg/riscv: Introduce OPC_NOP Richard Henderson
2022-12-06 4:17 ` [PATCH 22/22] tcg/riscv: Implement direct branch for goto_tb Richard Henderson
21 siblings, 0 replies; 35+ messages in thread
From: Richard Henderson @ 2022-12-06 4:17 UTC (permalink / raw)
To: qemu-devel; +Cc: r
Now that tcg can handle direct and indirect goto_tb
simultaneously, we can optimistically leave space for
a direct branch and fall back to loading the pointer
from the TB for an indirect branch.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
tcg/arm/tcg-target.c.inc | 52 ++++++++++++++++++++++++++++------------
1 file changed, 37 insertions(+), 15 deletions(-)
diff --git a/tcg/arm/tcg-target.c.inc b/tcg/arm/tcg-target.c.inc
index a44d1d969f..fad2be700e 100644
--- a/tcg/arm/tcg-target.c.inc
+++ b/tcg/arm/tcg-target.c.inc
@@ -135,6 +135,8 @@ typedef enum {
ARITH_BIC = 0xe << 21,
ARITH_MVN = 0xf << 21,
+ INSN_B = 0x0a000000,
+
INSN_CLZ = 0x016f0f10,
INSN_RBIT = 0x06ff0f30,
@@ -546,7 +548,7 @@ static bool tcg_target_const_match(int64_t val, TCGType type, int ct)
static void tcg_out_b_imm(TCGContext *s, ARMCond cond, int32_t offset)
{
- tcg_out32(s, (cond << 28) | 0x0a000000 |
+ tcg_out32(s, (cond << 28) | INSN_B |
(((offset - 8) >> 2) & 0x00ffffff));
}
@@ -1935,32 +1937,52 @@ static void tcg_out_exit_tb(TCGContext *s, uintptr_t arg)
static void tcg_out_goto_tb(TCGContext *s, int which)
{
- /* Indirect jump method */
- intptr_t ptr, dif, dil;
- TCGReg base = TCG_REG_PC;
+ uintptr_t i_addr;
+ intptr_t i_disp;
- ptr = get_jmp_target_addr(s, which);
- dif = tcg_pcrel_diff(s, (void *)ptr) - 8;
- dil = sextract32(dif, 0, 12);
- if (dif != dil) {
+ /* Direct branch will be patched by tb_target_set_jmp_target. */
+ set_jmp_insn_offset(s, which);
+ tcg_out32(s, INSN_NOP);
+
+ /* When branch is out of range, fall through to indirect. */
+ i_addr = get_jmp_target_addr(s, which);
+ i_disp = tcg_pcrel_diff(s, (void *)i_addr) - 8;
+ tcg_debug_assert(i_disp < 0);
+ if (i_disp >= -0xfff) {
+ tcg_out_ld32_12(s, COND_AL, TCG_REG_PC, TCG_REG_PC, i_disp);
+ } else {
/*
* The TB is close, but outside the 12 bits addressable by
* the load. We can extend this to 20 bits with a sub of a
- * shifted immediate from pc. In the vastly unlikely event
- * the code requires more than 1MB, we'll use 2 insns and
- * be no worse off.
+ * shifted immediate from pc.
*/
- base = TCG_REG_R0;
- tcg_out_movi32(s, COND_AL, base, ptr - dil);
+ int h = -i_disp;
+ int l = h & 0xfff;
+
+ h = encode_imm_nofail(h - l);
+ tcg_out_dat_imm(s, COND_AL, ARITH_SUB, TCG_REG_R0, TCG_REG_PC, h);
+ tcg_out_ld32_12(s, COND_AL, TCG_REG_PC, TCG_REG_R0, l);
}
- tcg_out_ld32_12(s, COND_AL, TCG_REG_PC, base, dil);
set_jmp_reset_offset(s, which);
}
void tb_target_set_jmp_target(const TranslationBlock *tb, int n,
uintptr_t jmp_rx, uintptr_t jmp_rw)
{
- /* Always indirect, nothing to do */
+ uintptr_t addr = tb->jmp_target_addr[n];
+ ptrdiff_t offset = addr - (jmp_rx + 8);
+ tcg_insn_unit insn;
+
+ /* Either directly branch, or fall through to indirect branch. */
+ if (offset == sextract64(offset, 0, 26)) {
+ /* B <addr> */
+ insn = (COND_AL << 28) | INSN_B | ((offset >> 2) & 0x00ffffff);
+ } else {
+ insn = INSN_NOP;
+ }
+
+ qatomic_set((uint32_t *)jmp_rw, insn);
+ flush_idcache_range(jmp_rx, jmp_rw, 4);
}
static void tcg_out_op(TCGContext *s, TCGOpcode opc,
--
2.34.1
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH 21/22] tcg/riscv: Introduce OPC_NOP
2022-12-06 4:16 [PATCH 00/22] tcg: exit_tb tidy, goto_tb reorg Richard Henderson
` (19 preceding siblings ...)
2022-12-06 4:17 ` [PATCH 20/22] tcg/arm: Implement direct branch for goto_tb Richard Henderson
@ 2022-12-06 4:17 ` Richard Henderson
2022-12-06 7:46 ` Philippe Mathieu-Daudé
2022-12-06 4:17 ` [PATCH 22/22] tcg/riscv: Implement direct branch for goto_tb Richard Henderson
21 siblings, 1 reply; 35+ messages in thread
From: Richard Henderson @ 2022-12-06 4:17 UTC (permalink / raw)
To: qemu-devel; +Cc: r
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
tcg/riscv/tcg-target.c.inc | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/tcg/riscv/tcg-target.c.inc b/tcg/riscv/tcg-target.c.inc
index 37baae9cda..de029d62b4 100644
--- a/tcg/riscv/tcg-target.c.inc
+++ b/tcg/riscv/tcg-target.c.inc
@@ -254,6 +254,7 @@ typedef enum {
#endif
OPC_FENCE = 0x0000000f,
+ OPC_NOP = OPC_ADDI, /* nop = addi r0,r0,0 */
} RISCVInsn;
/*
@@ -390,7 +391,7 @@ static void tcg_out_nop_fill(tcg_insn_unit *p, int count)
{
int i;
for (i = 0; i < count; ++i) {
- p[i] = encode_i(OPC_ADDI, TCG_REG_ZERO, TCG_REG_ZERO, 0);
+ p[i] = OPC_NOP;
}
}
--
2.34.1
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH 22/22] tcg/riscv: Implement direct branch for goto_tb
2022-12-06 4:16 [PATCH 00/22] tcg: exit_tb tidy, goto_tb reorg Richard Henderson
` (20 preceding siblings ...)
2022-12-06 4:17 ` [PATCH 21/22] tcg/riscv: Introduce OPC_NOP Richard Henderson
@ 2022-12-06 4:17 ` Richard Henderson
2022-12-06 7:48 ` Philippe Mathieu-Daudé
21 siblings, 1 reply; 35+ messages in thread
From: Richard Henderson @ 2022-12-06 4:17 UTC (permalink / raw)
To: qemu-devel; +Cc: r
Now that tcg can handle direct and indirect goto_tb simultaneously,
we can optimistically leave space for a direct branch and fall back
to loading the pointer from the TB for an indirect branch.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
tcg/riscv/tcg-target.h | 5 +++++
tcg/riscv/tcg-target.c.inc | 19 +++++++++++++++++--
2 files changed, 22 insertions(+), 2 deletions(-)
diff --git a/tcg/riscv/tcg-target.h b/tcg/riscv/tcg-target.h
index 56f7bc3346..a75c84f2a6 100644
--- a/tcg/riscv/tcg-target.h
+++ b/tcg/riscv/tcg-target.h
@@ -159,6 +159,11 @@ typedef enum {
#define TCG_TARGET_HAS_mulsh_i64 1
#endif
+<<<<<<< HEAD
+=======
+void tb_target_set_jmp_target(uintptr_t, uintptr_t, uintptr_t, uintptr_t);
+
+>>>>>>> 89ab294271 (tcg/riscv: Implement TCG_TARGET_HAS_direct_jump)
#define TCG_TARGET_DEFAULT_MO (0)
#define TCG_TARGET_NEED_LDST_LABELS
diff --git a/tcg/riscv/tcg-target.c.inc b/tcg/riscv/tcg-target.c.inc
index de029d62b4..ee269ea157 100644
--- a/tcg/riscv/tcg-target.c.inc
+++ b/tcg/riscv/tcg-target.c.inc
@@ -1278,7 +1278,11 @@ static void tcg_out_exit_tb(TCGContext *s, uintptr_t a0)
static void tcg_out_goto_tb(TCGContext *s, int which)
{
- /* indirect jump method */
+ /* Direct branch will be patched by tb_target_set_jmp_target. */
+ set_jmp_insn_offset(s, which);
+ tcg_out32(s, OPC_NOP);
+
+ /* When branch is out of range, fall through to indirect. */
tcg_out_ld(s, TCG_TYPE_PTR, TCG_REG_TMP0, TCG_REG_ZERO,
get_jmp_target_addr(s, which));
tcg_out_opc_imm(s, OPC_JALR, TCG_REG_ZERO, TCG_REG_TMP0, 0);
@@ -1288,7 +1292,18 @@ static void tcg_out_goto_tb(TCGContext *s, int which)
void tb_target_set_jmp_target(const TranslationBlock *tb, int n,
uintptr_t jmp_rx, uintptr_t jmp_rw)
{
- /* Always indirect, nothing to do */
+ uintptr_t addr = tb->jmp_target_addr[n];
+ ptrdiff_t offset = addr - jmp_rx;
+ tcg_insn_unit insn;
+
+ /* Either directly branch, or fall through to indirect branch. */
+ if (offset == sextreg(offset, 0, 20)) {
+ insn = encode_uj(OPC_JAL, TCG_REG_ZERO, offset);
+ } else {
+ insn = OPC_NOP;
+ }
+ qatomic_set((uint32_t *)jmp_rw, insn);
+ flush_idcache_range(jmp_rx, jmp_rw, 4);
}
static void tcg_out_op(TCGContext *s, TCGOpcode opc,
--
2.34.1
^ permalink raw reply related [flat|nested] 35+ messages in thread
* Re: [PATCH 01/22] tcg: Split out tcg_out_exit_tb
2022-12-06 4:16 ` [PATCH 01/22] tcg: Split out tcg_out_exit_tb Richard Henderson
@ 2022-12-06 7:24 ` Philippe Mathieu-Daudé
0 siblings, 0 replies; 35+ messages in thread
From: Philippe Mathieu-Daudé @ 2022-12-06 7:24 UTC (permalink / raw)
To: Richard Henderson, qemu-devel; +Cc: r
On 6/12/22 05:16, Richard Henderson wrote:
> The INDEX_op_exit_tb opcode needs no register allocation.
> Split out a dedicated helper function for it.
>
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
> tcg/tcg.c | 4 ++++
> tcg/aarch64/tcg-target.c.inc | 22 ++++++++++--------
> tcg/arm/tcg-target.c.inc | 11 +++++----
> tcg/i386/tcg-target.c.inc | 21 +++++++++--------
> tcg/loongarch64/tcg-target.c.inc | 22 ++++++++++--------
> tcg/mips/tcg-target.c.inc | 33 +++++++++++++--------------
> tcg/ppc/tcg-target.c.inc | 11 +++++----
> tcg/riscv/tcg-target.c.inc | 22 ++++++++++--------
> tcg/s390x/tcg-target.c.inc | 23 ++++++++++---------
> tcg/sparc64/tcg-target.c.inc | 39 +++++++++++++++++---------------
> tcg/tci/tcg-target.c.inc | 10 ++++----
> 11 files changed, 121 insertions(+), 97 deletions(-)
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH 05/22] tcg: Replace asserts on tcg_jmp_insn_offset
2022-12-06 4:16 ` [PATCH 05/22] tcg: Replace asserts on tcg_jmp_insn_offset Richard Henderson
@ 2022-12-06 7:27 ` Philippe Mathieu-Daudé
0 siblings, 0 replies; 35+ messages in thread
From: Philippe Mathieu-Daudé @ 2022-12-06 7:27 UTC (permalink / raw)
To: Richard Henderson, qemu-devel; +Cc: r
On 6/12/22 05:16, Richard Henderson wrote:
> Test TCG_TARGET_HAS_direct_jump instead of testing an
> implementation pointer.
>
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
> tcg/aarch64/tcg-target.c.inc | 2 +-
> tcg/arm/tcg-target.c.inc | 2 +-
> tcg/loongarch64/tcg-target.c.inc | 2 +-
> tcg/mips/tcg-target.c.inc | 2 +-
> tcg/riscv/tcg-target.c.inc | 2 +-
> tcg/s390x/tcg-target.c.inc | 2 +-
> tcg/tci/tcg-target.c.inc | 2 +-
> 7 files changed, 7 insertions(+), 7 deletions(-)
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH 07/22] tcg: Introduce get_jmp_target_addr
2022-12-06 4:17 ` [PATCH 07/22] tcg: Introduce get_jmp_target_addr Richard Henderson
@ 2022-12-06 7:29 ` Philippe Mathieu-Daudé
0 siblings, 0 replies; 35+ messages in thread
From: Philippe Mathieu-Daudé @ 2022-12-06 7:29 UTC (permalink / raw)
To: Richard Henderson, qemu-devel; +Cc: r
On 6/12/22 05:17, Richard Henderson wrote:
> Similar to the existing set_jmp_reset_offset. Include the
> rw->rx address space coversion done by arm and s390x, and
Typo "conversion".
> forgotten by mips and riscv.
>
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
> tcg/tcg.c | 9 +++++++++
> tcg/arm/tcg-target.c.inc | 2 +-
> tcg/mips/tcg-target.c.inc | 2 +-
> tcg/riscv/tcg-target.c.inc | 2 +-
> tcg/s390x/tcg-target.c.inc | 2 +-
> tcg/tci/tcg-target.c.inc | 2 +-
> 6 files changed, 14 insertions(+), 5 deletions(-)
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH 08/22] tcg: Split out tcg_out_goto_tb
2022-12-06 4:17 ` [PATCH 08/22] tcg: Split out tcg_out_goto_tb Richard Henderson
@ 2022-12-06 7:36 ` Philippe Mathieu-Daudé
0 siblings, 0 replies; 35+ messages in thread
From: Philippe Mathieu-Daudé @ 2022-12-06 7:36 UTC (permalink / raw)
To: Richard Henderson, qemu-devel; +Cc: r
On 6/12/22 05:17, Richard Henderson wrote:
> The INDEX_op_goto_tb opcode needs no register allocation.
> Split out a dedicated helper function for it.
>
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
> tcg/tcg.c | 4 ++
> tcg/aarch64/tcg-target.c.inc | 40 +++++++++---------
> tcg/arm/tcg-target.c.inc | 49 +++++++++++-----------
> tcg/i386/tcg-target.c.inc | 33 +++++++--------
> tcg/loongarch64/tcg-target.c.inc | 38 +++++++++--------
> tcg/mips/tcg-target.c.inc | 21 ++++++----
> tcg/ppc/tcg-target.c.inc | 52 ++++++++++++------------
> tcg/riscv/tcg-target.c.inc | 20 +++++----
> tcg/s390x/tcg-target.c.inc | 70 ++++++++++++++++----------------
> tcg/sparc64/tcg-target.c.inc | 68 ++++++++++++++++---------------
> tcg/tci/tcg-target.c.inc | 16 ++++----
> 11 files changed, 219 insertions(+), 192 deletions(-)
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH 09/22] tcg: Rename TB_JMP_RESET_OFFSET_INVALID to TB_JMP_OFFSET_INVALID
2022-12-06 4:17 ` [PATCH 09/22] tcg: Rename TB_JMP_RESET_OFFSET_INVALID to TB_JMP_OFFSET_INVALID Richard Henderson
@ 2022-12-06 7:36 ` Philippe Mathieu-Daudé
0 siblings, 0 replies; 35+ messages in thread
From: Philippe Mathieu-Daudé @ 2022-12-06 7:36 UTC (permalink / raw)
To: Richard Henderson, qemu-devel; +Cc: r
On 6/12/22 05:17, Richard Henderson wrote:
> This will shortly be used for more than reset.
>
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
> include/exec/exec-all.h | 2 +-
> accel/tcg/translate-all.c | 8 ++++----
> tcg/tcg.c | 4 ++--
> 3 files changed, 7 insertions(+), 7 deletions(-)
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH 13/22] tcg: Move tb_target_set_jmp_target declaration to tcg.h
2022-12-06 4:17 ` [PATCH 13/22] tcg: Move tb_target_set_jmp_target declaration to tcg.h Richard Henderson
@ 2022-12-06 7:38 ` Philippe Mathieu-Daudé
0 siblings, 0 replies; 35+ messages in thread
From: Philippe Mathieu-Daudé @ 2022-12-06 7:38 UTC (permalink / raw)
To: Richard Henderson, qemu-devel; +Cc: r
On 6/12/22 05:17, Richard Henderson wrote:
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
> include/tcg/tcg.h | 3 +++
> tcg/aarch64/tcg-target.h | 4 ----
> tcg/arm/tcg-target.h | 5 -----
> tcg/i386/tcg-target.h | 3 ---
> tcg/loongarch64/tcg-target.h | 3 ---
> tcg/mips/tcg-target.h | 5 -----
> tcg/ppc/tcg-target.h | 4 ----
> tcg/riscv/tcg-target.h | 4 ----
> tcg/s390x/tcg-target.h | 4 ----
> tcg/sparc64/tcg-target.h | 4 ----
> tcg/tci/tcg-target.h | 4 ----
> 11 files changed, 3 insertions(+), 40 deletions(-)
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH 14/22] tcg: Always define tb_target_set_jmp_target
2022-12-06 4:17 ` [PATCH 14/22] tcg: Always define tb_target_set_jmp_target Richard Henderson
@ 2022-12-06 7:39 ` Philippe Mathieu-Daudé
0 siblings, 0 replies; 35+ messages in thread
From: Philippe Mathieu-Daudé @ 2022-12-06 7:39 UTC (permalink / raw)
To: Richard Henderson, qemu-devel; +Cc: r
On 6/12/22 05:17, Richard Henderson wrote:
> Install empty versions for !TCG_TARGET_HAS_direct_jump hosts.
>
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
> tcg/arm/tcg-target.c.inc | 6 ++++++
> tcg/mips/tcg-target.c.inc | 6 ++++++
> tcg/riscv/tcg-target.c.inc | 6 ++++++
> tcg/tci/tcg-target.c.inc | 6 ++++++
> 4 files changed, 24 insertions(+)
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH 18/22] tcg/sparc64: Remove USE_REG_TB
2022-12-06 4:17 ` [PATCH 18/22] tcg/sparc64: Remove USE_REG_TB Richard Henderson
@ 2022-12-06 7:44 ` Philippe Mathieu-Daudé
2022-12-06 15:40 ` Richard Henderson
0 siblings, 1 reply; 35+ messages in thread
From: Philippe Mathieu-Daudé @ 2022-12-06 7:44 UTC (permalink / raw)
To: Richard Henderson, qemu-devel; +Cc: r
On 6/12/22 05:17, Richard Henderson wrote:
> This is always true for sparc64, so this is dead since 3a5f6805c7ca.
>
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
> tcg/sparc64/tcg-target.c.inc | 57 ++++++++++++++----------------------
> 1 file changed, 22 insertions(+), 35 deletions(-)
> @@ -1897,7 +1884,7 @@ void tb_target_set_jmp_target(const TranslationBlock *tb, int n,
> tcg_debug_assert(tb_disp == (int32_t)tb_disp);
> tcg_debug_assert(br_disp == (int32_t)br_disp);
>
> - if (!USE_REG_TB) {
> + if (0) {
> qatomic_set((uint32_t *)jmp_rw,
> deposit32(CALL, 0, 30, br_disp >> 2));
> flush_idcache_range(jmp_rx, jmp_rw, 4);
Why remove in the next patch and not here?
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH 21/22] tcg/riscv: Introduce OPC_NOP
2022-12-06 4:17 ` [PATCH 21/22] tcg/riscv: Introduce OPC_NOP Richard Henderson
@ 2022-12-06 7:46 ` Philippe Mathieu-Daudé
0 siblings, 0 replies; 35+ messages in thread
From: Philippe Mathieu-Daudé @ 2022-12-06 7:46 UTC (permalink / raw)
To: Richard Henderson, qemu-devel; +Cc: r
On 6/12/22 05:17, Richard Henderson wrote:
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
> tcg/riscv/tcg-target.c.inc | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH 22/22] tcg/riscv: Implement direct branch for goto_tb
2022-12-06 4:17 ` [PATCH 22/22] tcg/riscv: Implement direct branch for goto_tb Richard Henderson
@ 2022-12-06 7:48 ` Philippe Mathieu-Daudé
2022-12-06 15:41 ` Richard Henderson
0 siblings, 1 reply; 35+ messages in thread
From: Philippe Mathieu-Daudé @ 2022-12-06 7:48 UTC (permalink / raw)
To: Richard Henderson, qemu-devel; +Cc: r
On 6/12/22 05:17, Richard Henderson wrote:
> Now that tcg can handle direct and indirect goto_tb simultaneously,
> we can optimistically leave space for a direct branch and fall back
> to loading the pointer from the TB for an indirect branch.
>
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
> tcg/riscv/tcg-target.h | 5 +++++
> tcg/riscv/tcg-target.c.inc | 19 +++++++++++++++++--
> 2 files changed, 22 insertions(+), 2 deletions(-)
>
> diff --git a/tcg/riscv/tcg-target.h b/tcg/riscv/tcg-target.h
> index 56f7bc3346..a75c84f2a6 100644
> --- a/tcg/riscv/tcg-target.h
> +++ b/tcg/riscv/tcg-target.h
> @@ -159,6 +159,11 @@ typedef enum {
> #define TCG_TARGET_HAS_mulsh_i64 1
> #endif
>
> +<<<<<<< HEAD
> +=======
> +void tb_target_set_jmp_target(uintptr_t, uintptr_t, uintptr_t, uintptr_t);
> +
> +>>>>>>> 89ab294271 (tcg/riscv: Implement TCG_TARGET_HAS_direct_jump)
HEAD is correct :)
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH 18/22] tcg/sparc64: Remove USE_REG_TB
2022-12-06 7:44 ` Philippe Mathieu-Daudé
@ 2022-12-06 15:40 ` Richard Henderson
0 siblings, 0 replies; 35+ messages in thread
From: Richard Henderson @ 2022-12-06 15:40 UTC (permalink / raw)
To: Philippe Mathieu-Daudé, qemu-devel; +Cc: r
On 12/6/22 01:44, Philippe Mathieu-Daudé wrote:
> On 6/12/22 05:17, Richard Henderson wrote:
>> This is always true for sparc64, so this is dead since 3a5f6805c7ca.
>>
>> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
>> ---
>> tcg/sparc64/tcg-target.c.inc | 57 ++++++++++++++----------------------
>> 1 file changed, 22 insertions(+), 35 deletions(-)
>
>> @@ -1897,7 +1884,7 @@ void tb_target_set_jmp_target(const TranslationBlock *tb, int n,
>> tcg_debug_assert(tb_disp == (int32_t)tb_disp);
>> tcg_debug_assert(br_disp == (int32_t)br_disp);
>> - if (!USE_REG_TB) {
>> + if (0) {
>> qatomic_set((uint32_t *)jmp_rw,
>> deposit32(CALL, 0, 30, br_disp >> 2));
>> flush_idcache_range(jmp_rx, jmp_rw, 4);
>
> Why remove in the next patch and not here?
Heh. I did that so I could move this code in the next patch.
I meant to go back and edit this patch to delete, after I'd done that.
r~
>
> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH 22/22] tcg/riscv: Implement direct branch for goto_tb
2022-12-06 7:48 ` Philippe Mathieu-Daudé
@ 2022-12-06 15:41 ` Richard Henderson
0 siblings, 0 replies; 35+ messages in thread
From: Richard Henderson @ 2022-12-06 15:41 UTC (permalink / raw)
To: Philippe Mathieu-Daudé, qemu-devel; +Cc: r
On 12/6/22 01:48, Philippe Mathieu-Daudé wrote:
> On 6/12/22 05:17, Richard Henderson wrote:
>> Now that tcg can handle direct and indirect goto_tb simultaneously,
>> we can optimistically leave space for a direct branch and fall back
>> to loading the pointer from the TB for an indirect branch.
>>
>> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
>> ---
>> tcg/riscv/tcg-target.h | 5 +++++
>> tcg/riscv/tcg-target.c.inc | 19 +++++++++++++++++--
>> 2 files changed, 22 insertions(+), 2 deletions(-)
>>
>> diff --git a/tcg/riscv/tcg-target.h b/tcg/riscv/tcg-target.h
>> index 56f7bc3346..a75c84f2a6 100644
>> --- a/tcg/riscv/tcg-target.h
>> +++ b/tcg/riscv/tcg-target.h
>> @@ -159,6 +159,11 @@ typedef enum {
>> #define TCG_TARGET_HAS_mulsh_i64 1
>> #endif
>> +<<<<<<< HEAD
>> +=======
>> +void tb_target_set_jmp_target(uintptr_t, uintptr_t, uintptr_t, uintptr_t);
>> +
>> +>>>>>>> 89ab294271 (tcg/riscv: Implement TCG_TARGET_HAS_direct_jump)
>
> HEAD is correct :)
Ouch. Clearly didn't get the fixed branch pushed back.
I wonder what else I missed...
r~
^ permalink raw reply [flat|nested] 35+ messages in thread
end of thread, other threads:[~2022-12-06 15:42 UTC | newest]
Thread overview: 35+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-12-06 4:16 [PATCH 00/22] tcg: exit_tb tidy, goto_tb reorg Richard Henderson
2022-12-06 4:16 ` [PATCH 01/22] tcg: Split out tcg_out_exit_tb Richard Henderson
2022-12-06 7:24 ` Philippe Mathieu-Daudé
2022-12-06 4:16 ` [PATCH 02/22] tcg/i386: Remove unused goto_tb code for indirect jump Richard Henderson
2022-12-06 4:16 ` [PATCH 03/22] tcg/ppc: " Richard Henderson
2022-12-06 4:16 ` [PATCH 04/22] tcg/sparc64: " Richard Henderson
2022-12-06 4:16 ` [PATCH 05/22] tcg: Replace asserts on tcg_jmp_insn_offset Richard Henderson
2022-12-06 7:27 ` Philippe Mathieu-Daudé
2022-12-06 4:16 ` [PATCH 06/22] tcg: Introduce set_jmp_insn_offset Richard Henderson
2022-12-06 4:17 ` [PATCH 07/22] tcg: Introduce get_jmp_target_addr Richard Henderson
2022-12-06 7:29 ` Philippe Mathieu-Daudé
2022-12-06 4:17 ` [PATCH 08/22] tcg: Split out tcg_out_goto_tb Richard Henderson
2022-12-06 7:36 ` Philippe Mathieu-Daudé
2022-12-06 4:17 ` [PATCH 09/22] tcg: Rename TB_JMP_RESET_OFFSET_INVALID to TB_JMP_OFFSET_INVALID Richard Henderson
2022-12-06 7:36 ` Philippe Mathieu-Daudé
2022-12-06 4:17 ` [PATCH 10/22] tcg: Add gen_tb to TCGContext Richard Henderson
2022-12-06 4:17 ` [PATCH 11/22] tcg: Add TranslationBlock.jmp_insn_offset Richard Henderson
2022-12-06 4:17 ` [PATCH 12/22] tcg: Change tb_target_set_jmp_target arguments Richard Henderson
2022-12-06 4:17 ` [PATCH 13/22] tcg: Move tb_target_set_jmp_target declaration to tcg.h Richard Henderson
2022-12-06 7:38 ` Philippe Mathieu-Daudé
2022-12-06 4:17 ` [PATCH 14/22] tcg: Always define tb_target_set_jmp_target Richard Henderson
2022-12-06 7:39 ` Philippe Mathieu-Daudé
2022-12-06 4:17 ` [PATCH 15/22] tcg: Remove TCG_TARGET_HAS_direct_jump Richard Henderson
2022-12-06 4:17 ` [PATCH 16/22] tcg/aarch64: Reorg goto_tb implementation Richard Henderson
2022-12-06 4:17 ` [PATCH 17/22] tcg/ppc: " Richard Henderson
2022-12-06 4:17 ` [PATCH 18/22] tcg/sparc64: Remove USE_REG_TB Richard Henderson
2022-12-06 7:44 ` Philippe Mathieu-Daudé
2022-12-06 15:40 ` Richard Henderson
2022-12-06 4:17 ` [PATCH 19/22] tcg/sparc64: Reorg goto_tb implementation Richard Henderson
2022-12-06 4:17 ` [PATCH 20/22] tcg/arm: Implement direct branch for goto_tb Richard Henderson
2022-12-06 4:17 ` [PATCH 21/22] tcg/riscv: Introduce OPC_NOP Richard Henderson
2022-12-06 7:46 ` Philippe Mathieu-Daudé
2022-12-06 4:17 ` [PATCH 22/22] tcg/riscv: Implement direct branch for goto_tb Richard Henderson
2022-12-06 7:48 ` Philippe Mathieu-Daudé
2022-12-06 15:41 ` Richard Henderson
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.