[PATCH V3 2/2] arm64: bpf: make BPF prologue and epilogue align with ARM64 AAPCS

[PATCH V3 2/2] arm64: bpf: make BPF prologue and epilogue align with ARM64 AAPCS

[Yang Shi]

Save and restore FP/LR in BPF prog prologue and epilogue, save SP to FP
in prologue in order to get the correct stack backtrace.

However, ARM64 JIT used FP (x29) as eBPF fp register, FP is subjected to
change during function call so it may cause the BPF prog stack base address
change too.

Use x25 to replace FP as BPF stack base register (fp). Since x25 is callee
saved register, so it will keep intact during function call.
It is initialized in BPF prog prologue when BPF prog is started to run
everytime. When BPF prog exits, it could be just tossed.

So, the BPF stack layout looks like:

                                 high
         original A64_SP =>   0:+-----+ BPF prologue
                                |FP/LR|
         current A64_FP =>  -16:+-----+
                                | ... | callee saved registers
         BPF fp register => -64:+-----+
                                |     |
                                | ... | BPF prog stack
                                |     |
                                |     |
         current A64_SP =>      +-----+
                                |     |
                                | ... | Function call stack
                                |     |
                                +-----+
                                  low

CC: Zi Shen Lim <zlim.lnx@gmail.com>
CC: Xi Wang <xi.wang@gmail.com>
Signed-off-by: Yang Shi <yang.shi@linaro.org>
---
V3 --> V2:
* Make FP point to FP'
* Fix a compile warning

 arch/arm64/net/bpf_jit_comp.c | 37 +++++++++++++++++++++++++++++++------
 1 file changed, 31 insertions(+), 6 deletions(-)

diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index ac8b548..c131e38 100644
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -50,7 +50,7 @@ static const int bpf2a64[] = {
 	[BPF_REG_8] = A64_R(21),
 	[BPF_REG_9] = A64_R(22),
 	/* read-only frame pointer to access stack */
-	[BPF_REG_FP] = A64_FP,
+	[BPF_REG_FP] = A64_R(25),
 	/* temporary register for internal BPF JIT */
 	[TMP_REG_1] = A64_R(23),
 	[TMP_REG_2] = A64_R(24),
@@ -155,16 +155,42 @@ static void build_prologue(struct jit_ctx *ctx)
 	stack_size += 4; /* extra for skb_copy_bits buffer */
 	stack_size = STACK_ALIGN(stack_size);
 
+	/*
+	 * BPF prog stack layout
+	 *
+	 *                         high
+	 * original A64_SP =>   0:+-----+ BPF prologue
+	 *                        |FP/LR|
+	 * current A64_FP =>  -16:+-----+
+	 *                        | ... | callee saved registers
+	 * BPF fp register => -64:+-----+
+	 *                        |     |
+	 *                        | ... | BPF prog stack
+	 *                        |     |
+	 *                        |     |
+	 * current A64_SP =>      +-----+
+	 *                        |     |
+	 *                        | ... | Function call stack
+	 *                        |     |
+	 *                        +-----+
+	 *                          low
+	 *
+	 */
+
+	/* Save FP and LR registers to stay align with ARM64 AAPCS */
+	emit(A64_PUSH(A64_FP, A64_LR, A64_SP), ctx);
+	emit(A64_MOV(1, A64_FP, A64_SP), ctx);
+
 	/* Save callee-saved register */
 	emit(A64_PUSH(r6, r7, A64_SP), ctx);
 	emit(A64_PUSH(r8, r9, A64_SP), ctx);
 	if (ctx->tmp_used)
 		emit(A64_PUSH(tmp1, tmp2, A64_SP), ctx);
 
-	/* Set up frame pointer */
+	/* Set up BPF prog stack base register (x25) */
 	emit(A64_MOV(1, fp, A64_SP), ctx);
 
-	/* Set up BPF stack */
+	/* Set up function call stack */
 	emit(A64_SUB_I(1, A64_SP, A64_SP, stack_size), ctx);
 
 	/* Clear registers A and X */
@@ -179,7 +205,6 @@ static void build_epilogue(struct jit_ctx *ctx)
 	const u8 r7 = bpf2a64[BPF_REG_7];
 	const u8 r8 = bpf2a64[BPF_REG_8];
 	const u8 r9 = bpf2a64[BPF_REG_9];
-	const u8 fp = bpf2a64[BPF_REG_FP];
 	const u8 tmp1 = bpf2a64[TMP_REG_1];
 	const u8 tmp2 = bpf2a64[TMP_REG_2];
 	int stack_size = MAX_BPF_STACK;
@@ -196,8 +221,8 @@ static void build_epilogue(struct jit_ctx *ctx)
 	emit(A64_POP(r8, r9, A64_SP), ctx);
 	emit(A64_POP(r6, r7, A64_SP), ctx);
 
-	/* Restore frame pointer */
-	emit(A64_MOV(1, fp, A64_SP), ctx);
+	/* Restore FP/LR registers */
+	emit(A64_POP(A64_FP, A64_LR, A64_SP), ctx);
 
 	/* Set return value */
 	emit(A64_MOV(1, A64_R(0), r0), ctx);
-- 
2.0.2

[PATCH V3 2/2] arm64: bpf: make BPF prologue and epilogue align with ARM64 AAPCS

[Yang Shi]

Save and restore FP/LR in BPF prog prologue and epilogue, save SP to FP
in prologue in order to get the correct stack backtrace.

However, ARM64 JIT used FP (x29) as eBPF fp register, FP is subjected to
change during function call so it may cause the BPF prog stack base address
change too.

Use x25 to replace FP as BPF stack base register (fp). Since x25 is callee
saved register, so it will keep intact during function call.
It is initialized in BPF prog prologue when BPF prog is started to run
everytime. When BPF prog exits, it could be just tossed.

So, the BPF stack layout looks like:

                                 high
         original A64_SP =>   0:+-----+ BPF prologue
                                |FP/LR|
         current A64_FP =>  -16:+-----+
                                | ... | callee saved registers
         BPF fp register => -64:+-----+
                                |     |
                                | ... | BPF prog stack
                                |     |
                                |     |
         current A64_SP =>      +-----+
                                |     |
                                | ... | Function call stack
                                |     |
                                +-----+
                                  low

CC: Zi Shen Lim <zlim.lnx@gmail.com>
CC: Xi Wang <xi.wang@gmail.com>
Signed-off-by: Yang Shi <yang.shi@linaro.org>
---
V3 --> V2:
* Make FP point to FP'
* Fix a compile warning

 arch/arm64/net/bpf_jit_comp.c | 37 +++++++++++++++++++++++++++++++------
 1 file changed, 31 insertions(+), 6 deletions(-)

diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index ac8b548..c131e38 100644
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -50,7 +50,7 @@ static const int bpf2a64[] = {
 	[BPF_REG_8] = A64_R(21),
 	[BPF_REG_9] = A64_R(22),
 	/* read-only frame pointer to access stack */
-	[BPF_REG_FP] = A64_FP,
+	[BPF_REG_FP] = A64_R(25),
 	/* temporary register for internal BPF JIT */
 	[TMP_REG_1] = A64_R(23),
 	[TMP_REG_2] = A64_R(24),
@@ -155,16 +155,42 @@ static void build_prologue(struct jit_ctx *ctx)
 	stack_size += 4; /* extra for skb_copy_bits buffer */
 	stack_size = STACK_ALIGN(stack_size);
 
+	/*
+	 * BPF prog stack layout
+	 *
+	 *                         high
+	 * original A64_SP =>   0:+-----+ BPF prologue
+	 *                        |FP/LR|
+	 * current A64_FP =>  -16:+-----+
+	 *                        | ... | callee saved registers
+	 * BPF fp register => -64:+-----+
+	 *                        |     |
+	 *                        | ... | BPF prog stack
+	 *                        |     |
+	 *                        |     |
+	 * current A64_SP =>      +-----+
+	 *                        |     |
+	 *                        | ... | Function call stack
+	 *                        |     |
+	 *                        +-----+
+	 *                          low
+	 *
+	 */
+
+	/* Save FP and LR registers to stay align with ARM64 AAPCS */
+	emit(A64_PUSH(A64_FP, A64_LR, A64_SP), ctx);
+	emit(A64_MOV(1, A64_FP, A64_SP), ctx);
+
 	/* Save callee-saved register */
 	emit(A64_PUSH(r6, r7, A64_SP), ctx);
 	emit(A64_PUSH(r8, r9, A64_SP), ctx);
 	if (ctx->tmp_used)
 		emit(A64_PUSH(tmp1, tmp2, A64_SP), ctx);
 
-	/* Set up frame pointer */
+	/* Set up BPF prog stack base register (x25) */
 	emit(A64_MOV(1, fp, A64_SP), ctx);
 
-	/* Set up BPF stack */
+	/* Set up function call stack */
 	emit(A64_SUB_I(1, A64_SP, A64_SP, stack_size), ctx);
 
 	/* Clear registers A and X */
@@ -179,7 +205,6 @@ static void build_epilogue(struct jit_ctx *ctx)
 	const u8 r7 = bpf2a64[BPF_REG_7];
 	const u8 r8 = bpf2a64[BPF_REG_8];
 	const u8 r9 = bpf2a64[BPF_REG_9];
-	const u8 fp = bpf2a64[BPF_REG_FP];
 	const u8 tmp1 = bpf2a64[TMP_REG_1];
 	const u8 tmp2 = bpf2a64[TMP_REG_2];
 	int stack_size = MAX_BPF_STACK;
@@ -196,8 +221,8 @@ static void build_epilogue(struct jit_ctx *ctx)
 	emit(A64_POP(r8, r9, A64_SP), ctx);
 	emit(A64_POP(r6, r7, A64_SP), ctx);
 
-	/* Restore frame pointer */
-	emit(A64_MOV(1, fp, A64_SP), ctx);
+	/* Restore FP/LR registers */
+	emit(A64_POP(A64_FP, A64_LR, A64_SP), ctx);
 
 	/* Set return value */
 	emit(A64_MOV(1, A64_R(0), r0), ctx);
-- 
2.0.2

Re: [PATCH V3 2/2] arm64: bpf: make BPF prologue and epilogue align with ARM64 AAPCS

[Z Lim]

Yang, I noticed another thing...

On Fri, Nov 13, 2015 at 10:09 AM, Yang Shi <yang.shi@linaro.org> wrote:
> Save and restore FP/LR in BPF prog prologue and epilogue, save SP to FP
> in prologue in order to get the correct stack backtrace.
>
> However, ARM64 JIT used FP (x29) as eBPF fp register, FP is subjected to
> change during function call so it may cause the BPF prog stack base address
> change too.
>
> Use x25 to replace FP as BPF stack base register (fp). Since x25 is callee
> saved register, so it will keep intact during function call.

Can you please add save/restore for x25 also? :)

> It is initialized in BPF prog prologue when BPF prog is started to run
> everytime. When BPF prog exits, it could be just tossed.
>

Re: [PATCH V3 2/2] arm64: bpf: make BPF prologue and epilogue align with ARM64 AAPCS

[Z Lim]

Yang, I noticed another thing...

On Fri, Nov 13, 2015 at 10:09 AM, Yang Shi <yang.shi@linaro.org> wrote:
> Save and restore FP/LR in BPF prog prologue and epilogue, save SP to FP
> in prologue in order to get the correct stack backtrace.
>
> However, ARM64 JIT used FP (x29) as eBPF fp register, FP is subjected to
> change during function call so it may cause the BPF prog stack base address
> change too.
>
> Use x25 to replace FP as BPF stack base register (fp). Since x25 is callee
> saved register, so it will keep intact during function call.

Can you please add save/restore for x25 also? :)

> It is initialized in BPF prog prologue when BPF prog is started to run
> everytime. When BPF prog exits, it could be just tossed.
>

[PATCH V3 2/2] arm64: bpf: make BPF prologue and epilogue align with ARM64 AAPCS

[Z Lim]

Yang, I noticed another thing...

On Fri, Nov 13, 2015 at 10:09 AM, Yang Shi <yang.shi@linaro.org> wrote:
> Save and restore FP/LR in BPF prog prologue and epilogue, save SP to FP
> in prologue in order to get the correct stack backtrace.
>
> However, ARM64 JIT used FP (x29) as eBPF fp register, FP is subjected to
> change during function call so it may cause the BPF prog stack base address
> change too.
>
> Use x25 to replace FP as BPF stack base register (fp). Since x25 is callee
> saved register, so it will keep intact during function call.

Can you please add save/restore for x25 also? :)

> It is initialized in BPF prog prologue when BPF prog is started to run
> everytime. When BPF prog exits, it could be just tossed.
>

Re: [PATCH V3 2/2] arm64: bpf: make BPF prologue and epilogue align with ARM64 AAPCS

[Shi, Yang]

On 11/13/2015 6:39 PM, Z Lim wrote:
> Yang, I noticed another thing...
>
> On Fri, Nov 13, 2015 at 10:09 AM, Yang Shi <yang.shi@linaro.org> wrote:
>> Save and restore FP/LR in BPF prog prologue and epilogue, save SP to FP
>> in prologue in order to get the correct stack backtrace.
>>
>> However, ARM64 JIT used FP (x29) as eBPF fp register, FP is subjected to
>> change during function call so it may cause the BPF prog stack base address
>> change too.
>>
>> Use x25 to replace FP as BPF stack base register (fp). Since x25 is callee
>> saved register, so it will keep intact during function call.
>
> Can you please add save/restore for x25 also? :)

Sure. BTW, since PUSH invokes stp instruction and SP need 16-bytes 
alignment, so we have to save x26 with x25 together. Anyway, it won't 
introduce any harm overhead since one instruction saves two registers.

Yang

>
>> It is initialized in BPF prog prologue when BPF prog is started to run
>> everytime. When BPF prog exits, it could be just tossed.
>>

Re: [PATCH V3 2/2] arm64: bpf: make BPF prologue and epilogue align with ARM64 AAPCS

[Shi, Yang]

On 11/13/2015 6:39 PM, Z Lim wrote:
> Yang, I noticed another thing...
>
> On Fri, Nov 13, 2015 at 10:09 AM, Yang Shi <yang.shi@linaro.org> wrote:
>> Save and restore FP/LR in BPF prog prologue and epilogue, save SP to FP
>> in prologue in order to get the correct stack backtrace.
>>
>> However, ARM64 JIT used FP (x29) as eBPF fp register, FP is subjected to
>> change during function call so it may cause the BPF prog stack base address
>> change too.
>>
>> Use x25 to replace FP as BPF stack base register (fp). Since x25 is callee
>> saved register, so it will keep intact during function call.
>
> Can you please add save/restore for x25 also? :)

Sure. BTW, since PUSH invokes stp instruction and SP need 16-bytes 
alignment, so we have to save x26 with x25 together. Anyway, it won't 
introduce any harm overhead since one instruction saves two registers.

Yang

>
>> It is initialized in BPF prog prologue when BPF prog is started to run
>> everytime. When BPF prog exits, it could be just tossed.
>>

[PATCH V3 2/2] arm64: bpf: make BPF prologue and epilogue align with ARM64 AAPCS

[Shi, Yang]

On 11/13/2015 6:39 PM, Z Lim wrote:
> Yang, I noticed another thing...
>
> On Fri, Nov 13, 2015 at 10:09 AM, Yang Shi <yang.shi@linaro.org> wrote:
>> Save and restore FP/LR in BPF prog prologue and epilogue, save SP to FP
>> in prologue in order to get the correct stack backtrace.
>>
>> However, ARM64 JIT used FP (x29) as eBPF fp register, FP is subjected to
>> change during function call so it may cause the BPF prog stack base address
>> change too.
>>
>> Use x25 to replace FP as BPF stack base register (fp). Since x25 is callee
>> saved register, so it will keep intact during function call.
>
> Can you please add save/restore for x25 also? :)

Sure. BTW, since PUSH invokes stp instruction and SP need 16-bytes 
alignment, so we have to save x26 with x25 together. Anyway, it won't 
introduce any harm overhead since one instruction saves two registers.

Yang

>
>> It is initialized in BPF prog prologue when BPF prog is started to run
>> everytime. When BPF prog exits, it could be just tossed.
>>

[PATCH V4 2/2] arm64: bpf: make BPF prologue and epilogue align with ARM64 AAPCS

[Yang Shi]

Save and restore FP/LR in BPF prog prologue and epilogue, save SP to FP
in prologue in order to get the correct stack backtrace.

However, ARM64 JIT used FP (x29) as eBPF fp register, FP is subjected to
change during function call so it may cause the BPF prog stack base address
change too.

Use x25 to replace FP as BPF stack base register (fp). Since x25 is callee
saved register, so it will keep intact during function call.
It is initialized in BPF prog prologue when BPF prog is started to run
everytime. Save and restore x25/x26 in BPF prologue and epilogue to keep
them intact for the outside of BPF. Actually, x26 is unnecessary, but SP
requires 16 bytes alignment.

So, the BPF stack layout looks like:

                                 high
         original A64_SP =>   0:+-----+ BPF prologue
                                |FP/LR|
         current A64_FP =>  -16:+-----+
                                | ... | callee saved registers
                                +-----+
                                |     | x25/x26
         BPF fp register => -80:+-----+
                                |     |
                                | ... | BPF prog stack
                                |     |
                                |     |
         current A64_SP =>      +-----+
                                |     |
                                | ... | Function call stack
                                |     |
                                +-----+
                                  low

CC: Zi Shen Lim <zlim.lnx@gmail.com>
CC: Xi Wang <xi.wang@gmail.com>
Signed-off-by: Yang Shi <yang.shi@linaro.org>
---
V4 --> V3:
* Save/restore x25 and x26

V3 --> V2:
* Make FP point to FP'
* Fix a compile warning

 arch/arm64/net/bpf_jit_comp.c | 44 ++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 39 insertions(+), 5 deletions(-)

diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index ac8b548..86a3253 100644
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -50,7 +50,7 @@ static const int bpf2a64[] = {
 	[BPF_REG_8] = A64_R(21),
 	[BPF_REG_9] = A64_R(22),
 	/* read-only frame pointer to access stack */
-	[BPF_REG_FP] = A64_FP,
+	[BPF_REG_FP] = A64_R(25),
 	/* temporary register for internal BPF JIT */
 	[TMP_REG_1] = A64_R(23),
 	[TMP_REG_2] = A64_R(24),
@@ -155,16 +155,47 @@ static void build_prologue(struct jit_ctx *ctx)
 	stack_size += 4; /* extra for skb_copy_bits buffer */
 	stack_size = STACK_ALIGN(stack_size);
 
+	/*
+	 * BPF prog stack layout
+	 *
+	 *                         high
+	 * original A64_SP =>   0:+-----+ BPF prologue
+	 *                        |FP/LR|
+	 * current A64_FP =>  -16:+-----+
+	 *                        | ... | callee saved registers
+	 *                        +-----+
+	 *                        |     | x25/x26
+	 * BPF fp register => -80:+-----+
+	 *                        |     |
+	 *                        | ... | BPF prog stack
+	 *                        |     |
+	 *                        |     |
+	 * current A64_SP =>      +-----+
+	 *                        |     |
+	 *                        | ... | Function call stack
+	 *                        |     |
+	 *                        +-----+
+	 *                          low
+	 *
+	 */
+
+	/* Save FP and LR registers to stay align with ARM64 AAPCS */
+	emit(A64_PUSH(A64_FP, A64_LR, A64_SP), ctx);
+	emit(A64_MOV(1, A64_FP, A64_SP), ctx);
+
 	/* Save callee-saved register */
 	emit(A64_PUSH(r6, r7, A64_SP), ctx);
 	emit(A64_PUSH(r8, r9, A64_SP), ctx);
 	if (ctx->tmp_used)
 		emit(A64_PUSH(tmp1, tmp2, A64_SP), ctx);
 
-	/* Set up frame pointer */
+	/* Save fp (x25) and x26. SP requires 16 bytes alignment */
+	emit(A64_PUSH(fp, A64_R(26), A64_SP), ctx);
+
+	/* Set up BPF prog stack base register (x25) */
 	emit(A64_MOV(1, fp, A64_SP), ctx);
 
-	/* Set up BPF stack */
+	/* Set up function call stack */
 	emit(A64_SUB_I(1, A64_SP, A64_SP, stack_size), ctx);
 
 	/* Clear registers A and X */
@@ -190,14 +221,17 @@ static void build_epilogue(struct jit_ctx *ctx)
 	/* We're done with BPF stack */
 	emit(A64_ADD_I(1, A64_SP, A64_SP, stack_size), ctx);
 
+	/* Restore fs (x25) and x26 */
+	emit(A64_POP(fp, A64_R(26), A64_SP), ctx);
+
 	/* Restore callee-saved register */
 	if (ctx->tmp_used)
 		emit(A64_POP(tmp1, tmp2, A64_SP), ctx);
 	emit(A64_POP(r8, r9, A64_SP), ctx);
 	emit(A64_POP(r6, r7, A64_SP), ctx);
 
-	/* Restore frame pointer */
-	emit(A64_MOV(1, fp, A64_SP), ctx);
+	/* Restore FP/LR registers */
+	emit(A64_POP(A64_FP, A64_LR, A64_SP), ctx);
 
 	/* Set return value */
 	emit(A64_MOV(1, A64_R(0), r0), ctx);
-- 
2.0.2

[PATCH V4 2/2] arm64: bpf: make BPF prologue and epilogue align with ARM64 AAPCS

[Yang Shi]

Save and restore FP/LR in BPF prog prologue and epilogue, save SP to FP
in prologue in order to get the correct stack backtrace.

However, ARM64 JIT used FP (x29) as eBPF fp register, FP is subjected to
change during function call so it may cause the BPF prog stack base address
change too.

Use x25 to replace FP as BPF stack base register (fp). Since x25 is callee
saved register, so it will keep intact during function call.
It is initialized in BPF prog prologue when BPF prog is started to run
everytime. Save and restore x25/x26 in BPF prologue and epilogue to keep
them intact for the outside of BPF. Actually, x26 is unnecessary, but SP
requires 16 bytes alignment.

So, the BPF stack layout looks like:

                                 high
         original A64_SP =>   0:+-----+ BPF prologue
                                |FP/LR|
         current A64_FP =>  -16:+-----+
                                | ... | callee saved registers
                                +-----+
                                |     | x25/x26
         BPF fp register => -80:+-----+
                                |     |
                                | ... | BPF prog stack
                                |     |
                                |     |
         current A64_SP =>      +-----+
                                |     |
                                | ... | Function call stack
                                |     |
                                +-----+
                                  low

CC: Zi Shen Lim <zlim.lnx@gmail.com>
CC: Xi Wang <xi.wang@gmail.com>
Signed-off-by: Yang Shi <yang.shi@linaro.org>
---
V4 --> V3:
* Save/restore x25 and x26

V3 --> V2:
* Make FP point to FP'
* Fix a compile warning

 arch/arm64/net/bpf_jit_comp.c | 44 ++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 39 insertions(+), 5 deletions(-)

diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index ac8b548..86a3253 100644
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -50,7 +50,7 @@ static const int bpf2a64[] = {
 	[BPF_REG_8] = A64_R(21),
 	[BPF_REG_9] = A64_R(22),
 	/* read-only frame pointer to access stack */
-	[BPF_REG_FP] = A64_FP,
+	[BPF_REG_FP] = A64_R(25),
 	/* temporary register for internal BPF JIT */
 	[TMP_REG_1] = A64_R(23),
 	[TMP_REG_2] = A64_R(24),
@@ -155,16 +155,47 @@ static void build_prologue(struct jit_ctx *ctx)
 	stack_size += 4; /* extra for skb_copy_bits buffer */
 	stack_size = STACK_ALIGN(stack_size);
 
+	/*
+	 * BPF prog stack layout
+	 *
+	 *                         high
+	 * original A64_SP =>   0:+-----+ BPF prologue
+	 *                        |FP/LR|
+	 * current A64_FP =>  -16:+-----+
+	 *                        | ... | callee saved registers
+	 *                        +-----+
+	 *                        |     | x25/x26
+	 * BPF fp register => -80:+-----+
+	 *                        |     |
+	 *                        | ... | BPF prog stack
+	 *                        |     |
+	 *                        |     |
+	 * current A64_SP =>      +-----+
+	 *                        |     |
+	 *                        | ... | Function call stack
+	 *                        |     |
+	 *                        +-----+
+	 *                          low
+	 *
+	 */
+
+	/* Save FP and LR registers to stay align with ARM64 AAPCS */
+	emit(A64_PUSH(A64_FP, A64_LR, A64_SP), ctx);
+	emit(A64_MOV(1, A64_FP, A64_SP), ctx);
+
 	/* Save callee-saved register */
 	emit(A64_PUSH(r6, r7, A64_SP), ctx);
 	emit(A64_PUSH(r8, r9, A64_SP), ctx);
 	if (ctx->tmp_used)
 		emit(A64_PUSH(tmp1, tmp2, A64_SP), ctx);
 
-	/* Set up frame pointer */
+	/* Save fp (x25) and x26. SP requires 16 bytes alignment */
+	emit(A64_PUSH(fp, A64_R(26), A64_SP), ctx);
+
+	/* Set up BPF prog stack base register (x25) */
 	emit(A64_MOV(1, fp, A64_SP), ctx);
 
-	/* Set up BPF stack */
+	/* Set up function call stack */
 	emit(A64_SUB_I(1, A64_SP, A64_SP, stack_size), ctx);
 
 	/* Clear registers A and X */
@@ -190,14 +221,17 @@ static void build_epilogue(struct jit_ctx *ctx)
 	/* We're done with BPF stack */
 	emit(A64_ADD_I(1, A64_SP, A64_SP, stack_size), ctx);
 
+	/* Restore fs (x25) and x26 */
+	emit(A64_POP(fp, A64_R(26), A64_SP), ctx);
+
 	/* Restore callee-saved register */
 	if (ctx->tmp_used)
 		emit(A64_POP(tmp1, tmp2, A64_SP), ctx);
 	emit(A64_POP(r8, r9, A64_SP), ctx);
 	emit(A64_POP(r6, r7, A64_SP), ctx);
 
-	/* Restore frame pointer */
-	emit(A64_MOV(1, fp, A64_SP), ctx);
+	/* Restore FP/LR registers */
+	emit(A64_POP(A64_FP, A64_LR, A64_SP), ctx);
 
 	/* Set return value */
 	emit(A64_MOV(1, A64_R(0), r0), ctx);
-- 
2.0.2

Re: [PATCH V4 2/2] arm64: bpf: make BPF prologue and epilogue align with ARM64 AAPCS

[Z Lim]

On Mon, Nov 16, 2015 at 2:35 PM, Yang Shi <yang.shi@linaro.org> wrote:
> Save and restore FP/LR in BPF prog prologue and epilogue, save SP to FP
> in prologue in order to get the correct stack backtrace.
>
> However, ARM64 JIT used FP (x29) as eBPF fp register, FP is subjected to
> change during function call so it may cause the BPF prog stack base address
> change too.
>
> Use x25 to replace FP as BPF stack base register (fp). Since x25 is callee
> saved register, so it will keep intact during function call.
> It is initialized in BPF prog prologue when BPF prog is started to run
> everytime. Save and restore x25/x26 in BPF prologue and epilogue to keep
> them intact for the outside of BPF. Actually, x26 is unnecessary, but SP
> requires 16 bytes alignment.
>
> So, the BPF stack layout looks like:
>
>                                  high
>          original A64_SP =>   0:+-----+ BPF prologue
>                                 |FP/LR|
>          current A64_FP =>  -16:+-----+
>                                 | ... | callee saved registers
>                                 +-----+
>                                 |     | x25/x26
>          BPF fp register => -80:+-----+
>                                 |     |
>                                 | ... | BPF prog stack
>                                 |     |
>                                 |     |
>          current A64_SP =>      +-----+
>                                 |     |
>                                 | ... | Function call stack
>                                 |     |
>                                 +-----+
>                                   low
>
> CC: Zi Shen Lim <zlim.lnx@gmail.com>
> CC: Xi Wang <xi.wang@gmail.com>
> Signed-off-by: Yang Shi <yang.shi@linaro.org>

Acked-by: Zi Shen Lim <zlim.lnx@gmail.com>

Re: [PATCH V4 2/2] arm64: bpf: make BPF prologue and epilogue align with ARM64 AAPCS

[Z Lim]

On Mon, Nov 16, 2015 at 2:35 PM, Yang Shi <yang.shi@linaro.org> wrote:
> Save and restore FP/LR in BPF prog prologue and epilogue, save SP to FP
> in prologue in order to get the correct stack backtrace.
>
> However, ARM64 JIT used FP (x29) as eBPF fp register, FP is subjected to
> change during function call so it may cause the BPF prog stack base address
> change too.
>
> Use x25 to replace FP as BPF stack base register (fp). Since x25 is callee
> saved register, so it will keep intact during function call.
> It is initialized in BPF prog prologue when BPF prog is started to run
> everytime. Save and restore x25/x26 in BPF prologue and epilogue to keep
> them intact for the outside of BPF. Actually, x26 is unnecessary, but SP
> requires 16 bytes alignment.
>
> So, the BPF stack layout looks like:
>
>                                  high
>          original A64_SP =>   0:+-----+ BPF prologue
>                                 |FP/LR|
>          current A64_FP =>  -16:+-----+
>                                 | ... | callee saved registers
>                                 +-----+
>                                 |     | x25/x26
>          BPF fp register => -80:+-----+
>                                 |     |
>                                 | ... | BPF prog stack
>                                 |     |
>                                 |     |
>          current A64_SP =>      +-----+
>                                 |     |
>                                 | ... | Function call stack
>                                 |     |
>                                 +-----+
>                                   low
>
> CC: Zi Shen Lim <zlim.lnx@gmail.com>
> CC: Xi Wang <xi.wang@gmail.com>
> Signed-off-by: Yang Shi <yang.shi@linaro.org>

Acked-by: Zi Shen Lim <zlim.lnx@gmail.com>

[PATCH V4 2/2] arm64: bpf: make BPF prologue and epilogue align with ARM64 AAPCS

[Z Lim]

On Mon, Nov 16, 2015 at 2:35 PM, Yang Shi <yang.shi@linaro.org> wrote:
> Save and restore FP/LR in BPF prog prologue and epilogue, save SP to FP
> in prologue in order to get the correct stack backtrace.
>
> However, ARM64 JIT used FP (x29) as eBPF fp register, FP is subjected to
> change during function call so it may cause the BPF prog stack base address
> change too.
>
> Use x25 to replace FP as BPF stack base register (fp). Since x25 is callee
> saved register, so it will keep intact during function call.
> It is initialized in BPF prog prologue when BPF prog is started to run
> everytime. Save and restore x25/x26 in BPF prologue and epilogue to keep
> them intact for the outside of BPF. Actually, x26 is unnecessary, but SP
> requires 16 bytes alignment.
>
> So, the BPF stack layout looks like:
>
>                                  high
>          original A64_SP =>   0:+-----+ BPF prologue
>                                 |FP/LR|
>          current A64_FP =>  -16:+-----+
>                                 | ... | callee saved registers
>                                 +-----+
>                                 |     | x25/x26
>          BPF fp register => -80:+-----+
>                                 |     |
>                                 | ... | BPF prog stack
>                                 |     |
>                                 |     |
>          current A64_SP =>      +-----+
>                                 |     |
>                                 | ... | Function call stack
>                                 |     |
>                                 +-----+
>                                   low
>
> CC: Zi Shen Lim <zlim.lnx@gmail.com>
> CC: Xi Wang <xi.wang@gmail.com>
> Signed-off-by: Yang Shi <yang.shi@linaro.org>

Acked-by: Zi Shen Lim <zlim.lnx@gmail.com>

Re: [PATCH V4 2/2] arm64: bpf: make BPF prologue and epilogue align with ARM64 AAPCS

[Alexei Starovoitov]

On Mon, Nov 16, 2015 at 08:37:11PM -0800, Z Lim wrote:
> On Mon, Nov 16, 2015 at 2:35 PM, Yang Shi <yang.shi@linaro.org> wrote:
> > Save and restore FP/LR in BPF prog prologue and epilogue, save SP to FP
> > in prologue in order to get the correct stack backtrace.
...
> > CC: Zi Shen Lim <zlim.lnx@gmail.com>
> > CC: Xi Wang <xi.wang@gmail.com>
> > Signed-off-by: Yang Shi <yang.shi@linaro.org>
> 
> Acked-by: Zi Shen Lim <zlim.lnx@gmail.com>

great that it's finalized.
Yang, please resubmit both patches to netdev as fresh submission so it
gets picked up by patchwork.

Re: [PATCH V4 2/2] arm64: bpf: make BPF prologue and epilogue align with ARM64 AAPCS

[Alexei Starovoitov]

On Mon, Nov 16, 2015 at 08:37:11PM -0800, Z Lim wrote:
> On Mon, Nov 16, 2015 at 2:35 PM, Yang Shi <yang.shi@linaro.org> wrote:
> > Save and restore FP/LR in BPF prog prologue and epilogue, save SP to FP
> > in prologue in order to get the correct stack backtrace.
...
> > CC: Zi Shen Lim <zlim.lnx@gmail.com>
> > CC: Xi Wang <xi.wang@gmail.com>
> > Signed-off-by: Yang Shi <yang.shi@linaro.org>
> 
> Acked-by: Zi Shen Lim <zlim.lnx@gmail.com>

great that it's finalized.
Yang, please resubmit both patches to netdev as fresh submission so it
gets picked up by patchwork.

[PATCH V4 2/2] arm64: bpf: make BPF prologue and epilogue align with ARM64 AAPCS

[Alexei Starovoitov]

On Mon, Nov 16, 2015 at 08:37:11PM -0800, Z Lim wrote:
> On Mon, Nov 16, 2015 at 2:35 PM, Yang Shi <yang.shi@linaro.org> wrote:
> > Save and restore FP/LR in BPF prog prologue and epilogue, save SP to FP
> > in prologue in order to get the correct stack backtrace.
...
> > CC: Zi Shen Lim <zlim.lnx@gmail.com>
> > CC: Xi Wang <xi.wang@gmail.com>
> > Signed-off-by: Yang Shi <yang.shi@linaro.org>
> 
> Acked-by: Zi Shen Lim <zlim.lnx@gmail.com>

great that it's finalized.
Yang, please resubmit both patches to netdev as fresh submission so it
gets picked up by patchwork.

Re: [PATCH V4 2/2] arm64: bpf: make BPF prologue and epilogue align with ARM64 AAPCS

[Shi, Yang]

On 11/16/2015 8:41 PM, Alexei Starovoitov wrote:
> On Mon, Nov 16, 2015 at 08:37:11PM -0800, Z Lim wrote:
>> On Mon, Nov 16, 2015 at 2:35 PM, Yang Shi <yang.shi@linaro.org> wrote:
>>> Save and restore FP/LR in BPF prog prologue and epilogue, save SP to FP
>>> in prologue in order to get the correct stack backtrace.
> ...
>>> CC: Zi Shen Lim <zlim.lnx@gmail.com>
>>> CC: Xi Wang <xi.wang@gmail.com>
>>> Signed-off-by: Yang Shi <yang.shi@linaro.org>
>>
>> Acked-by: Zi Shen Lim <zlim.lnx@gmail.com>
>
> great that it's finalized.
> Yang, please resubmit both patches to netdev as fresh submission so it
> gets picked up by patchwork.

OK, btw the first one has been applied by David.

Yang

>

Re: [PATCH V4 2/2] arm64: bpf: make BPF prologue and epilogue align with ARM64 AAPCS

[Shi, Yang]

On 11/16/2015 8:41 PM, Alexei Starovoitov wrote:
> On Mon, Nov 16, 2015 at 08:37:11PM -0800, Z Lim wrote:
>> On Mon, Nov 16, 2015 at 2:35 PM, Yang Shi <yang.shi@linaro.org> wrote:
>>> Save and restore FP/LR in BPF prog prologue and epilogue, save SP to FP
>>> in prologue in order to get the correct stack backtrace.
> ...
>>> CC: Zi Shen Lim <zlim.lnx@gmail.com>
>>> CC: Xi Wang <xi.wang@gmail.com>
>>> Signed-off-by: Yang Shi <yang.shi@linaro.org>
>>
>> Acked-by: Zi Shen Lim <zlim.lnx@gmail.com>
>
> great that it's finalized.
> Yang, please resubmit both patches to netdev as fresh submission so it
> gets picked up by patchwork.

OK, btw the first one has been applied by David.

Yang

>

[PATCH V4 2/2] arm64: bpf: make BPF prologue and epilogue align with ARM64 AAPCS

[Shi, Yang]

On 11/16/2015 8:41 PM, Alexei Starovoitov wrote:
> On Mon, Nov 16, 2015 at 08:37:11PM -0800, Z Lim wrote:
>> On Mon, Nov 16, 2015 at 2:35 PM, Yang Shi <yang.shi@linaro.org> wrote:
>>> Save and restore FP/LR in BPF prog prologue and epilogue, save SP to FP
>>> in prologue in order to get the correct stack backtrace.
> ...
>>> CC: Zi Shen Lim <zlim.lnx@gmail.com>
>>> CC: Xi Wang <xi.wang@gmail.com>
>>> Signed-off-by: Yang Shi <yang.shi@linaro.org>
>>
>> Acked-by: Zi Shen Lim <zlim.lnx@gmail.com>
>
> great that it's finalized.
> Yang, please resubmit both patches to netdev as fresh submission so it
> gets picked up by patchwork.

OK, btw the first one has been applied by David.

Yang

>

Re: [PATCH V4 2/2] arm64: bpf: make BPF prologue and epilogue align with ARM64 AAPCS

[David Miller]

From: Alexei Starovoitov <alexei.starovoitov@gmail.com>
Date: Mon, 16 Nov 2015 20:41:46 -0800

> On Mon, Nov 16, 2015 at 08:37:11PM -0800, Z Lim wrote:
>> On Mon, Nov 16, 2015 at 2:35 PM, Yang Shi <yang.shi@linaro.org> wrote:
>> > Save and restore FP/LR in BPF prog prologue and epilogue, save SP to FP
>> > in prologue in order to get the correct stack backtrace.
> ...
>> > CC: Zi Shen Lim <zlim.lnx@gmail.com>
>> > CC: Xi Wang <xi.wang@gmail.com>
>> > Signed-off-by: Yang Shi <yang.shi@linaro.org>
>> 
>> Acked-by: Zi Shen Lim <zlim.lnx@gmail.com>
> 
> great that it's finalized.
> Yang, please resubmit both patches to netdev as fresh submission so it
> gets picked up by patchwork.

He doesn't need to, I applied patch #1 independently the other day while
the details of #2 were being worked out.

[PATCH V4 2/2] arm64: bpf: make BPF prologue and epilogue align with ARM64 AAPCS

[David Miller]

From: Alexei Starovoitov <alexei.starovoitov@gmail.com>
Date: Mon, 16 Nov 2015 20:41:46 -0800

> On Mon, Nov 16, 2015 at 08:37:11PM -0800, Z Lim wrote:
>> On Mon, Nov 16, 2015 at 2:35 PM, Yang Shi <yang.shi@linaro.org> wrote:
>> > Save and restore FP/LR in BPF prog prologue and epilogue, save SP to FP
>> > in prologue in order to get the correct stack backtrace.
> ...
>> > CC: Zi Shen Lim <zlim.lnx@gmail.com>
>> > CC: Xi Wang <xi.wang@gmail.com>
>> > Signed-off-by: Yang Shi <yang.shi@linaro.org>
>> 
>> Acked-by: Zi Shen Lim <zlim.lnx@gmail.com>
> 
> great that it's finalized.
> Yang, please resubmit both patches to netdev as fresh submission so it
> gets picked up by patchwork.

He doesn't need to, I applied patch #1 independently the other day while
the details of #2 were being worked out.

Re: [PATCH V4 2/2] arm64: bpf: make BPF prologue and epilogue align with ARM64 AAPCS

[David Miller]

From: Yang Shi <yang.shi@linaro.org>
Date: Mon, 16 Nov 2015 14:35:35 -0800

> Save and restore FP/LR in BPF prog prologue and epilogue, save SP to FP
> in prologue in order to get the correct stack backtrace.
> 
> However, ARM64 JIT used FP (x29) as eBPF fp register, FP is subjected to
> change during function call so it may cause the BPF prog stack base address
> change too.
> 
> Use x25 to replace FP as BPF stack base register (fp). Since x25 is callee
> saved register, so it will keep intact during function call.
> It is initialized in BPF prog prologue when BPF prog is started to run
> everytime. Save and restore x25/x26 in BPF prologue and epilogue to keep
> them intact for the outside of BPF. Actually, x26 is unnecessary, but SP
> requires 16 bytes alignment.
> 
> So, the BPF stack layout looks like:
 ...
> Signed-off-by: Yang Shi <yang.shi@linaro.org>

Applied, thank you.

[PATCH V4 2/2] arm64: bpf: make BPF prologue and epilogue align with ARM64 AAPCS

[David Miller]

From: Yang Shi <yang.shi@linaro.org>
Date: Mon, 16 Nov 2015 14:35:35 -0800

> Save and restore FP/LR in BPF prog prologue and epilogue, save SP to FP
> in prologue in order to get the correct stack backtrace.
> 
> However, ARM64 JIT used FP (x29) as eBPF fp register, FP is subjected to
> change during function call so it may cause the BPF prog stack base address
> change too.
> 
> Use x25 to replace FP as BPF stack base register (fp). Since x25 is callee
> saved register, so it will keep intact during function call.
> It is initialized in BPF prog prologue when BPF prog is started to run
> everytime. Save and restore x25/x26 in BPF prologue and epilogue to keep
> them intact for the outside of BPF. Actually, x26 is unnecessary, but SP
> requires 16 bytes alignment.
> 
> So, the BPF stack layout looks like:
 ...
> Signed-off-by: Yang Shi <yang.shi@linaro.org>

Applied, thank you.