[PATCH 00/10]: KVM updates for 2.6.23.9 (-stable)

[PATCH 00/10]: KVM updates for 2.6.23.9 (-stable)

[Avi Kivity]

The following patchset fixes some bugs in KVM for the next 2.6.23.y stable
release.  Please apply.

Amit Shah (2):
      KVM: x86 emulator: Use emulator_write_emulated and not emulator_write_std
      KVM: SVM: Fix FPU leak while emulating clts

Aurelien Jarno (1):
      KVM: x86 emulator: fix access registers for instructions with ModR/M byte and Mod = 3

Avi Kivity (4):
      KVM: x86 emulator: invd instruction
      KVM: SVM: Intercept the 'invd' and 'wbinvd' instructions
      KVM: Skip pio instruction when it is emulated, not executed
      KVM: VMX: Force vm86 mode if setting flags during real mode

Eddie Dong (1):
      KVM: VMX: Reset mmu context when entering real mode

Marko Kohtala (1):
      KVM: Fix hang on uniprocessor

Sheng Yang (1):
      KVM: x86 emulator: implement 'movnti mem, reg'


 drivers/kvm/kvm_main.c    |   16 ++++++++++------
 drivers/kvm/mmu.c         |    1 +
 drivers/kvm/svm.c         |    4 ++++
 drivers/kvm/vmx.c         |    3 +++
 drivers/kvm/x86_emulate.c |   30 +++++++++++++++++++++++++++---
 5 files changed, 45 insertions(+), 9 deletions(-)

[PATCH 01/10] KVM: x86 emulator: implement 'movnti mem, reg'

[Avi Kivity]

From: Sheng Yang <sheng.yang@intel.com>

Implement emulation of instruction:
    movnti m32/m64, r32/r64
    opcode: 0x0f 0xc3

Needed to support Linux 2.6.16 as guest (used for mmio).

Signed-off-by: Sheng Yang <sheng.yang@intel.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
---
 drivers/kvm/x86_emulate.c |    7 ++++++-
 1 files changed, 6 insertions(+), 1 deletions(-)

diff --git a/drivers/kvm/x86_emulate.c b/drivers/kvm/x86_emulate.c
index 4b8a0cc..804e86c 100644
--- a/drivers/kvm/x86_emulate.c
+++ b/drivers/kvm/x86_emulate.c
@@ -198,7 +198,8 @@ static u16 twobyte_table[256] = {
 	0, 0, ByteOp | DstReg | SrcMem | ModRM | Mov,
 	    DstReg | SrcMem16 | ModRM | Mov,
 	/* 0xC0 - 0xCF */
-	0, 0, 0, 0, 0, 0, 0, ImplicitOps | ModRM, 0, 0, 0, 0, 0, 0, 0, 0,
+	0, 0, 0, DstMem | SrcReg | ModRM | Mov, 0, 0, 0, ImplicitOps | ModRM,
+	0, 0, 0, 0, 0, 0, 0, 0,
 	/* 0xD0 - 0xDF */
 	0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
 	/* 0xE0 - 0xEF */
@@ -1324,6 +1325,10 @@ twobyte_insn:
 		dst.bytes = op_bytes;
 		dst.val = (d & ByteOp) ? (s8) src.val : (s16) src.val;
 		break;
+	case 0xc3:		/* movnti */
+		dst.bytes = op_bytes;
+		dst.val = (op_bytes == 4) ? (u32) src.val : (u64) src.val;
+		break;
 	}
 	goto writeback;
 
-- 
1.5.3

[PATCH 02/10] KVM: x86 emulator: fix access registers for instructions with ModR/M byte and Mod = 3

[Avi Kivity]

From: Aurelien Jarno <aurelien@aurel32.net>

The patch belows changes the access type to register from memory for
instructions that are declared as SrcMem or DstMem, but have a
ModR/M byte with Mod = 3.

It fixes (at least) the lmsw and smsw instructions on an AMD64 CPU,
which are needed for FreeBSD.

Signed-off-by: Aurelien Jarno <aurelien@aurel32.net>
Signed-off-by: Avi Kivity <avi@qumranet.com>
---
 drivers/kvm/x86_emulate.c |   17 +++++++++++++++++
 1 files changed, 17 insertions(+), 0 deletions(-)

diff --git a/drivers/kvm/x86_emulate.c b/drivers/kvm/x86_emulate.c
index 804e86c..08cd4a3 100644
--- a/drivers/kvm/x86_emulate.c
+++ b/drivers/kvm/x86_emulate.c
@@ -773,6 +773,14 @@ done_prefixes:
 	case SrcMem:
 		src.bytes = (d & ByteOp) ? 1 : op_bytes;
 	      srcmem_common:
+		/*
+		 * For instructions with a ModR/M byte, switch to register
+		 * access if Mod = 3.
+		 */
+		if ((d & ModRM) && modrm_mod == 3) {
+			src.type = OP_REG;
+			break;
+		}
 		src.type = OP_MEM;
 		src.ptr = (unsigned long *)cr2;
 		if ((rc = ops->read_emulated((unsigned long)src.ptr,
@@ -839,6 +847,15 @@ done_prefixes:
 		dst.type = OP_MEM;
 		dst.ptr = (unsigned long *)cr2;
 		dst.bytes = (d & ByteOp) ? 1 : op_bytes;
+		dst.val = 0;
+		/*
+		 * For instructions with a ModR/M byte, switch to register
+		 * access if Mod = 3.
+		 */
+		if ((d & ModRM) && modrm_mod == 3) {
+			dst.type = OP_REG;
+			break;
+		}
 		if (d & BitOp) {
 			unsigned long mask = ~(dst.bytes * 8 - 1);
 
-- 
1.5.3

[PATCH 03/10] KVM: x86 emulator: invd instruction

[Avi Kivity]

Emulate the 'invd' instruction (opcode 0f 08).

Signed-off-by: Avi Kivity <avi@qumranet.com>
---
 drivers/kvm/x86_emulate.c |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)

diff --git a/drivers/kvm/x86_emulate.c b/drivers/kvm/x86_emulate.c
index 08cd4a3..14ad4b4 100644
--- a/drivers/kvm/x86_emulate.c
+++ b/drivers/kvm/x86_emulate.c
@@ -156,7 +156,7 @@ static u8 opcode_table[256] = {
 static u16 twobyte_table[256] = {
 	/* 0x00 - 0x0F */
 	0, SrcMem | ModRM | DstReg, 0, 0, 0, 0, ImplicitOps, 0,
-	0, ImplicitOps, 0, 0, 0, ImplicitOps | ModRM, 0, 0,
+	ImplicitOps, ImplicitOps, 0, 0, 0, ImplicitOps | ModRM, 0, 0,
 	/* 0x10 - 0x1F */
 	0, 0, 0, 0, 0, 0, 0, 0, ImplicitOps | ModRM, 0, 0, 0, 0, 0, 0, 0,
 	/* 0x20 - 0x2F */
@@ -1353,6 +1353,8 @@ twobyte_special_insn:
 	/* Disable writeback. */
 	no_wb = 1;
 	switch (b) {
+	case 0x08:		/* invd */
+		break;
 	case 0x09:		/* wbinvd */
 		break;
 	case 0x0d:		/* GrpP (prefetch) */
-- 
1.5.3

[PATCH 04/10] KVM: SVM: Intercept the 'invd' and 'wbinvd' instructions

[Avi Kivity]

'invd' can destroy host data, and 'wbinvd' allows the guest to induce
long (milliseconds) latencies.

Noted by Ben Serebrin.

Signed-off-by: Avi Kivity <avi@qumranet.com>
---
 drivers/kvm/svm.c |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)

diff --git a/drivers/kvm/svm.c b/drivers/kvm/svm.c
index bc818cc..fae8cc5 100644
--- a/drivers/kvm/svm.c
+++ b/drivers/kvm/svm.c
@@ -506,6 +506,7 @@ static void init_vmcb(struct vmcb *vmcb)
 		 */
 		/*              (1ULL << INTERCEPT_SELECTIVE_CR0) | */
 				(1ULL << INTERCEPT_CPUID) |
+				(1ULL << INTERCEPT_INVD) |
 				(1ULL << INTERCEPT_HLT) |
 				(1ULL << INTERCEPT_INVLPGA) |
 				(1ULL << INTERCEPT_IOIO_PROT) |
@@ -519,6 +520,7 @@ static void init_vmcb(struct vmcb *vmcb)
 				(1ULL << INTERCEPT_STGI) |
 				(1ULL << INTERCEPT_CLGI) |
 				(1ULL << INTERCEPT_SKINIT) |
+				(1ULL << INTERCEPT_WBINVD) |
 				(1ULL << INTERCEPT_MONITOR) |
 				(1ULL << INTERCEPT_MWAIT);
 
@@ -1319,6 +1321,7 @@ static int (*svm_exit_handlers[])(struct kvm_vcpu *vcpu,
 	[SVM_EXIT_VINTR]			= interrupt_window_interception,
 	/* [SVM_EXIT_CR0_SEL_WRITE]		= emulate_on_interception, */
 	[SVM_EXIT_CPUID]			= cpuid_interception,
+	[SVM_EXIT_INVD]                         = emulate_on_interception,
 	[SVM_EXIT_HLT]				= halt_interception,
 	[SVM_EXIT_INVLPG]			= emulate_on_interception,
 	[SVM_EXIT_INVLPGA]			= invalid_op_interception,
@@ -1333,6 +1336,7 @@ static int (*svm_exit_handlers[])(struct kvm_vcpu *vcpu,
 	[SVM_EXIT_STGI]				= invalid_op_interception,
 	[SVM_EXIT_CLGI]				= invalid_op_interception,
 	[SVM_EXIT_SKINIT]			= invalid_op_interception,
+	[SVM_EXIT_WBINVD]                       = emulate_on_interception,
 	[SVM_EXIT_MONITOR]			= invalid_op_interception,
 	[SVM_EXIT_MWAIT]			= invalid_op_interception,
 };
-- 
1.5.3

[PATCH 05/10] KVM: x86 emulator: Use emulator_write_emulated and not emulator_write_std

[Avi Kivity]

From: Amit Shah <amit.shah@qumranet.com>

emulator_write_std() is not implemented, and calling write_emulated should
work just as well in place of write_std.

Fixes emulator failures with the push r/m instruction.

Signed-off-by: Amit Shah <amit.shah@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
---
 drivers/kvm/x86_emulate.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/drivers/kvm/x86_emulate.c b/drivers/kvm/x86_emulate.c
index 14ad4b4..9fce95b 100644
--- a/drivers/kvm/x86_emulate.c
+++ b/drivers/kvm/x86_emulate.c
@@ -1066,7 +1066,7 @@ done_prefixes:
 			}
 			register_address_increment(_regs[VCPU_REGS_RSP],
 						   -dst.bytes);
-			if ((rc = ops->write_std(
+			if ((rc = ops->write_emulated(
 				     register_address(ctxt->ss_base,
 						      _regs[VCPU_REGS_RSP]),
 				     &dst.val, dst.bytes, ctxt)) != 0)
-- 
1.5.3

[PATCH 06/10] KVM: Fix hang on uniprocessor

[Avi Kivity]

From: Marko Kohtala <marko.kohtala@gmail.com>

first_cpu(cpus) returns the only CPU when NR_CPUS is 1 regardless of
the cpus mask. Therefore we avoid a kernel hang in
KVM_SET_MEMORY_REGION ioctl on uniprocessor by not entering the loop at
all.

Signed-off-by: Marko Kohtala <marko.kohtala@gmail.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
---
 drivers/kvm/kvm_main.c |    5 +++++
 1 files changed, 5 insertions(+), 0 deletions(-)

diff --git a/drivers/kvm/kvm_main.c b/drivers/kvm/kvm_main.c
index cd05579..b514dfb 100644
--- a/drivers/kvm/kvm_main.c
+++ b/drivers/kvm/kvm_main.c
@@ -273,6 +273,11 @@ void kvm_flush_remote_tlbs(struct kvm *kvm)
 			}
 	}
 
+	/* Uniprocessor kernel does not respect cpus in first_cpu. So
+	 * do not go there if we have nothing to do. */
+	if (cpus_empty(cpus))
+		return;
+
 	/*
 	 * We really want smp_call_function_mask() here.  But that's not
 	 * available, so ipi all cpus in parallel and wait for them
-- 
1.5.3

[PATCH 07/10] KVM: SVM: Fix FPU leak while emulating clts

[Avi Kivity]

From: Amit Shah <amit.shah@qumranet.com>

The clts code didn't use set_cr0 properly, so our lazy FPU
processing wasn't being done by the clts instruction at all.

(this isn't called on Intel as the hardware does the decode for us)

Signed-off-by: Amit Shah <amit.shah@qumranet.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
---
 drivers/kvm/kvm_main.c |    5 +----
 1 files changed, 1 insertions(+), 4 deletions(-)

diff --git a/drivers/kvm/kvm_main.c b/drivers/kvm/kvm_main.c
index b514dfb..504e81d 100644
--- a/drivers/kvm/kvm_main.c
+++ b/drivers/kvm/kvm_main.c
@@ -1163,10 +1163,7 @@ int emulate_invlpg(struct kvm_vcpu *vcpu, gva_t address)
 
 int emulate_clts(struct kvm_vcpu *vcpu)
 {
-	unsigned long cr0;
-
-	cr0 = vcpu->cr0 & ~CR0_TS_MASK;
-	kvm_arch_ops->set_cr0(vcpu, cr0);
+	kvm_arch_ops->set_cr0(vcpu, vcpu->cr0 & ~X86_CR0_TS);
 	return X86EMUL_CONTINUE;
 }
 
-- 
1.5.3

[PATCH 08/10] KVM: Skip pio instruction when it is emulated, not executed

[Avi Kivity]

If we defer updating rip until pio instructions are executed, we have a
problem with reset:  a pio reset updates rip, and when the instruction
completes we skip the emulated instruction, pointing rip somewhere completely
unrelated.

Fix by updating rip when we see decode the instruction, not after emulation.

Signed-off-by: Avi Kivity <avi@qumranet.com>
---
 drivers/kvm/kvm_main.c |    6 ++++--
 1 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/drivers/kvm/kvm_main.c b/drivers/kvm/kvm_main.c
index 504e81d..b58fdf3 100644
--- a/drivers/kvm/kvm_main.c
+++ b/drivers/kvm/kvm_main.c
@@ -1757,8 +1757,6 @@ static int complete_pio(struct kvm_vcpu *vcpu)
 	io->count -= io->cur_count;
 	io->cur_count = 0;
 
-	if (!io->count)
-		kvm_arch_ops->skip_emulated_instruction(vcpu);
 	return 0;
 }
 
@@ -1804,6 +1802,7 @@ int kvm_setup_pio(struct kvm_vcpu *vcpu, struct kvm_run *run, int in,
 
 	pio_dev = vcpu_find_pio_dev(vcpu, port);
 	if (!string) {
+		kvm_arch_ops->skip_emulated_instruction(vcpu);
 		kvm_arch_ops->cache_regs(vcpu);
 		memcpy(vcpu->pio_data, &vcpu->regs[VCPU_REGS_RAX], 4);
 		kvm_arch_ops->decache_regs(vcpu);
@@ -1850,6 +1849,9 @@ int kvm_setup_pio(struct kvm_vcpu *vcpu, struct kvm_run *run, int in,
 	vcpu->run->io.count = now;
 	vcpu->pio.cur_count = now;
 
+	if (now == count)
+		kvm_arch_ops->skip_emulated_instruction(vcpu);
+
 	for (i = 0; i < nr_pages; ++i) {
 		spin_lock(&vcpu->kvm->lock);
 		page = gva_to_page(vcpu, address + i * PAGE_SIZE);
-- 
1.5.3

[PATCH 09/10] KVM: VMX: Force vm86 mode if setting flags during real mode

[Avi Kivity]

When resetting from userspace, we need to handle the flags being cleared
even after we are in real mode.

Signed-off-by: Avi Kivity <avi@qumranet.com>
---
 drivers/kvm/vmx.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/drivers/kvm/vmx.c b/drivers/kvm/vmx.c
index 80628f6..c158a2d 100644
--- a/drivers/kvm/vmx.c
+++ b/drivers/kvm/vmx.c
@@ -463,6 +463,8 @@ static unsigned long vmx_get_rflags(struct kvm_vcpu *vcpu)
 
 static void vmx_set_rflags(struct kvm_vcpu *vcpu, unsigned long rflags)
 {
+	if (vcpu->rmode.active)
+		rflags |= IOPL_MASK | X86_EFLAGS_VM;
 	vmcs_writel(GUEST_RFLAGS, rflags);
 }
 
-- 
1.5.3

[PATCH 10/10] KVM: VMX: Reset mmu context when entering real mode

[Avi Kivity]

From: Eddie Dong <eddie.dong@intel.com>

Resetting an SMP guest will force AP enter real mode (RESET) with
paging enabled in protected mode. While current enter_rmode() can
only handle mode switch from nonpaging mode to real mode which leads
to SMP reboot failure.

Fix by reloading the mmu context on entering real mode.

Signed-off-by: Yaozu (Eddie) Dong <eddie.dong@intel.com>
Signed-off-by: Qing He <qing.he@intel.com>
Signed-off-by: Avi Kivity <avi@qumranet.com>
---
 drivers/kvm/mmu.c |    1 +
 drivers/kvm/vmx.c |    1 +
 2 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/drivers/kvm/mmu.c b/drivers/kvm/mmu.c
index 23965aa..56ab369 100644
--- a/drivers/kvm/mmu.c
+++ b/drivers/kvm/mmu.c
@@ -1066,6 +1066,7 @@ int kvm_mmu_reset_context(struct kvm_vcpu *vcpu)
 	destroy_kvm_mmu(vcpu);
 	return init_kvm_mmu(vcpu);
 }
+EXPORT_SYMBOL_GPL(kvm_mmu_reset_context);
 
 int kvm_mmu_load(struct kvm_vcpu *vcpu)
 {
diff --git a/drivers/kvm/vmx.c b/drivers/kvm/vmx.c
index c158a2d..916da29 100644
--- a/drivers/kvm/vmx.c
+++ b/drivers/kvm/vmx.c
@@ -957,6 +957,7 @@ static void enter_rmode(struct kvm_vcpu *vcpu)
 	fix_rmode_seg(VCPU_SREG_GS, &vcpu->rmode.gs);
 	fix_rmode_seg(VCPU_SREG_FS, &vcpu->rmode.fs);
 
+	kvm_mmu_reset_context(vcpu);
 	init_rmode_tss(vcpu->kvm);
 }
 
-- 
1.5.3

Re: [stable] [PATCH 00/10]: KVM updates for 2.6.23.9 (-stable)

[Greg KH]

On Sun, Dec 02, 2007 at 01:18:37PM +0200, Avi Kivity wrote:
> The following patchset fixes some bugs in KVM for the next 2.6.23.y stable
> release.  Please apply.

I'm guessing that all of these are already upstream in Linus's tree?

If so, can you give me the git commit ids for them all if you happen to
know them, so that I don't have to guess at them?  :)

thanks,

greg k-h

Re: [kvm-devel] [stable] [PATCH 00/10]: KVM updates for 2.6.23.9 (-stable)

[Avi Kivity]

Greg KH wrote:
> On Sun, Dec 02, 2007 at 01:18:37PM +0200, Avi Kivity wrote:
>   
>> The following patchset fixes some bugs in KVM for the next 2.6.23.y stable
>> release.  Please apply.
>>     
>
> I'm guessing that all of these are already upstream in Linus's tree?
>
>   

Yes, except one, which has been changed in a different way for 2.6.24.

> If so, can you give me the git commit ids for them all if you happen to
> know them, so that I don't have to guess at them?  :)
>
>   

1 a012e65aee48379a7a87eadafa74f878b61522b9
2 4e62417bf317504c0b85e0d7abd236f334f54eaf
3 651a3e29b3d19418d7a8a9787906061f9be7cc5f
4 cf5a94d1331b411b84414c13e43f578260942d6b
5 00b2ef475d4728ca53a2bc788c7978042907e354
6 N/A
7 404fb881b82cf0cf6981832f8d31a7484e4dee81
8 0967b7bf1c22b55777aba46ff616547feed0b141
9 78f7826868da8e27d097802139a3fec39f47f3b8
10 8668a3c468ed55d19514117a5a959d91d3d03823


-- 
error compiling committee.c: too many arguments to function

Re: [stable] [kvm-devel] [PATCH 00/10]: KVM updates for 2.6.23.9 (-stable)

[Greg KH]

On Mon, Dec 03, 2007 at 11:23:19AM +0200, Avi Kivity wrote:
> Greg KH wrote:
> > On Sun, Dec 02, 2007 at 01:18:37PM +0200, Avi Kivity wrote:
> >   
> >> The following patchset fixes some bugs in KVM for the next 2.6.23.y stable
> >> release.  Please apply.
> >>     
> >
> > I'm guessing that all of these are already upstream in Linus's tree?
> >
> >   
> 
> Yes, except one, which has been changed in a different way for 2.6.24.
> 
> > If so, can you give me the git commit ids for them all if you happen to
> > know them, so that I don't have to guess at them?  :)
> >
> >   
> 
> 1 a012e65aee48379a7a87eadafa74f878b61522b9
> 2 4e62417bf317504c0b85e0d7abd236f334f54eaf
> 3 651a3e29b3d19418d7a8a9787906061f9be7cc5f
> 4 cf5a94d1331b411b84414c13e43f578260942d6b
> 5 00b2ef475d4728ca53a2bc788c7978042907e354
> 6 N/A
> 7 404fb881b82cf0cf6981832f8d31a7484e4dee81
> 8 0967b7bf1c22b55777aba46ff616547feed0b141
> 9 78f7826868da8e27d097802139a3fec39f47f3b8
> 10 8668a3c468ed55d19514117a5a959d91d3d03823

Thanks, I appreciate it.

greg k-h

Re: [kvm-devel] [stable] [PATCH 00/10]: KVM updates for 2.6.23.9 (-stable)

[Avi Kivity]

Greg KH wrote:

  

>> 1 a012e65aee48379a7a87eadafa74f878b61522b9
>> 2 4e62417bf317504c0b85e0d7abd236f334f54eaf
>> 3 651a3e29b3d19418d7a8a9787906061f9be7cc5f
>> 4 cf5a94d1331b411b84414c13e43f578260942d6b
>> 5 00b2ef475d4728ca53a2bc788c7978042907e354
>> 6 N/A
>> 7 404fb881b82cf0cf6981832f8d31a7484e4dee81
>> 8 0967b7bf1c22b55777aba46ff616547feed0b141
>> 9 78f7826868da8e27d097802139a3fec39f47f3b8
>> 10 8668a3c468ed55d19514117a5a959d91d3d03823
>>     
>
> Thanks, I appreciate it.
>
>   

How do you want them in the future?  Inside the individual commit logs 
or in the introductory message?


-- 
error compiling committee.c: too many arguments to function

Re: [stable] [kvm-devel] [PATCH 00/10]: KVM updates for 2.6.23.9 (-stable)

[Greg KH]

On Mon, Dec 03, 2007 at 03:11:03PM +0200, Avi Kivity wrote:
> Greg KH wrote:
> 
>   
> 
> >> 1 a012e65aee48379a7a87eadafa74f878b61522b9
> >> 2 4e62417bf317504c0b85e0d7abd236f334f54eaf
> >> 3 651a3e29b3d19418d7a8a9787906061f9be7cc5f
> >> 4 cf5a94d1331b411b84414c13e43f578260942d6b
> >> 5 00b2ef475d4728ca53a2bc788c7978042907e354
> >> 6 N/A
> >> 7 404fb881b82cf0cf6981832f8d31a7484e4dee81
> >> 8 0967b7bf1c22b55777aba46ff616547feed0b141
> >> 9 78f7826868da8e27d097802139a3fec39f47f3b8
> >> 10 8668a3c468ed55d19514117a5a959d91d3d03823
> >>     
> >
> > Thanks, I appreciate it.
> >
> >   
> 
> How do you want them in the future?  Inside the individual commit logs 
> or in the introductory message?

Inside the individual commit logs is best, as that is where I have been
adding them by hand if you look at the last few releases.

thanks,

greg k-h