[Bug 764] New: btime in /proc/stat wobbles (even over 30 seconds)

[Bug 764] New: btime in /proc/stat wobbles (even over 30 seconds)

[Martin J. Bligh]

           Summary: btime in /proc/stat wobbles (even over 30 seconds)
    Kernel Version: 2.5.70 but also in 2.2.20
            Status: NEW
          Severity: normal
             Owner: johnstul@us.ibm.com
         Submitter: h.lambermont@aramiska.net


Distribution: Debian and Red Hat
Hardware Environment: i386
Software Environment: /proc
Problem Description:

btime in /proc/stat changes over time. We even see it wobble over 30 seconds.
See also
http://www.google.nl/search?q=cache:ISSy3HrMcvQJ:bugzilla.redhat.com/bugzilla/long_list.cgi%3Fbuglist%3D75107+btime+/proc/stat&hl=nl&ie=UTF-8

Steps to reproduce:

Comparing /proc/stat's btime every minute shows the differences.
We see this behaviour on all of our 1500 Linux machines.

Re: [Bug 764] New: btime in /proc/stat wobbles (even over 30 seconds)

[Mike Dresser]

On Mon, 2 Jun 2003, Martin J. Bligh wrote:

>            Summary: btime in /proc/stat wobbles (even over 30 seconds)
>     Kernel Version: 2.5.70 but also in 2.2.20

Happens in 2.4.20 as well, it wobbles by one every couple seconds.

Re: [Bug 764] New: btime in /proc/stat wobbles (even over 30 seconds)

[Kingsley Cheung]

On Mon, Jun 02, 2003 at 11:08:01AM -0400, Mike Dresser wrote:
> On Mon, 2 Jun 2003, Martin J. Bligh wrote:
> 
> >            Summary: btime in /proc/stat wobbles (even over 30 seconds)
> >     Kernel Version: 2.5.70 but also in 2.2.20
> 
> Happens in 2.4.20 as well, it wobbles by one every couple seconds.
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

I raised this earlier in March this year. See:

http://marc.theaimsgroup.com/?l=linux-kernel&m=104804927502272&w=2

I sent to Rusty trivial patch for the fix against 2.4.20 back then. 

-- 
		Kingsley

Re: [Bug 764] New: btime in /proc/stat wobbles (even over 30 seconds)

[Kingsley Cheung]

[-- Attachment #1: Type: text/plain, Size: 429 bytes --]

On Thu, Jun 05, 2003 at 05:19:15PM +1000, Kingsley Cheung wrote:
 
> I raised this earlier in March this year. See:
> 
> http://marc.theaimsgroup.com/?l=linux-kernel&m=104804927502272&w=2
> 
> I sent to Rusty trivial patch for the fix against 2.4.20 back then. 
> 

Attached is a trivial patch to fix the problem against 2.5.70.  I've
also attached the trivial 2.4.20 patch I sent to Rusty back for
completeness.

-- 
		Kingsley

[-- Attachment #2: proc_misc-2.5.70.patch --]
[-- Type: text/plain, Size: 1474 bytes --]

--- linux/fs/proc/proc_misc.c.orig	Fri Jun  6 12:13:44 2003
+++ linux/fs/proc/proc_misc.c	Fri Jun  6 12:14:46 2003
@@ -51,6 +51,9 @@
 #include <asm/tlb.h>
 #include <asm/div64.h>
 
+/* System boot time in seconds since the UNIX epoch.  */
+static time_t boottime;
+
 #define LOAD_INT(x) ((x) >> FSHIFT)
 #define LOAD_FRAC(x) LOAD_INT(((x) & (FIXED_1-1)) * 100)
 /*
@@ -378,7 +381,6 @@
 {
 	int i, len;
 	extern unsigned long total_forks;
-	u64 jif = get_jiffies_64() - INITIAL_JIFFIES;
 	unsigned int sum = 0, user = 0, nice = 0, system = 0, idle = 0, iowait = 0;
 
 	for (i = 0 ; i < NR_CPUS; i++) {
@@ -419,15 +421,14 @@
 		len += sprintf(page + len, " %u", kstat_irqs(i));
 #endif
 
-	do_div(jif, HZ);
 	len += sprintf(page + len,
 		"\nctxt %lu\n"
-		"btime %lu\n"
+		"btime %ld\n"
 		"processes %lu\n"
 		"procs_running %lu\n"
 		"procs_blocked %lu\n",
 		nr_context_switches(),
-		xtime.tv_sec - (unsigned long) jif,
+		boottime,
 		total_forks,
 		nr_running(),
 		nr_iowait());
@@ -612,6 +613,7 @@
 
 void __init proc_misc_init(void)
 {
+	u64 jif = get_jiffies_64() - INITIAL_JIFFIES;
 	struct proc_dir_entry *entry;
 	static struct {
 		char *name;
@@ -638,6 +640,10 @@
 		{"execdomains",	execdomains_read_proc},
 		{NULL,}
 	};
+
+	/* Intialise system boot time before creating /proc/stat entry. */
+	do_div(jif, HZ);
+	boottime = xtime.tv_sec - (time_t)jif;
 	for (p = simple_ones; p->name; p++)
 		create_proc_read_entry(p->name, 0, NULL, p->read_proc, NULL);
 

[-- Attachment #3: proc_misc-2.4.20.patch --]
[-- Type: text/plain, Size: 963 bytes --]

--- linux-2.4.20/fs/proc/proc_misc.c	Fri Mar 28 15:53:52 2003
+++ linux-2.4.20_boottime/fs/proc/proc_misc.c	Fri Mar 28 15:34:21 2003
@@ -41,6 +41,9 @@
 #include <asm/pgtable.h>
 #include <asm/io.h>
 
+/* System boot time in seconds since the UNIX epoch.  */
+static time_t boottime;
+
 #define LOAD_INT(x) ((x) >> FSHIFT)
 #define LOAD_FRAC(x) LOAD_INT(((x) & (FIXED_1-1)) * 100)
 /*
@@ -372,10 +375,10 @@
 
 	proc_sprintf(page, &off, &len,
 		"\nctxt %u\n"
-		"btime %lu\n"
+		"btime %ld\n"
 		"processes %lu\n",
 		kstat.context_swtch,
-		xtime.tv_sec - jif / HZ,
+		boottime,     
 		total_forks);
 
 	return proc_calc_metrics(page, start, off, count, eof, len);
@@ -580,6 +583,9 @@
 		{"execdomains",	execdomains_read_proc},
 		{NULL,}
 	};
+
+	/* Intialise system boot time before creating /proc/stat entry. */
+	boottime = xtime.tv_sec - jiffies / HZ;
 	for (p = simple_ones; p->name; p++)
 		create_proc_read_entry(p->name, 0, NULL, p->read_proc, NULL);
 

Re: [Bug 764] New: btime in /proc/stat wobbles (even over 30 seconds)

[Herbert Xu]

Kingsley Cheung <kingsley@aurema.com> wrote:
> 
> Attached is a trivial patch to fix the problem against 2.5.70.  I've
> also attached the trivial 2.4.20 patch I sent to Rusty back for
> completeness.

What happens when the system time is changed later on?
-- 
Debian GNU/Linux 3.0 is out! ( http://www.debian.org/ )
Email:  Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [Bug 764] New: btime in /proc/stat wobbles (even over 30 seconds)

[Kingsley Cheung]

On Fri, Jun 06, 2003 at 06:32:04PM +1000, Herbert Xu wrote:
> Kingsley Cheung <kingsley@aurema.com> wrote:
> > 
> > Attached is a trivial patch to fix the problem against 2.5.70.  I've
> > also attached the trivial 2.4.20 patch I sent to Rusty back for
> > completeness.
> 
> What happens when the system time is changed later on?

Well, without the patch the boottime would change as you change the
system time.  So if you set the system time forward an hour, your
boottime would go forward as well, and so forth.

With the patch, the boottime would remain the same regardless of
changes to the system time.  IMHO, this is probably for the better,
since now as it stands we have issues with the boottime changing under
us due to the way xtime and jiffies are updated.  To me, having an
unchanging boottime is more profitable than one that changes.
Applications could use the value as a reliable absolute time
reference.  For example, to find out the absolute time a process
started, you can add the boottime and the starttime of the process,
the latter being in jiffies after the system booted, and not expect
this value to change.

The tradeoff, though, is that it is possible to have the boottime
greater than the current time if you set the system time back enough.
I think setting it forward is a non-issue.  I could be wrong but so
far I believe that is worth putting up this with tradeoff given the
benefits of an unchanging boottime time.  There is no affect obtaining
the system uptime - people shouldn't go calculating system time minus
boottime, since uptime itself is provided.  Moreover, a similar
problem is what to do with file modification times in the future - we
do nothing.

What do you or others think?  If people wanted to keep the old
semantics of a boottime that changed with the system time then we'll
need another way to avoid the wobble.

--
		Kingsley

Re: [Bug 764] New: btime in /proc/stat wobbles (even over 30 seconds)

[Kingsley Cheung]

[-- Attachment #1: Type: text/plain, Size: 1519 bytes --]

On Mon, Jun 02, 2003 at 08:02:22AM -0700, Martin J. Bligh wrote:
>            Summary: btime in /proc/stat wobbles (even over 30 seconds)
>     Kernel Version: 2.5.70 but also in 2.2.20
>             Status: NEW
>           Severity: normal
>              Owner: johnstul@us.ibm.com
>          Submitter: h.lambermont@aramiska.net
> 
> 
> Distribution: Debian and Red Hat
> Hardware Environment: i386
> Software Environment: /proc
> Problem Description:
> 
> btime in /proc/stat changes over time. We even see it wobble over 30 seconds.
> See also
> http://www.google.nl/search?q=cache:ISSy3HrMcvQJ:bugzilla.redhat.com/bugzilla/long_list.cgi%3Fbuglist%3D75107+btime+/proc/stat&hl=nl&ie=UTF-8
> 
> Steps to reproduce:
> 
> Comparing /proc/stat's btime every minute shows the differences.
> We see this behaviour on all of our 1500 Linux machines.

I see that there has been a fix made for this since 2.5.70-bk13 or
2.5.70-bk14 that solves this problem by using the seqlock to ensure
that the jiffies and time of day are atomically read.

However, wouldn't it be better to have the boottime calculated only
once so that it is independent of changes in the system time that may
occur later?  Even with the fix with seqlock, the boottime can still
change back or forwards whenever the system time is set back or
forwards.  IMHO an unchanging boottime that is independent of the time
of day is the best approach.  Maybe something like the patch against
2.5.70-bk14 that I've attached.

What do people think?

-- 
		Kingsley

[-- Attachment #2: proc_misc-2.5.70-bk14.patch --]
[-- Type: text/plain, Size: 2150 bytes --]

--- linux/fs/proc/proc_misc.c.orig	Tue Jun 10 11:03:31 2003
+++ linux/fs/proc/proc_misc.c	Tue Jun 10 11:13:08 2003
@@ -51,6 +51,9 @@
 #include <asm/tlb.h>
 #include <asm/div64.h>
 
+/* System boot time in seconds since the UNIX epoch.  */
+static time_t boottime;
+
 #define LOAD_INT(x) ((x) >> FSHIFT)
 #define LOAD_FRAC(x) LOAD_INT(((x) & (FIXED_1-1)) * 100)
 /*
@@ -378,23 +381,7 @@
 {
 	int i, len;
 	extern unsigned long total_forks;
-	u64 jif;
 	unsigned int sum = 0, user = 0, nice = 0, system = 0, idle = 0, iowait = 0;
-	struct timeval now; 
-	unsigned long seq;
-
-	/* Atomically read jiffies and time of day */ 
-	do {
-		seq = read_seqbegin(&xtime_lock);
-
-		jif = get_jiffies_64();
-		do_gettimeofday(&now);
-	} while (read_seqretry(&xtime_lock, seq));
-
-	/* calc # of seconds since boot time */
-	jif -= INITIAL_JIFFIES;
-	jif = ((u64)now.tv_sec * HZ) + (now.tv_usec/(1000000/HZ)) - jif;
-	do_div(jif, HZ);
 
 	for (i = 0 ; i < NR_CPUS; i++) {
 		int j;
@@ -436,12 +423,12 @@
 
 	len += sprintf(page + len,
 		"\nctxt %lu\n"
-		"btime %lu\n"
+		"btime %ld\n"
 		"processes %lu\n"
 		"procs_running %lu\n"
 		"procs_blocked %lu\n",
 		nr_context_switches(),
-		(unsigned long)jif,
+		boottime,
 		total_forks,
 		nr_running(),
 		nr_iowait());
@@ -626,6 +613,9 @@
 
 void __init proc_misc_init(void)
 {
+	u64 jif;
+	unsigned long seq;
+	struct timeval now; 
 	struct proc_dir_entry *entry;
 	static struct {
 		char *name;
@@ -652,6 +642,24 @@
 		{"execdomains",	execdomains_read_proc},
 		{NULL,}
 	};
+
+	/* 
+	 * Intialise system boot time before creating /proc/stat
+	 * entry.  This way we avoid it changing whenever the system
+	 * time is changed.  Atomically read jiffies and time of day.
+	 */ 
+	do {
+		seq = read_seqbegin(&xtime_lock);
+		jif = get_jiffies_64();
+		do_gettimeofday(&now);
+	} while (read_seqretry(&xtime_lock, seq));
+
+	/* calc # of seconds since boot time */
+	jif -= INITIAL_JIFFIES;
+	jif = ((u64)now.tv_sec * HZ) + (now.tv_usec/(1000000/HZ)) - jif;
+	do_div(jif, HZ);
+	boottime = (time_t)jif;
+
 	for (p = simple_ones; p->name; p++)
 		create_proc_read_entry(p->name, 0, NULL, p->read_proc, NULL);
 

Re: [Bug 764] New: btime in /proc/stat wobbles (even over 30 seconds)

[john stultz]

On Wed, 2003-06-11 at 18:20, Kingsley Cheung wrote:

> I see that there has been a fix made for this since 2.5.70-bk13 or
> 2.5.70-bk14 that solves this problem by using the seqlock to ensure
> that the jiffies and time of day are atomically read.

And further then that, we loose less precision in some of the math so we
don't have the single second wobbles that were initially seen. 


> However, wouldn't it be better to have the boottime calculated only
> once so that it is independent of changes in the system time that may
> occur later?  Even with the fix with seqlock, the boottime can still
> change back or forwards whenever the system time is set back or
> forwards.  IMHO an unchanging boottime that is independent of the time
> of day is the best approach.  Maybe something like the patch against
> 2.5.70-bk14 that I've attached.
> 
> What do people think?

Really, I'm fine with the current semantics. At boot time the system
clock may not have been correct, and was corrected only after NTP
started up later in the boot sequence. So you could have some very funky
btimes. 

Even the current definition of btime = now - uptime has its own quirks
(when systems are suspended uptime doesn't increment, etc) but I think
its more likely to be correct then any other method (assuming the time
now is more accurate then time at boot thanks to ntp or whatnot). 

I'm curious, how are people using the btime value? I'd think uptime and
gettimeofday would be more useful bits of info, so I'd like to hear
more.

thanks
-john

Re: [Bug 764] New: btime in /proc/stat wobbles (even over 30 seconds)

[Kingsley Cheung]

On Fri, Jun 20, 2003 at 05:34:39PM -0700, john stultz wrote:
> On Wed, 2003-06-11 at 18:20, Kingsley Cheung wrote:
> 
> > I see that there has been a fix made for this since 2.5.70-bk13 or
> > 2.5.70-bk14 that solves this problem by using the seqlock to ensure
> > that the jiffies and time of day are atomically read.
> 
> And further then that, we loose less precision in some of the math so we
> don't have the single second wobbles that were initially seen. 
> 
> 
> > However, wouldn't it be better to have the boottime calculated only
> > once so that it is independent of changes in the system time that may
> > occur later?  Even with the fix with seqlock, the boottime can still
> > change back or forwards whenever the system time is set back or
> > forwards.  IMHO an unchanging boottime that is independent of the time
> > of day is the best approach.  Maybe something like the patch against
> > 2.5.70-bk14 that I've attached.
> > 
> > What do people think?
> 
> Really, I'm fine with the current semantics. At boot time the system
> clock may not have been correct, and was corrected only after NTP
> started up later in the boot sequence. So you could have some very funky
> btimes. 
> 

I guess that one of the tradeoffs to be considered for having a fixed
boottime.  If after boot time NTP corrects the system time, then the
boot time would not change in accordance with NTP's correction.

> Even the current definition of btime = now - uptime has its own quirks
> (when systems are suspended uptime doesn't increment, etc) but I think
> its more likely to be correct then any other method (assuming the time
> now is more accurate then time at boot thanks to ntp or whatnot). 
> 

When you mention that quirk above, wouldn't that mean then that the
boottime would be off by the amount of time the system is suspended?
If uptime or jiffies is not updated but 'now' (the system time) is,
then btime would be pushed forward...

> I'm curious, how are people using the btime value? I'd think uptime and
> gettimeofday would be more useful bits of info, so I'd like to hear
> more.

Yes, that would be interesting to know.

Personally, I have been using it to determine when processes started.
Starttime, and some other process data that is recorded and presented
in /proc in jiffies.  So to determine when a process started I took
that value, converted to seconds, and added it to btime.  I was a bit
surpised to find that btime changed, however.  And the way it is now
it'll still change whenever the system time is changed.  However, at
least with the fix from 2.5.70, it doesn't wobble anymore due to loss
of precision.

-- 
		Kingsley